176 iPS Mobile Devops with Donovan Brown and Josh Weber

1:25: Mobile Devops definition

  • Unique challenges
  • “It’s not just tools, it’s people in progress.”
  • Friction

4:30: Difficulty of mobile devops

7:15: Unit testing

  • C Sharp
  • React Native
  • Manual testing
  • UI type test
  • Verification

14:50: HockeyApp

27:00: Stories

  • Keynote: acquisition of Xamarin
  • Keeping your customers by delivering value

30:00: Experiments

32:40: Feature Flags


Pappadeaux (Jaim)

The Lean Startup (Charles)

Lyft (Charles)

BMW (Donovan)

Air hockey (Donovan)

HockeyApp (Josh)

Donovan on Twitter

This episode is sponsored by

comments powered by Disqus


Charles: Hey everybody and welcome to the iPhreaks show! This week on our panel we have Jaim Zuber.

Jaim: You know I know the locals don’t say this but I’m an old Allman Brothers fan from way back. Hello from Hot-lanta.

Charles: I’m Charles Max Wood from devchat.tv and this week we have two guests, we have Donovan Brown.

Donovan: Yep, hi I’m Donovan Brown I’m a senior devops program manager here at Microsoft.

Charles: And Josh Weber.

Josh: Hi, I’m Josh Weber. I’m a senior PM with the HockeyApp team.

Charles: Awesome. Now, the topic of our discussion today is mobile devops which isn’t really a term that I’ve heard before and I had it explained to me a little bit before this but do you want to kind of give us an introduction to what this is? One of you?

Donovan: Sure. I actually defined devops for Microsoft and we defined it very succinctly as the union of people, process, and products. It enables continuous delivery of value to our enjuicers and that applies to all software, mobile included but what you realize when you’re dealing with mobile is there’s very unique challenges with mobile. The device proliferation, the distribution of that app, the testing of the app, even the development of the app because before Xamarin and Cordova, you were having to write it multiple times. The reason that we classify mobile devops differently is because it’s just has a unique set of challenges but the goal of devops is the same no matter what application it is you’re writing.

Jaim: We should even back up and talk about devops because we’re mobile developers that’s what our backend teams deal with. We want to know what it is generally.

Donovan: Do you want to take that or?

Josh: Yes, my take has always been devops is the kind of new evolution of the Agile like Manifesto kind of an approach. Ultimately, as developers, we produce a product and the goal is to get that product into the customer’s hands. Anything that gets between you and customer’s really kind of like waste, it gets in your way. devops is basically encompassing not just the dev process but the entire kind of release process to that. I really like Donovan’s definition that it says, it’s not just tools, it’s people in process because I think it’s really critical that if you think through devops, you got to think through the entire pipeline and a lot of that is the human portion of that like people make decisions and based on how do you empower those people to make those decisions and how you encode that into a repeatable process so that it’s actually refined.

Donovan: What’s ironic is that people is the most difficult part and it’s the part that we try to get rid off as much as we possibly can. We literally try to automate every portion that we can so that humans don’t make mistakes because we do. That’s what makes us humans is that we make mistakes when we’re told to go do something. We have a lot of pressure where a machine will happily do the same thing over and over and over again and you made an interesting statement that, “Oh, I’m a mobile developer, that’s the backend people’s problem but how do you get the changes that you made to the front in the mobile application to your enjuicers? There’s friction there. There’s you having to stop building on a Mac if you’re doing iOS and then you’re going to have to then take that and get into the store somehow and how do you get it to your testers before it got in history on the side loaded on the devices so that they could go test it and give you feedback.

There’s a lot of friction. I always like to say they take the value from your fingertips and get it to the hands of your users. There’s a lot of friction in mobile and what we’re trying to do here with the acquisition of Xamarin, or the acquisition HockeyApp, and the integration with team services is to reduce that friction not only for your backend but for your frontend as well because as a user, I need them both to be in sync. I can’t use your new feature if the backend that supports it is not there and vice versa. I need them to move to the pipeline as a unit and with the integrations that we have in team services, you can now do that, the database, the web services and the mobile frontend all move as a unit. They go through their process together.

Charles: Do you think that mobile devops is more or less complicated than traditional devops because I mean, my first tech job was essentially managing servers for our University and so our operation stuff basically managed a few hundred maybe a thousand or a couple of thousand servers but we’re talking, especially, on a popular app, we’re talking tens of thousands or hundreds of thousands of devices, millions of devices. If there’s a change to Pokemon Go or something, a lot of people are getting this. Overall do you think that’s easier or harder or about the same?

Donovan: I think mobile devops is substantially more difficult than traditional devops. Let’s do a web app. If I’m doing a web application, I’m pushing it to one web server and it’s done. Everyone gets it. I only have to test on maybe Chrome, Firefox, Safari and IE, right? I’m done. To do mobile devops, you have to test it on as many physical devices as you possibly can and that’s why things like Perfecto Mobile’s test cloud, Xamarin test cloud, that’s why those are so important is because no one in their right mind should even go on a tip to buy all devices that you would need to be able to test successfully. What you do is you offload that to a service but again, that comes with waste, that comes with time. Unless you do what we’ve done at Microsoft and we integrate that into the pipeline. We take the lightest bits, we deploy them to thousands of machines for you and we run automated tests for you so that you don’t have to do it manually. Based on the output of that, we then say, “Okay, we believe this is ready for the store,” and the store is really going to do the mass distribution for you. Our goal is to get it into the store for you with the highest possible quality, and the lowest friction, and the highest velocity.

Charles: What are the pieces then that mobile team needs in their devops pipeline in order to successfully deploy apps that don’t have bugs and do all the builds and everything you’re talking about?

Donovan: That’s a good one too. I’ll let you do it.

Josh: It’s just kind of required. I wouldn’t say anything is necessarily required.

Donovan: Best practices.

Josh: But the best practices, they’re going to help you. I think really best practices are not that required it’s just that a lot of people have found that doing this activities gains more value than they benefit to your development cycle. Ultimately, it all starts with the build. You want to have a repeatable build that’s safe and usually triggered automatically. The faster you can detect even a build rate is the faster you know that there’s something wrong with your server. With the build rate, you should be doing automated testing so then includes things like unit tests which everybody hopefully is doing unit tests in their aim to software quality. I don’t know if you guys are unit testers or not.

Charles: I’m a fan. I think Jaim has done a fair bit too.

Josh: So I’d say, you definitely want to get unit testing but even beyond that especially in the mobile world where the devices are so different, the environments are heterogeneous that you really need to test on the actual physical devices. That’s for something like the test cloud really kicks in where you can actually run the actual real software before it gets out in production in an automated, repeatable way on actual devices so you can pitch these regressions that are really hard to catch kind of ad hoc. You’re not just going to load onto every device possible, you’re not really going to catch stuff.

Charles: My ad hoc testing is I play it out there and I have people complain.

Donovan: Your users are your testers at that.

Josh: It was good, I was at a conference one time and I ask some people if they did unit testing and some guy was like, “Yeah, I do a ton of unit testing. I load it up in the stimulator and like whatever section of the thing that I tested, I make sure I click right on that and I unit test that part that I just built. So I click manually on that every single time.” I was like, “What?” It’s a great start for testing.

Donovan: I always qualify automated unit tests because I get that answer a lot. In his defense, he was right. He did unit test that but automated unit test is the key because he only unit tested that one thing that he changed he didn’t go and unit test everything else he’s ever changed, where automated unit test will go and find that for you and I always give the–we’ve all done this as developers where your boss comes into your office and he’s breathing down your neck and says, “Oh my god, stop what you’re doing. There’s a bug, go fix it,” What do we do? We stop everything. We go and we code up a fix and it works now and we push it out the door. And then a week later here he is back in your office for some other reason, another bug. You do the exact same thing, you fix it, there’s no unit test, you push it out the door, everyone’s happy and all of a sudden you have deja vu. Three days later, that first bug comes back, and you’re like, “What the hell is this? I know I fixed this already.” You don’t realize the change that you’re making to fix one is actually breaking the other. Had I stopped, before he’s like “Fix this,” and written a unit test for the first one and let it pass, “Got it, good. It’s fixed. Push out the door.”

When I had made the change for the second one, that test would have failed and immediately they told me, “Donovan, these are related. The way that you’re fixing this isn’t correct.” Automated unit test is key. One other thing that you mentioned is that the way you do manual testing. I still think manual things is crucially important. You cannot rely solely on automated testing because users just do things that you didn’t program your test to go do. Your test uses it correctly.

Charles: But for unit test, the other thing for automated test in a lot of cases is I can run that on my machine really quickly and just know that it works.

Donovan: Sure.

Charles: So that’s the other thing. You don’t have to wait till it goes up to the build surface.

Donovan: Absolutely, very good point. You don’t have to wait on the test cloud, you can actually find that and it should be very quick as you pointed out. Unit tests are supposed to be isolated, they’re supposed to be extremely fast and let me know, “Yup, you’re good, go ahead and check this and we’ll go through.”

Charles: We were talking about manual testing.

Donovan: Yeah, I get a lot of people like, “Donovan, I love all your preaching about devops does that mean I don’t have to do manual tests?” I’m like, “No! I’m trying to automate as much as I can but you don’t get rid of that.” The story that I always tell which is great. I think Sam Guckenheimer told me this where he was saying they had run all their battery of automated testing, pacifying to close. The guy made a simple change, it was like a CSS change, no big deal, of course all the tests will run a path and all of a sudden, she has this huge drop off on completions of this wizard and they didn’t understand like, “What’s going on?” The CSS change made a text and a button of the same color so no one could see. What was I committing to? No human would click it but that automation happen to click the button because they know better not to. The perfect example of where you can rely on automation a lot but if ever there’s a user interface, you better have a human being look at that. That’s my personal opinion.

Jaim: If you’re talking about unit testing and in the test cloud we’re talking about Xamarin, C Sharp, how are you writing your unit tests? A lot of our fans are Native iPhone developers so we’re doing XC unit testing, how does testing work in the C Sharp world or how does that integrate with the test cloud?

Josh: I’m actually maybe not the best for that.

Donovan: I can fill that. We actually have a generator inside a studio that will actually allow you to connect either a physical device or an emulator and actually click through the app while we write the actual C Sharp for you. We generate the code that you need and you can go check that code , and that code then gets executed for you as your testing inside the Xamarin test.

Jaim: Those are like UI type test. Click on this button over here.

Donovan: Yes, good question. These are not unit tests. These are what I classify as integration test or a UI test because that is an installed application literally clicking on the button and doing what the app is designed to do, a unit test would never do that. It runs in isolation, completely, it doesn’t touch your database, it doesn’t touch file systems, it’s a unit. When you go into the test cloud, you are now doing full integration test because it’s going to try to call to a backend, it’s going to touch your database, it’s a real life test but those tests are actually written in C Sharp but they’re generated for you so even though you don’t know C Sharp you can still go pretty far with it.

Charles: I was going to ask, I mean most of our listeners are either Objective-C or Swift.

Donovan: Got it.

Charles: I was curious, is test cloud something that I can use if I write my stuff in Objective-C or Swift or JavaScript even?

Donovan: Great question. How your app is designed is of no importance to us. It’s the test that we need to make them understand. If you’re writing Native, I don’t want to say Native because Xamarin guys get mad at me, but if you’re writing in Objective-C or your writing directly in Java for the other platform, Google platform, it doesn’t matter to us because we don’t know. It’s just an app. You give us your IPK or your IPA file for other one, APK for the other guys and then we basically just install that on the app like we normally would. It’s the test that we need to have to understand then we’re just going to beat your app up and then tell you if it performed that way we that we expected to. Great question, but yeah.

Charles: So I could take my Native app, I can generate the test with the Xamarin tools and it doesn’t matter if they’re in C-Sharp, they just work.

Donovan: Exactly, exactly. Great question. Absolutely.

Jaim: Does it work also like with JavaScript framework like React Native?

Donovan: Cordova or whatever, again, it doesn’t matter. If you can get that app installed on a machine, we can test it for you.

Jaim: So how does verification work? You can press the button, obviously you can get a location of where to touch on the screen, how you verify that, it did the right thing.

Donovan: What you can do for example is let’s say we have an autofield to where you should now me, I click this button now I want you to load my address so you now have a text box. What I can say is that this textbox value should equal my address. This zip code better equal this number after I’ve taken this action of clicking that button. We’re able to then go back and code the verifications that we want. It also could be, “After I click this button, I better end up on this page of your application.” If I don’t, then you’ve navigated to the wrong place or maybe we got an error or things like that. There’s literally a search that you put in the code that you define. How do I say this  passed or failed? You’re in complete control over that.

Jaim: How does that work? To screen A versus screen B. How do you write a test for that?

Donovan: I wouldn’t exactly write the test. I would record the test. I would have the tool record the test. What I would do is I would fire up the application in either an emulator or simulator or even if a physical device tethered to my machine and then I would go inside Visual Studio create a Xamarin test project, then I would literally start saying record and what it would do then is actually mimic everything that I’m doing on the screen and when I click on the button it navigates to another screen, what I would be able to do then is verify it either by the title or by some information that expect me on that screen that yes, I landed on the right one it could be any attribute that says, “Yup, I know I ended up where I’m supposed to end up,” and then continue on with your testing. It’s the same way we test our fat clients or web app applications, it’s the same technology.

Josh: You just apply it to mobile.

Charles: We talked a lot about testing but what other things should people be thinking about putting into their devops process for mobile apps?

Josh: Distribution, the HockeyApp stuff. A lot of the stuff that we talk about especially with the automated testing and the build is usually kind of turned into continuous integration part but I think when we say mobile, we really try to extend that from continuous integration to continuous delivery and so just having an IK produced is a great first step and adds lot of value but really being able to get that and deployed onto the device automatically really changed it from integration to delivery.

Charles: It’s like we kill that for users, right?

Josh: That’s right. HockeyApp has a really great solution for that. It basically automates the entire process of uploading that stuff so after your basic tool gets produced, you produce your IPA, you produce that APK, you just upload it. It’s just a rest call that does an HTTP post up to the cloud base service and then basically the cloud HockeyApp service will host that file for you. When a device wants to download that app, all it needs to do is basically communicate, HockeyApp web app, it has Native apps for all the major platforms, you can log in, you can write authentication for your users so that you don’t leak that data out to the community you don’t want to or broadly distribute it as you want and then it’s just single click so we trigger the automatic installer to install it in each of the devices automatically.

Charles: I have a few questions about this and people can go if they want more details. We interviewed Thomas built in April. Just to give people an idea, you build the IPA, you hand it off to HockeyApp, HockeyApp pushes it up to the app store. How much does it really manage for you though as far as A getting it up to people who are going to test it out before you deliver it and B, if there’s some sort of approval process say for the IOS app store, does that communication then happen with HockeyApp or does that happen between you and them or how does that go on?

Josh: I would say that that depends really on how you like to do development and so we have the full range of customers that uses HockeyApp. I would say HockeyApp supports kind of all the scenarios in there. A lot of people have open data community where they don’t care who uses their app and in fact they actually prefer that as many people as possible use their beta app. They upload the HockeyApp and they actually just set it to be a public distribution and then they tweet it a bunch, they post it on Facebook, and they sent out emails to everybody with the URL and they’re just hoping that as many as possible do that. The idea there there being that every one extra beta tester is one better chance that I might be able to catch a crash or catch a bug in my software.

Charles: They go to that link, they sign up and then it just installs it on their phone?

Josh: You don’t even actually need to sign up so if you’re doing a public distribution, there’s no authentication and so all they need is the URL. They can directly install from this URL. It makes it really, really lightweight, the entire time from basically a dev post that they built to the time somebody could download, it would be like five seconds. It’s like multi-day review process to the app store and it was like a long process of maybe a response and all that sort, whatever your app is, just push it out. Then you go all the way to the site to where some of our enterprise or some of our corporate customers tend to be very restrictive. They may have corporate data or this is an app that they only want to distribute to a very, very, small number of people. HockeyApp allows you to really manage that kind of user restriction, that user access into it.

We integrate the things like directory stuff using for AAD corporate credentials that kind of things then you can actually enforce that, they have to sign up with corporate credentials and validate with the active server on the backend before they’re allowed to download this application. I would say that we have kind of a full spectrum there as to how comfortable you are it depends and I would say this is a scenario where I don’t think there is necessarily best practice. I think usually there’s some set of early release where you have  a small group let’s say the dev team or maybe just the QA team. You usually get to a certain quality level where you feel happy with that and you want to release to a larger group. Usually a larger group is either within your company or your organization or sometimes, sometimes not. It’s an external audience depending upon if you feel like that that kind of release mechanism being transparent is a value to your application and then you want to monitor make sure there’s some quality metrics back in there and go to finally prep the actual release, it helps support the production release.

I’m actually talking to a customer who says they actually use us for their deployment. Rather than emailing files back and forth between them and their release manager who actually goes to the Apple store and has the keys to the kingdom in the Apple store, they actually use HockeyApp. Since that person can install the app, they can actually install the app on their phone and then they know that they can get the exact same app that they just installed and test and download it from HockeyApp on the machine so they can manually upload to the store. It definitely has cute advantages that help reassure you that the production build you’re going to release correction is exactly the same as the one you’re about to launch.

Jaim: That’s very cool. I have a question on how you actually created a URL because one of the problems if you’re doing any type of enterprise distribution in the Apple ecosystem is, okay you’ve got this IPA and the bugs can find us. If it’s a new company, if they never set up the distribution, you got to talk to their web person and make sure they can handle all the different file types so HockeyApp handles all that for you.

Josh: Yeah. HockeyApp takes care of all of that. I mean ultimately no matter what kind of package you do, you do the standard packages that you do for a device so IPAs, APKs, that kind of thing you upload then to HockeyApp, and then HockeyApp automatically generates that URL for you. So you just share the URL. In fact, they actually have kind of a fancy feature where you just have the QR code so you can share just the QR code which codes the URL if you want it with your users but all of that is kind of transparently handled by HockeyApp. You just basically share the URL and they click the URL to install. If you like to, you can add the access restrictions in there depending upon how restrictive you want to be in your distribution but that’s somewhat independent from the actual distribution method. Ultimately, it’s just a couple of calls that get validated through the rest.

Jaim: That’s a great feature because that’s kind of a headache especially for the smaller clients who doesn’t really have a sophisticated web team. You have to send all these medication things like you can just do it. Click, and there’s people on, they’re allowed so not, it’s a really useful feature.

Josh: The really interesting there is when you start to couple something like through the VSTS offering where the STS sets up the continuous integration build task but it also includes the integration SAAS with HockeyApp so

Charles: I’m going to stop you for a second to just clarify, VSTS for those who are not

Josh: Visual Studio Team Services is the twin of Team Foundation Server which is our product and what we’ve done is we also have a SAAS offering called Visual Studio Team Services so it is, we’re kind of tracking source control, continuous integration, continuous package management. It’s this holistic set of tools that allows you to build the pipeline that we keep preaching. You can build that entire pipeline strictly with Microsoft but each of those verticals can also be replaced of the tool of your choice. We don’t force you to use everything together, we encourage you to because we’ve done all the integration for you but if you only want our build system or you only want our source control and you already have other tools you want to use, you can continue to use those but that’s VSTS and TFS. Thanks for stopping us on those acronyms, We throw them around quite a bit but I think you were going towards release management.

Josh: Yeah. I think the exciting part that happens for me as kind of a developer is really that it’s kind of setup and then forget. The dream scenario there is that I really get it to the place where I set up a VSTS CI build and the CI build is triggered automatically on basically check-in to get. I just check in the master and it automatically builds my master bridge and since HockeyApp integrates there as well, not only do I get like a dev build that comes out of my master branch on every commit, but it’s also deployed actually out to every single like kind of stuff. This is like a nightly build on steroids like every single commit automatically gets built, automatically gets deployed to that and if you’re using like HockeyApp through the automatic upgrade, then everybody dev build will get like a notification on their phone instantaneously to get the latest build.

The speed of that collapsing down to just like a few minutes to do your build where it’s actually on devices is really powerful. And then basically you can take that where now all you have to think about as a developer is it’s just like, “Oh, I’m just deving code, one of my features done. I just integrated to master.” That’s it. I don’t have to think about anything else. You can actually start to extend that to even more of the process moving forward. Now, let’s say we have that dev build like every time we integrate and stuff like that. we got people in our company that want to see this build and stuff like that so let’s say we do something on a lower frequency like maybe they want to get an update every week or something like that. We just set up another build and every single week, Friday night comes along, it just pulls the whatever the full master is branches out to another branch sends that out and they get an update so they can actually keep an eye. By being automated, it takes away all that stuff.

No matter status email updates of like what have we delivered this week. dev team just gets the right code check in to the source control look where they’re happy instead like they commit task come up easy to release stuff like that. Just take over who want to see what happened to that build, your marketing guy wants to be like, “Once that new feature come in, you can just log in on Monday and get the latest build that happens.”

Jaim: That’s usually valuable because everyone supports– the company CEO comes in like, “Give me the build right now. What are you working on? Stop what you’re doing. Make me that build.”

Josh: Just go to the URL and you can get away from that. It’s not just within the company you can send this all the way to like production. Instead you say you’re doing something like GitFlow and so you have your master branch but then when you want to do a release you branch off of that to sub-branch like a release branch. With the VSTS tools and stuff like that, you can actually set up that it automatically triggers on the VSTS build, produces a build, releases that built to HockeyApp. This time maybe you call it gold master like this is gold master build that we’re about to release to production for taking advantage of the release management features of the VSTS then you can actually put gates and checks on there. Maybe you’d like to,  maybe you’re CEO actually like to try the app and assure that it’s exactly what he want before they release out to the companies.

You can have your CEO be signed as an approver so that he can go in, he can install the app, he can take the gold master, put it on his phone, make sure that everything is right and automatically click. It’s one click so all of the automation of the test, the log in into the apple portal, it can all be automated away so that you only have to focus on the things that matter which is like what is your app doing and when do you want to deploy instead of kind of constantly dealing with this automation tasks. It turns something that usually is like an ongoing tax for a dev team in an organization into a one time cost to get things set up but after that it’s all automated and as a developer, all I do is check in code when I want to do a release, all I do is walk off a branch to an early release branch and mix the process way, way faster and much easier.

Jaim: How much control do you have over what source control events trigger or built?

Donovan: Right now you can control which branches you watch. If you have multiple branches, you can say, “I only want to watch master,” so if people are going crazy and all of their feature branches, I don’t want to hear about that. I don’t want those building–they can send what they call private builds if they wanted to but we don’t want those guys just creating noise and just creating dozens and dozens of releases. I can say, “I want you to watch master,” and that scenario will get you, normally you have a poor request workflow which is I’m going to submit a pool request into masters saying, “Hey, these are changes I think go into master, these are the people who have to review it, when they say everything’s good, that code finally gets merged into master and then this process that Joshua just talked about would kick off and that code will start going.

We also have the ability–he was talking about approvers, I’m just going to double click on that a little bit. Approvers are a list of individuals or groups that have to give a thumbs up before our pipeline will continue. Again, to control how many x you get deployed to go anywhere where someone can see them, we could all be approvers and if I don’t like it, I’ ll reject it and the whole pipeline will just stop and say, “Okay, this build is not going any further because Donovan said it wasn’t okay and I can give a comment on why I don’t think that this should go any further, I might found a critical bug that you guys missed or–I know, I’m pretty good. That’s all I got to say. But no, there’s any reason and it’s really nice how we can have that complete control over what gets released and what doesn’t gets released. I think the approval is one of the bigger features of release management that gives you complete control as much as I want to remove every human being I can from my devops pipeline.

There’s simply a point where you need to get that approval and what’s really nice is that again, it’s like you set it up and you forget it. The tools going to make sure that it gets the approval and ought it that it got it and then move it on. I don’t have to go back in and say, “Did everyone say thumbs up? Okay now let me click another button.” No, it’ll go solicit all the approvals sending out alerts and then you come and you say yes or no and then when everyone that I’ve said needs to say yes it’ll go start rushing through. It’s such a great system.

Charles: I really like the stories.

Donovan: Sure.

Charles: I’m hoping to hear about some company that have like a nightmare process. It’s like the CEO sister in law’s cousin was pushing the globe button and FTP-ing it up to somewhere that eventually it wound up to the app store. Then they start using a system like this and they’re like, “Oh, the heaven’s opened.” I mean, do you have stories like that?

Donovan: They played some of them in the KeyNote today. If you go back and watch the KeyNote when Scott talked about the acquisition of Xamarin, it kind of talked about that story of how hard our life was before we had this unified way of writing the application just focusing on the value and not focusing on the low-level technology and being able to reach all three of them. I don’t have an exact anecdote from my customer that’s told me that but it’s the same transformation that they’re having. I have it from non-mobile teams that go through the same transformation. Where it’s just–my god, we have human intervention every time it was firefighting, it was all hands on deck, where we stay up into the middle of the night try to deploy our app because there’s always a train-wreck. All of that goes away when you start implementing devops and it’s true for mobile as well.

Charles: Oh totally.

Donovan: It was interesting. It’s like whatever hurts the most, you need to do that over and over and over again because what ends up happening is that if I have to pay that tax everyday, I’m going to fix this. I don’t have to pay the tax once every release. I’m going to dread that day but it’s only one day every six months. I’m like, “Who cares?” It’s a rough day but the rest of the months are great.

Charles: My users will forgive me.

Donovan: Exactly. but if you have to do this every week because your competitors are doing this every week, but if you don’t keep up, especially in the mobile world, there’s 12 guys with the same exact app that you have.

Charles: It’s true.

Donovan: I see them getting updated with five stars, I’m going to go there because I’m sick of waiting on you. You have to speed up but if you speed up just recklessly, you’ll do more damage than good because what you’re pushing out there is going to get rated really low because it’s really buggy. What I’ve noticed too and I’ve run some websites myself, you lose a customer, getting them back is

Charles: It’s hard.

Donovan: Goodness, right? Your competitor has to fail and screw up really bad for them to even re-evaluate you. You need to make sure that you hold on to the customers that you have because once they’re gone, getting them back is like pulling teeth.

Charles: Yeah, and keeping them it’s a lot easier than getting new people too.

Donovan: Absolutely. Keeping who you have is so much easier than even getting them back or trying to get new people to come on. You want to do the devops best practices so that you can accelerate your velocity but not jeopardize your quality in the process. That’s why we spoke so long on testing because you don’t want to speed up crap to a production. You want to deliver value to production. There’s a big difference there and I tell people all the time, devops is not about just copying files to a server, it’s about delivering value to our end customers.

Charles: Oh absolutely.

Donovan: Yeah and sometimes it might just be an simple structure change, sometimes it might be a code change, there’s lots of different ways to deliver value that aren’t specifically your application and that’s what I want my customers to focus on, delivering value. That goes back to another feature of Hockey App. How do I know if I delivered value? Just because I put an application out there then I deliver value, is anyone using that new feature? The only way I can determine-

Charles: Twitter acclaim.

Donovan: Well, the best way to determine is to actually monitor your application. To put custom  telemetry in there. I added this new feature, is anyone clicking on it? We can use the telemetry that you can get from HockeyApp to say yes, we deploy this app and look at how many 50% of our customers are now clicking that button that wasn’t there a week ago but if we add a new button and only 2% are clicking it, there’s a couple of things that we can infer from that, one, are you as horrible and not intuitive or that was a useless investment of our time. but either way I now know, I need to go back in and look at that again. Let’s run another experiment. Maybe we need to change the color, change the background, change the position of the button on the screen, draw more attention to it or it’s just a crap idea and we need to pull that because even when we try that, no one uses it so that’s how I know I delivered value and not just copied a new version of my software to a server.

Charles: That sounds a lot like what Eric Ries has to do with The Lean Startup where you do that experiment and.

Donovan: You can do that on–maybe you can do A/B testing, you can do all sorts of different testing on mobile and on normal devops. I was not a mobile developer. I wouldn’t still claim that I’m a mobile developer. I can get an app built but I wouldn’t claim myself a mobile developer. I am a devops guy and what I do is I take these best practices and I find every language in which I can apply those, every technology which I can apply those. They fit perfectly in the mobile world because the goal of everyone of us that’s writing software is to get that software to our customers as fast as we possibly can regardless of what platform we’re writing on. I just apply this practices to every piece of software I get my hands on.

Charles: Well, the other thing is we’re talking essentially about the science of human behavior. Be it our users and how they cut crap on their phone.

Donovan: Sure.

Charles: Or our teams and how they interact with our systems and what hurts and what doesn’t and how we deal with that in devops. If you can focus on that and you configure this things out by having good process surrounded, by having good analytics that help you get there, then you can start to make good decisions with good information because you know that people essentially are going to do what’s easy and least painful. You take away the pain, you make it easier for them to do the right thing, you make it easier for them to the thing that makes you money and whatever. That’s the pay-off. What you’re saying, the things that you do that work, they work everywhere because people are generally the same everywhere.

Donovan: It’s always the same.

Charles: Yep, exactly.

Jaim: Absolutely. You mentioned A/B testing, how can we add A/B testing to our apps?

Donovan: There’s several ways to do that. One of them is feature flags, is one way of being able to do that where you’re able to distribute an application and only turn on that particular feature for a subset of all your users and see how their experiences with that app versus other. We actually have a partner called LaunchDarkly that has an extension for Visual Studio Team Services. So Visual Studio Team Services is an ecosystem to where benders could actually add more value to it, feature flags for a lot of people, even for me at first I thought it was like this mythical technique. It’s an if statement.

Let’s just break it down. It’s an if statement that the trick is where is the decision story, the decision is stored in the database and then based on that decision, I either show you this feature and I show him a different feature and I can look at the telemetry that I get back and say, “Well, did he finish shopping when he saw that one experience? Well, he always seem to finish shopping and have a lot of money in his shopping cart when he had this experience that showed him discounts or free shipping if you add one more item and we didn’t showed it to you.” so that’s how we’re able to do A/B testing and say, “Wow, that seems to be effective because the majority of people that we showed this experience to ended up spending a lot more money that we showed your experience to.” It’s about managing the distribution of those features and being able to correlate who I gave what to make sure that when I’m looking at the data, it’s not mixed up.

There’s an example where even at Microsoft, we took one of these experiments, we did some A/B testing but the numbers are way off. What we didn’t realize is two people are running experiments at the same time and they all came together and they were skewing our number. There’s a lot of responsibility that comes to feature flagging and A/B testing to make sure that what the numbers and the values that you’re getting back, you can interpret those correctly but that’s one example.

Charles: We had a discussion with Neal Ford from Thought Works and he talked a lot–I mean he’s–the main focus of the episode was about just doing all your development master branch and then using all your feature flags to manage who saw what but he talked a lot about managing feature flags so if our listeners are interested about that, they could definitely check that out as well.

Donovan: They are powerful techniques but—I have a show devops Interviews, Aaron Bjork and I made him talk about feature flags because so many people–and it was amazing because I said, “Okay, they’re not a silver bullet, tell me that bad side of feature flags.” Did you talk about the fact that eventually you have to go back and clean those feature flags up?

Charles: Yes, that was one of the big ones.

Donovan: Exactly. This can’t be a silver bullet because every time I add a feature flag, the cyclomatic complexity is going up. When do I decide that I go back to clean that up because there’s also a good way to remediate an issue. I put out a feature and it’s wreaking havoc. I can just turn it off for everyone and I don’t have to then rush up a new fix. It’s a great way to only roll forward. A lot of our customers are concerned with, “Okay, the deployment went bad, how do I roll back?” Even in the VSTS team we don’t, we roll forward. What we do is we use feature flags and if something really haywire goes, we can just turn that feature off, give us time to remediate and then push out to fix it the proper way and not sitting there trying to pull our hair out.

Jaim: You can have something that’s actually released to customers in the app store and turn off the feature so that you you run your tests, see what’s working better and say, “Okay, this one wins,” turn it off.

Donovan: Then everyone gets the other feature and then eventually you would go back and clean your code up to where the previous feature of the other experiment gets deleted completely.

Charles: Even if you don’t have the backend for your mobile app, you can store those somewhere where you can go check them.

Donovan: Absolutely.

Charles: What feature flag should I have on that’s a quick call, they don’t use a ton of data, and it’s turned off?

Donovan: In LaunchDarkly, is a service that provides that feature flagging for you so you don’t have to worry about where are my flags stores, to who gets to see the them, it manages all that magic for you, you just make an API call saying is this flag on or not and if it is, your app performs one way and if it’s not, it performs another, absolutely.

Charles: Like you said, I mean, there are some gotchas, one of the things that I think Neal pointed out was like you said you get like 30, 40, 50 feature flags, pulling them out then it’s like why is this feature showing up, why isn’t this new thing showing up when it should.

Donovan: I just want everyone to know that. I think they are fantastic technique but they come with across like everything else.

Charles: Everything has a tradeoff.

Donovan: Not exactly. You don’t get anything for free.

Jaim: I was scarred for a year with C++. I rejected feature flags for a long time, until that episode. If we clean it up, we’re okay?

Donovan: Yeah, it’s the maintenance thing there.

Josh: Since you’re saying you talk about feature flags is actually even more advanced technique that you can use in mobile which is if you’re using one of the JavaScript-based frameworks like Cordova or ReactNative they you use JavaScript code which is interpretive, it’s not compiled. You can actually change the code on the fly and so my stuff actually has a tool called Code Push that basically works service. Code Push is almost like A/B testing on steroids because rather than just deploying an actual feature flag, you’re actually deploying the feature itself like in an A/B testing manner.

What we can do is we can actually change the dif and the tool automatically goes through, looks at your JavaScript code, diffs it with the existing code that’s actually out there in production, creates a delta package that represents the difference of what that package is, I mean you can selectively apply that to who you want so if you want to roll it out to just this percentage of your customers. You can do this not just to feature flag but to feature that’s already been deployed or we have both packages out there but basically after you’ve gone to the store, you can actually live update that whole feature site or roll out an additional feature or change some code basically all automated like in a real time manner. You could change some code, watch the telemetry come back in real time in production and maybe an hour later actually roll out to the rest.

Jaim: Just don’t tell Apple.

Charles: Apple allows you to do that. The thing that’s interesting about that though is that once you figured out what you want to work and deploy, then you push that up to the app-store as the regular app because you don’t want someone to download your app and have it immediately say, “I have to update.” Other than that, that’s a terrific way to go.

Josh: That’s really exciting especially things like, if you have a bug fixed, there’s no need to wait for that time to go through the attribute process to push out another stuff, you can just change the bug fix, fix it up right away. Definitely something interesting if you’re on the JavaScript persuasion there and mobile development.

Charles: Yeah. I don’t know how I really feel about JavaScript.

Donovan: I’m actually warming up to it as I get older, it’s weird. I used to hate it, I’d loathe it because it was just so free and you can make a function that has a functions of it, and has properties it’s like what. I come from C++ too so that was my lineage is that very fast. It was just like, “Yes, I need a semicolon that just does what I tell it to do.” All of a sudden it’s just like, “Nah, I’ll just do whatever I want. I’ll change my value. I was a string and now I’m a number like no, I just couldn’t. I need a strongly type.Things like TypeScript are helping the JavaScript community quite a bit but at this point, I’m trying to get used to it as often enough. I do more no JS development now than anything which is kind of odd.

Charles: Yeah, we have a pretty big showing of JavaScript. I’m wearing the shirt. Yeah, if people are interested in that, they could definitely go check out JavaScript Jabber. We also have a show on ReactNative so if you want to go check out ReactNative Radio.

Jaim: It’s actually pretty exciting. I think ReactNative community is one of the most exciting communities I think from all these days.

Charles: There’s so much going on, so much potential in there. Anyway we’re kind of getting to the end and we like to do picks. We’re not doing as many interviews as we did at build so I think we’ll do picks too. If you have a pick you want to share Jaim?

Jaim: Alright. I got one. So hotel’s up in Marietta which–not much going on in Marietta, sorry for the people who live there, they love it but we don’t have much hotel, drive a car so I walked around and a place called Pappadeaux, packed to the gills, Sunday night. I had stuffed shrimp wrapped in bacon with Monterey jack cheese and peppers. I can’t remember what they call it. It was amazing. If you have Pappadeaux near you, you want some shrimp and dirty rice, just get so.

Charles: That sounds really good.

Jaim: I’m right by the hotel.

Donovan: Now, I’m hungry. We have those down in Texas too, they’re delicious. They have a Pappasito’s, it’s their Mexican food chain that mimics that one. The food is always fantastic.

Charles: Very nice. I’ve got a couple of picks. The first one is The Lean Startup because we just kind of talked about it here. They walk through a lot of that experimentation, how to think about, and when to build feature and when to just put a button in and see how many people want it and stuff like that. It’s really good. The other pick that I have is Jaim had to go to dinner by himself lasts night I think because I actually have a family in the Atlanta area and so I went to a three-year old’s birthday party. But anyway, I took Lift and it’s so interesting to me how over the last year or two, there have been these services that come up that just makes it easy to get around. Just taking Lift back and forth. I’ve had some terrific folks drive me around. It’s just been a lot of fun and it’s cheaper than a cab and the cars are usually nicer too.

Jaim: Oh, very cool.

Charles: Anyway, I’m definitely going to shout out about Lift.

Jaim: People use other service that I never use. I always Lift. I use the other one sometimes.

Charles: Uber’s good too.

Jaim: Yeah, there you go. I don’t know if we’re allowed to say that.

Charles: It’s all good. Donovan you have some picks for us?

Donovan: Yeah, we’re talking about it earlier. I’m a BMW fanatic. I am obsessed with them.

Charles: Nice cars.

Donovan: Yeah and I have more than any human should. They are a little bit of a burden sometimes but I love them to death. I’m a big fan of air hockey. I happened to be, at one point, ranked as 11 in the world. There’s actually such thing as a professional air hockey so I’ll say that too so people are aware.

Charles: I’ll watch you play. I won’t play against you.

Donovan: Most people don’t anymore. It’s hard to get into a play with somebody.

Charles: What about you Josh?

Josh: My pick is actually, I’m on the HockeyApp team with the HockeyApp product.

Charles: Not the air hockey app? No, no, no, not that one.

Josh: Unlike the entirety of the rest of the HockeyApp team, I actually really love Hockey. I actually played Hockey for 14 years so my pick is the best Hockey team that’s actually out there which will be the Detroit Red Wings. That’s my pick.

Jaim: I’m not sure Chuck is convinced hockey’s a real sport.

Charles: People believe it’s a real sport.

Donovan: There’s defense, it’s a sport. That’s my definition of it. There’s defense.

Josh: You sound like you’re going to ask question?

Jaim: I don’t remember what it was.

Donovan: Alright, thanks for having us.

Josh: Yeah, definitely.

Charles: If people want to find out what you’re working on personally or check out some of these tools?

Donovan: The best way to get a hold of me is Twitter. I check in more often than I check my email. I’m @donovanbrown on Twitter and it’s also a great way to get questions answered because my followers are adamant about devops and you’ll ask me a question, before I can get there, my followers have answered it for you. I will literally get people from the product group on that thread if I have to.

Charles: There’s no replacement for a good community. That’s cool.

Donovan: And I’m trying to build that for us in Microsoft as well.

Josh: I’m on Twitter @joshuawebermsft since Joshua Weber is a popular name.

Donovan: Weber with one B.

Josh: You can reach out to me on Twitter. I’m happy to talk to you.

Charles: Alright, well, thank you for coming. This has been really fun.

Donovan: My pleasure. Thanks a lot.