232 JSJ GunDB and Databases with Mark Nadal

00:00 3858
Download MP3

03:45 What makes the Gun database engine special07:00 Defining a database12:58 The CAP Theorem22:56 What Graphs are and how they function (circular references)30:32 Gun and rotational disk systems32:08 Gun’s optimizations for performance in ensuing versions39:55 The prevalence of open source companies42:45 Further discussing the CAP Theorem and its nuances50:33 Gun’s purpose and design52:13 What a Firebase is54:22 How to get started with Gun - Visit Gun Tutorial,  Gun's Github Page, andGun Node Module


“I think the database should bend to your application’s demands, rather than you having to bend to the database’s demands.” –Mark Nadal“…The protocol that GUN defines is something that can be implemented in any language. Because GUN is in the language, you don’t have the context which latency of having to make an HTTP call or socket request…” –AJ O’Neill“Let’s demystify the black magic of CAP.” –Mark Nadal


Dan North’s Deliberate Learning Video8Tracks Internet RadioPokemon Indigo League on NetflixDaplie Personal CloudYoung Frankenstein MovieMystic Vale Card GameJS Remote ConferenceReact Remote ConferenceFarm Heroes Super Saga Game App


Charles:        Hey everybody and welcome to Episode 232 of the JavaScript Jabber show, this week on our panel we have AJ O’Neal.AJ:                Yow, yow, yow coming at you live from South Provo.Charles:        Aimee Knight.Aimee:          Hello.Charles:        Joe Eames.Joe:               Hey everybody.Charles:        I’m Charles Max Wood from Devchat.tv. Quick shout out about React Remote Conf., go check that one out. We also have special guest this week and that is Mark Nadal.Mark:            Hello, thanks for having me on this show.Charles:        No problem, you want to give us a brief introduction to who you are?Mark:            Yeah. I’m the author and founder behind the GUN, it’s a database, the quickest way to summarize it’s like an open source Firebase but it can do quite a bit more and that’s what we’ll go through the talk. My background is what I want to start with which is I was originally working on a web design tool and in the middle of the night, everything crashed and blew up.                     I was wondering, “Why on Earth is my server crashing?” I quickly had to scramble and figure out what was wrong, my node JS app server was fine, my front-end code was fine, the cd in was fine but the database was down. When the database was down, it was the bottleneck to all the rest of my problems.                     Since then, I get in pain and frustration and I decided to try and solve that myself. That’s how I got into databases. We launched GUN and I’ve raised a seed ground for it and it then gets about 2,000 downloads on NPM every month. That was a quick summary.Charles:        If you raise the seed round, I gotta get to the business first, I guess. Is there some money making mechanism behind GUN then or is it just funded open source awesome?Mark:            It’s completely open source and fun and awesome, that’s how we classify it. Of course we have support plans, that standard route, that like MongoDB or MySQL goes but what differentiates us is we’re MIT, Apache 2.0 license so you can literally do anything you want with our software. It’s extraordinary open sourced to a fault and we then hope people wanna pay us for our services.Charles:        Very nice.Joe:               You could also try making money through your path controlling as well to support the whole workout.Mark:            That’s my favorite game.AJ:                Actually, the apache 2.0 license doesn’t support path controlling. Somebody here has actually read the license that people use. Oh my goodness.Charles:        Well, if it’s MIT license then it basically says you can do whatever you want with it, commercial or not.AJ:                And control, but with the apache 2.0 you cannot.Joe:            How does it work when we have a triple license?Charles:        I was wondering about that.AJ:                People just pick the one that they want, they decide under which license they want to use it. Somebody that wants to use it in GPL code will use it with either the zlib or the MIT license, but somebody that wants to use it in a commercial product will probably use it under the apache 2.0 license so that you can’t trademark or not trademark path control though. You can still trademark troll even with the apache 2.0, that’s the good thing. You can always trademark troll, there is no license that can help you there.Charles:        So mark, I guess my question is why in the world does the world need another database engine?Mark:            Going back to the story of how my server crashed, first off, most of the mainstream databases out there, what’s called Master/Slave, they’re centralized. If your master database goes down, everything topples with it.Joe:               Does that include the databases that we all are used to and no one loved like, does MongoDB worked that way?Mark:            Yes. Postgres, MongoDB, MySQL, there’s very few that are not that way. For instance, Cassandra, React and Couch, maybe Orient as well. Those are more distributed databases. You have to be careful with the word distributed because distributed is a more general category than decentralized. Peer to peer decentralized is referencing things specifically like BitTorrent. It’s very, very hard to take down BitTorrent for good or for bad.                     Distributed doesn’t necessarily mean it has to be peer to peer, it might be distributed across many machines but it’s still potentially centralized so I think RethinkDB for instance is distributed but not decentralized.                     Distributed systems are less fault tolerant than decentralized systems. If I’m speaking too much jargon, I’d always love to pull back and talk about these things from a more general scale because part of what we’re trying to do with GUN, another interesting, frightening fact, is that it’s written in JavaScript. Cue everybody laughing.Charles:        No.AJ:                The good news is Jamison is not here today so you can use all the jargon you want and no one will stop you.Aimee:          Oh, I’ll stop you.Mark:            Part of the movement that we’re trying to establish is that if you are developing anything or you’re a newbie, when you had a bug or you’re trying to debug something, if there is black magic in your system, you’re just not going to be able to figure out what’s going on.                     I think it’s really important for people to understand everything that’s happening in their system from the top to the bottom. Now, that doesn’t necessarily mean being an expert in a full stack but it means having clear, transparent, composable pieces that you can put together.                     GUN does not try and be a magic modelist that solves anything and everything. It instead takes the node JS or UNIX philosophy. It has one job that is responsible at and that piece that it focuses on is design such that you can compose that into many larger pieces, that it can be emergent.                     Quick example with the decentralized versus distributed, you can create federated systems on top of peer to peer systems. You can then build centralized systems on top of federated systems but if you start with the centralized system, you can’t go back down.                     At the core, at the bottom, peer to peer systems are more composable or more emergent. Then when you put those pieces together, it’s up to the application developer to know what they’re trying to do versus the database.Joe:               There’s like ten terms which I think might be good to define, could we just start off with database?Aimee:          I have some questions too then after we go over the terms.Mark:            Sure. A database, you can almost think of it as simply being a variable. Okay, let’s start with that. In JavaScript, you do var foo = BLAH. BLAH is a value that is stored into that variable, that’s data. Now, generally speaking, people think of database as a thing much more than just that. They think of it as being collections of variables, sets of data, and they also typically think of the data persisting to disk versus just being in memory and transit. A database, I’d like to generically define as something that stores data so you can retrieve it later on in the future.Charles:        I think Joe is being a smart aleck.Joe:               This actually great like, I love this definition and I’m really glad, I was being a little bit of a smart aleck but I’m actually really proud of it because this is actually a great definition because we live in a day and age when we’re no longer dealing with just relational databases where everybody says oh you want to understand a database. Well, you know excel. Just imagine that with multiple tabs and that’s a database and there are some potentially correlation between the tabs. We don’t live in that day and age like that anymore, that’s the more exact definition, this is actually been great, I really liked this.Charles:        One of the thing to keep in mind though is if you’re thinking about this as a variable or collection of variables, it’s generally a global system. It goes in fact to be on global if you’re thinking about a web system or something where it actually spans across all the apps that connect to that database so it’s super global.Mark:            In that global setting, you can almost think of a key value database like Redis. Those are primarily in memory but they will occasionally flush to disk. GUN at its core is in memory but it has adaptors that are built into it by default to flush to disk so that way it does store long term the data.                     But then, yes, talking about the global scale that those keys are accessible by any different machines. Of course, how you deal with security authorizations is a separate matter but many people can access that data and that’s a great point to bring up another fancy word called concurrency.                     Concurrency is a really nasty problem in computer science when you are dealing with multiple machines reading and writing the same thing. So much that there’s cliché quote that there is two hard things in computer science, cache invalidation and naming things and then a joke off by one errors since. They said two things for those actually, three things, off by one. It’s pretty cliché, I’m sure most people have heard that before. Cache invalidation lines up being the core of concurrency control because if multiple people are editing the same thing at the same time, they have that thing on their computer currently, that’s their local cache of it.                     If they’re editing it and somebody else has edited it, the other person’s edit on another machine needs to invalidate or update the cache of the other machines. That’s just a very general view of concurrency. You can think about it as a bunch of chefs at a restaurant trying to cook the same meal while communicating to each other.Aimee:          The one thing that I wanted to make sure that we went over before we went much further, can you explain the architecture for traditional databases that most people are probably used to like PostgreSQL, MongoDB, so those you have said are like Master/Slave architecture but GUN is Master/Master architecture. Can you talk about the differences there?Joe:               While talking about that, can you also talk about federated versus non federated as well?Mark:            Sure. GUN is real time, offline first, decentralized and a graph database. I’m going to use those four terms and I can explain them and compare them to other databases. For instance, MySQL is centralized, not decentralized, so the words Master/Master or Master Lists or Multi Master or Peer to Peer or decentralized and to some degree distributed, all kind of mean the same thing. There’s lots of words there.                     Compared to centralized or Master/Slave, I think there’s fewer terms around that jargon. MongoDB, PostgreSQL, MySQL, most mainstream databases that people are using are centralized or master databases. GUN is not, it is peer to peer.                     The big difference with those things is how you deal with fault tolerance. A master system is more likely to fail but a master list or a decentralized system like BitTorrent is much more difficult to fail. It is fault tolerant. However, when you have peer to peer systems, when you have Master/Master, Multi Master, decentralized, distributed, I’m probably confusing people just by repeating the words. I’m sorry.Aimee:          Yes.Mark:            I’ll just classify decentralized compared to centralized. When you have decentralized systems, they also might have slightly out of date data as they’re syncing between them. Einstein has told us in physics it takes time for information to travel. When two machines are trying to talk to each other, there’s gonna be some latency behind them. You might have heard more jargon in here, the famous CAP theorem that people talk about.Aimee:          I was going to say, yes, let’s get into that.AJ:                Yeah, we’ve all heard about it, nobody understands it, except for you.Mark:            Maybe I should just use concrete examples of MongoDB and MySQL versus the decentralize. MySQL should be what’s called strongly consistent, that the C in CAP and the reason why it can be strongly consistent is because it’s centralized, all the data stored in one place. It can give you the correct view of data at any given time. However, because all the data is in one place, don’t put all your eggs in the same basket. If that basket blows up, it’s also not as fault tolerant compared to GUN or these other databases that are decentralized.                     But the trade-off that you get is that if you want fault tolerance, data’s going to take a little bit more time to sync. That’s the next jargon I wanna move on to, which is GUN is real time versus most other databases being a pulling based architecture.                     MongoDB, PostgreSQL, MySQL, you all have to ask them questions for getting data like, “Hey, do you have any new updates? Hey, do you have any new updates? Hey, do you have any new updates?” That gets annoying really fast. While things like Firebase and GUN are real time, they push the data out to you.                     This piece makes it really, really nice because, before we just talked about like, oh what if your data’s out of sync or something like that, what if the data’s been edited on another machine, what if you have a stale copy of the data? If you have a real time system, you’re going to get those updates as fast as possible, as fast as the network can give you those updates.                     It mitigates a lot of the problems, not completely, not mathematically, but it mitigates a lot of the end user problems. That’s why Firebase took off and became this phenomenal success because people are like, “Duh, yeah, your database should be real time. That just make sense.”                     There’s not too many databases that have that yet. GUN, that’s why I introduced this, think of it as being open source Firebase.AJ:                I was gonna say, RethinkDB is also in that category, having real time.Mark:            Kind of. RethinkDB added that on top. Yes, they do have a push based system but the core architecture of their system, from my understanding, at least my discussions with Mike and others, is that the core system is not. That’s a layer that they’ve added on top.AJ:                Because I love them, I want the world to know that they are real time because that is one of their focuses. Despite how they may have evolved, they are a real time database now. Just so people know because I love them too.Aimee:          CouchDB too or no?Mark:            CouchDB is, well no, that’s incorrect because CouchDB with Pouch which is the front end version of Couch has a syncing method. You can also get Couch which is one of those Multi Master or decentralize databases out there. One of the few that actually has offline for support to sync and that syncing is a type of updating to it.AJ:                Couch is still in the model that you ask for data, they have not yet added the real time features, correct? You don’t subscribe to data on Couch like you subscribe to data on Firebase and on RethinkDB and on GUN.Mark:            I haven’t used Couch enough, last time I used Couch was back in 2011 for one week, I was writing a MongoDB driver for node JS and one of my friends said, “No, Couch is the way.” So I switched over to Couch and then I realized it didn’t have dynamic queries yet. They probably do now because it was 2011, a long time ago.                     I haven’t seen Couch since then. However, since MongoDB blew up on me, I realized my friend was right because Couch has that fault tolerant model to it while MongoDB doesn’t.AJ:                It kind of does now.Mark:            With charting and application but those are excuses for fault tolerance, it’s not really actual fault tolerance from at least perspective. Then, you have to coordinate really complicated algorithms like Paxos or Raft and there’s this brilliant guy out there Cal Kings, he goes by  a front line who writes all these really fascinating articles about how he just destroys databases. He does this as his job. Pretty much every single implementation of Raft and Paxos were not correct except for one or two. Even if you used the one that is correct, you still have to set up your entire infrastructure and devops to coordinate for all those things in the first place and you can easily screw that up.                     Anyway, I probably lost a bajillion people in the audience because we’re talking about all this really ethereal stuff. I want to bring it back down to just something that, this was stuff I did not wanna deal with. This is stuff that I think is stupid and dumb and it’s in this alida’s academic jargon and it doesn’t have to be.                     The concepts aren’t actually that difficult if you sit down and think about them and I hope that’s what we get to explore in this chat, that if somebody’s out there getting lost, at some point running to bring this back to front end development because GUN is written in JavaScript, you can use it there like Firebase but it’s just to explain the architecture.                     Last two things, GUN is offline first and anything offline first allows an end user on your app to be going through the subway or some place that they’re disconnected. They can make edits, it’ll save locally to their phone or whatever they’re using and then when they come back online, those updates will sync. You don’t have to worry, this is really nice with the mobile first syncing.                     Only, from my understanding, Pouch does that, although I think Firebase just recently added a feature for it. They’re starting to be this big divide, as you can see, from Firebase and GUN, compared to most other databases like PostgreSQL, MongoDB and then there’s these middle tier people like Couch and Rethink.                     Last category in this, I think it is really exciting, is GUN is a graph database. There is basically only new for J out there on the market that’s usable for web developers that is a graph database. Graphs are really cool because they let you do documents and tables and relations and the key value. You’ve slapped all these things together and you can do that with graphs.                     To summarize, I had problems with certain databases, I liked features from certain other databases, no database out there had the collection of all the features that I wanted and was also open source. A lot of them use like AGPL or GPL or something like that.                     GUN, and I’m biased of course, is to me, the holy grail of databases because it’s all the best features you could possibly imagine slapped together into a single database and then allows you to use it from the browser or JavaScript. Real time so you get data sync like Firebase. It’s graphs, you can do key value, document, relation or whatever. Offline first and fully peer to peer. I’ve rambled enough.Joe:               No, you really haven’t. I guess this is all super fascinating. A thought occurred to me and this is maybe bit of a divergence from where we’ve been talking but Firebase, you talked a lot about Firebase and you just mentioned that quite a bit. I love Firebase, I’ve done a lot with Firebase, really like and even taught some courses on it. That being said, when it comes to working with Firebase, there’s certainly a few things that are fairly difficult. Pretty quickly, you’ll get to a place where, okay, I gotta write the same piece of data in multiple times and multiple places. Firebase doesn’t have the concept to the join which some of the node sequel databases do even ones I have a fairly similar setup to construction of Firebase, they still support the concept of join.                     Personally, I don’t have an opinion as to which one is better to have joins and not have joins but it does add a little bit of friction as you’re putting things together. Does GUN feel the same way as Firebase when you’re writing it, like the amount of architecture you have to do as developer is definitely a big step up from say oh, I'm just building your relational database and that’s super easy, it’s like a nature that segment my data out, cleaned it up, make it dry and boom, I'm good?Mark:            It’s actually a perfect way to introduce graphs. I realized I just threw out the word, I didn’t really explain it so thank you for that. Yeah, it is a huge pain when you’re dealing with databases that can’t store circular references or relations.                     If you have a blog post and there’s the author to the blog post, you want to store that as it’s own string rather than actual pointer to the actual author. This is where it gets really, really exciting because graphs allow you to do that. In JavaScript, if you’ve ever had a JavaScript object that has circular references in it, how many times have people in this podcast ever gotten json.stringify invalid, circular reference error. Have you guys ever written that error?                     Super annoying, right? Because JavaScript by itself, with just regular object, can store circular references, relations and all sorts of different orders. You can have something that’s infinitely deep. We, as front end developers, are used to that, that is convenient, it’s nice especially when you’re dealing with a blog post with an author and then an author who has a list of blog posts that go back to that very blog post or a social network where you have a person, they have a girlfriend or a spouse or something and then the spouse has a spouse that points back to that same person. Those are all circular references, reality is full of these things.AJ:                You would never model those things in a database.Charles:        Are you saying that my spouse, her spouse is me?Mark:            Yes.Joe:               That’s not true on your case, Chuck.Mark:            Ouch. GUN actually allows you to take those JavaScript objects with all those circular references and just save it and then it saved and it will sync across all your computers and you can then read that data back out, the circular references and all. You don’t have to worry about the friction with Firebase or MongoDB or I’m gonna even say with relational databases because you have to set up the stinking join tables or the join, whatever, I don’t use them enough but I just know it’s a pain because you have to have the foreign index keys and junk like that that you have to remember to do. But with a regular JavaScript object, it’s all done for you already, you don’t have to think about that and GUN behaves the same way.                     You store something as circular references, it gets stored and synced to all your machines. You wanna read that stuff back out? You read it back out and you get the data back. That’s why I find it particularly exciting. I love graphs because they get rid of this friction like you were saying.                     The last remark on this is most databases out there force you to have to architect everything that you’re doing, schematize everything you’re doing, to put it into the constraint of the database itself. You have to do the dirty work, but why?                     If all these databases, if there’s so many databases out there with tons of people who know jargon-y things, how come they haven’t invented something yet that allow us to just put data in and get data back out? Is that too hard of a request?                     I think the database should bend to your application’s demands rather than you having to bend to the database’s demands. I want to put front end developers and back end developers back in power with what they’re doing, not as being the slave to their database.Charles:        One thing that I’ve seen though is that systems that have opinions or force you into certain paradigms, when your system or when your assumptions or your problem set match up nicely with those situations, all of a sudden you don’t have to make very many decisions and that’s actually really nice about some of these systems.                     When people say, “Well, you should be able to bend it to your will and make it do anything you want.” I hesitate a little bit to get excited about that capability just because it feels like more work to me because now I have to make decisions on how I want to think about this problem with this system instead of just using it as design and being done with it.Mark:            That is an excellent point. There is a couple things to this. First is going back to the point of emergents like NPM, node JS, UNIX philosophy. If you have a powerful and flexible system underneath, it’s very, very easy to create a wrapper on top that forces people to use that constraint.                     That’s the direction we’re going to go with GUN. Maybe you don’t know what you want and you want some framework key assumptions but you should be able to use a tool that has that built for you and that tool should be able to build on a more solid foundation that pre accepts.                     Just because the lower level system is more emergent or more flexible doesn’t mean you can’t put constraints on it, but those constraints are then up to the framework developers who are making those good decisions.The second point is it often times is hard to know what the good decision is to make in the first place. If you’re just starting an app, you probably don’t know what you’re going to do with it or you might not be experienced enough to know which framework with what assumptions you should use.                     I think it’s still very, very important to be able to start at a rapid prototyping level and build things up. But then once you understand what you’re doing, you should never push code into production without having a forcible schema around it because that schema’s gonna protect you from a lot of things but those are pieces that come later and are built on top, not necessarily the underlying system.                     I absolutely agree with you, you have a great point and I don’t want to deflect from it because it is important. However, it oddly requires experience or it requires knowing what you’re doing in the first place which I guess is experience. It either requires experience or requires a system underneath to build the constraints on top of.AJ:                Alright. Chuck, I wanna tell you a little something that, I wanna know if this helps you understand why this is interesting. In a traditional sql database, you have tables that are built and the way that they’re optimized is around the way that a traditional, rotating disk system works.                     One of the stories behind RethinkDB is actually that they were rethinking a database as if rotational disk wasn’t what we’re building a database on and that’s how they got their start. GUN is built in a similar way where it’s not built with the concept of this data has to go on a rotational drive. It’s built with the concept that this data has to be stored in some permanent medium. Whereas a sequel table has all these IDs and it has indexes and the tables join together looking at can I disk seek ahead this number of bytes knowing that each row in the tables is approximately this long and then find the column that is represented on disk as having that data, it’s all about this disk seeking action.                     GUN is built with the idea that the database actually operates primarily in memory. It’s primarily like Redis works, it’s primarily operating out of the cache and every time you retrieve an object, it’s doing all of those joins and it’s not as optimized for a disk system, a rotating disk system, as sequel is but it is optimized for a highly available system that has multiple nodes in it. Does that make sense? And Mark, am I correct?Mark:            Yeah. Although I do want to comment about optimizations performance, that’s stuff we’ve been working really, really hard on this next release and I’d like to proudly state that the upcoming release should be able to do about 30,000,000 reads per second and then over 6,000 writes per second in a browser, compared to a lot of other databases like Redis.                     I think Redis read for the same type of tests, GUN is about 50 times faster than Redis for the same type of test that I looked up. And then I think Cassandra writes at about 6,000 writes per second. Performance, sure it’s not optimized for disk and spinning disk head, that’s true but most of your performance and optimization comes from it being in memory. IO is not particularly interesting.AJ:                To that, I want to bring out something else for people to wrap their heads around why this is cool. You might think well here’s Mark, a fairly small team of developers that are creating their own database for this supposed to rival like these huge databases that have lots of developers.                     Why would GUN be interesting if other people that are probably smarter, that have more resources are solving this problem in a different way. I think the most unique thing about GUN is that I see it more as a protocol than as a database because as a plugin architecture, so that you can choose different types of storage like you can choose to store in Amazon a back storage or you could choose local disk storage or you could choose different types of storage as the permanent back end but the GUN itself is all in process. It’s meant to be used with JavaScript applications, and is there the still the Android port, that Java port happening or is that on pause for now?Mark:            I think the person who is working on it wound up getting a job and stopped contributing to the open source community.AJ:                The point is the protocol that GUN defines is something that can be implemented in any language. Because GUN is in the language, you don’t have the context which latency of having to make an HTTP call or socket request etc. etc. The only time that needs to happen is when it’s communicating between nodes on the network or between processes and a cluster.Mark:            It is a really, really good point. The core of GUN is more how do we sync data? And that’s pretty much just a protocol. I’ve implemented it first in JavaScript because it’s the easiest, from my perspective, to use. But it generally nails it on head there. Thank you for that, AJ.AJ:                Yeah. It’s really the protocol in the API that I see is being what makes GUN special and why it’s something that can be implemented by a small team and can still be really effective.Mark:            I want to use that as a launch vehicle to encourage others that yeah there are lots of really smart people out there solving really hard problems, but often times it takes a newcomer to come into the industry, into the field with a fresh perspective.                     If you are a newbie and you’re learning stuff and you’re having problems, maybe the reason why you’re having problems isn’t because you’re stupid but because everybody else’s is doing wrong. I’m not saying everybody else is wrong but you have unique insights when you encounter struggles.                     If you capitalize on those unique insights and try to build those things out like I did, I went from being a basement dwelling, just ran JavaScript developer, I had no interesting backgrounds or credentials to now having investment in our company, I’ve been giving talks around the world, and we’ve hit like the top 4% of NPM’s ecosystem.                     I want to encourage others, I’m not necessarily that smart elite person either. Maybe I sound like it after rambling off a bunch of words and jargon, but I really wanna encourage people that they should tackle ambitious problems and solve them with an outsider mentality. Because sometimes, not always, you come up with something different, unique and that’s a lot more viable, or sometimes you reinvent the wheel, but hey, at least now you have a master ship, a craftsmanship of understanding how wheels work and that’s valuable in it of itself.                     But, sometimes, with that outsider perspective, you also create something wonderful and new. I just want to encourage everybody out there, don’t let the elite or the jargon or haters out there get you down because part of the struggles you might be having is that you have a good insight, you have an outsider’s perspective and you should capitalize on that.On that subject, there’s off to this idea that it’s really important to dismantle the elitism that’s in any particular industry and to try and democratize that information or that talent or that skills to the masses. We’re only able to make progress as humanity if we’re willing to take the stab ourselves. To make something so open source, so free, so vulnerable, that other people are able to get value from it. That’s the message in the movement I really wanna push. If we let a bunch of black magic and academic jargon confuse us, part of the reason why there is all these jargon words is because the people who are building and creating the systems themselves don’t have a complete, formal understanding of it. Even the CAP theorem, which has been mathematically proven, has been critiqued for having nuances to it. Part of the reason, if you ever hear anybody in any industry using a bunch of buzzwords and jargon and stuff like that, it’s probably because they’re trying to cover their own butts. They’re probably not able to explain the systems themselves and they wanna mask or hide the fact that there are certain flaws or bugs they don’t yet understand. That’s again wanting a shout out to Cal Kings because he came into the database industry and actually looked at what people said, all the black magic that they claimed, and then tested it and found out most of the vendors didn’t stand up to their own claims.                     Just have as a warning notice when you hear jargon all over the place, that’s a red flag, the person themselves potentially doesn’t even know what on Earth is going on. That’s why I think it’s really important to dismantle the elitism and democratize the masses and why open source becomes so critical, because the more people that are there to freely look at the problem, the more collaboration we have, then the faster and better problems will be solved rather than locking everything into a jail cell for a few “smart” people to solve. Everybody can be smart if they try hard enough.Charles:        It’s a good thing the whole world doesn’t work that way, our politicians don’t speak in buzz words and sound bites.Mark:            Oh my, this just went political which I think brings up another interesting factor, that the politics of the web. A couple interesting things is that most companies out there, that are web companies are based on open source foundations. Look at how widespread WordPress and JQuery really are.                     The web, the World Wide Web is built on open sourced ideas and built on decentralized idea. Let’s look at the politics around JavaScript lately, ES2016, I don’t really wanna go into it because people are very opinionated. Even the question of whether a JavaScript is a joke and we need to keep on building it. But look at how many beautiful, grand things have been built upon a seemingly weird, strange, broken, distributed system. The foundation to the web is beautiful even if it looks ugly.                     When people are open about revealing their secrets, like people have done with WordPress and JQuery, entire ecosystems of the entire internet flourish. Compare that to any other industry where magicians don’t reveal their secrets, where everything is close sourced and proprietary. Those industries don’t flourish as much and that’s because open source, truly open source, allows for a worldwide community to come together and collaborate and build better and beautiful things.                     Sure, it might have quirks to it but it allows so much progress versus being locked and chained. That’s just other bits of encourage I wanna push out there and why it’s so important to have open source code and what we’re doing as being a movement pushing that.Aimee:          One thing, as somebody slightly newer, as I was learning about databases and stuff at my first job, CAP came up so I was going to see if we could discuss exactly what that means. It seems like traditionally, with my understanding, you can only have two. But like were just saying prior, I feel like you have a lot really good points because from everything that I’ve read, there is pretty strong opinions as to which two you should pick but then there is, as you push people on it, people will say, “Well, there is so much grey areas.” Maybe you could talk about how GUN approaches that and if we can actually talk about what the CAP theorem is.Mark:            Great question. I wanna strongly encourage people to not use CAP theorem in a fuzzy way because I think that’s leads to a lot of more black magic and bad interest out there. Yes, there are some nuances to it but I think it’s really, really important that we keep hard and fast established idea that you must choose two out of three because that keeps the conversation clearer and that’s what we’re trying to do. Not to confuse people more but to keep it clear. Let’s demystify the black magic of CAP and use concrete examples. GUN is AP where AP means highly available (A) and the P means partition resilient. AP systems, aka GUN, should not be used for banking or accounting. You can use GUN for a lot of things but there is the author of GUN itself telling you, GUN would be a bad solution for banking.The reason why banking is important is because you want a strongly consistent view of how much money people have in their bank account. The reason why that’s important is so that they don’t double spend. They don’t go out and purchase two things at once. If I go to the 711, let’s say I only have $2 in my bank account, I go to the 711 and I purchased a $2 slurpee and then I quickly run across the street to some other restaurant, let’s say Denny's, because I know it’s going to take time for the transaction at 711 to go to my bank account. I quickly run over to Denny's and I purchase a $2 ice cream cone or something like that.                     I have now spent the same $2 twice because it took time for the bank account to get the updates. If you have a strongly consistent database, a centralized database, that says, “Oh no, you’re not allowed to make that purchase until we check what the current value is in your banking account and then authorize it,” then that’s fantastic. You don’t have that problem.Charles:        Can you imagine that online banking without the consistency, you either have $12 or $20 in your account.Joe:               What are you talking about? You tell me that all the time, I have an actual balance and then available balance. Not to be like a real jerk about this but have you ever heard of what it was like when there were just checks and no computer, no online banking. We took advantage of it, we floated checks around it, I have five checking accounts open with no money in it but it looks like there is money, it was great, I loved it.Mark:            This is where the nuance comes in because I think you’re actually, absolutely right. The fact is that even though banking is the one industry that should use strongly consistent databases, guess what, they don’t because they value the customer being able to withdraw money. And guess what, if the customer doubles withdraws or something like that, they know who did it, they’re able to penalize your account. I’m going to say from a mathematical and academic perspective, I have to be honest, don’t use GUN for banking.AJ:                You can extrapolate that, maybe it works for the banking system because, at least on the consumers side, when we overspend, we overspend by a few dollars and they come track us down. But if you’re talking about the trading at the New York stock exchange, it’s an entirely different matter.Charles:        If you overspend, they’ll just charge you a bunch of fees, that’s how they make their money.AJ:                Yeah, they like it.Charles:        It’s a disincentive.Mark:            Right.AJ:                I’ve ordered on Amazon, around Christmas time and sent me an email three days later saying, “Oh actually, you’re the customer that didn’t get the last of this item, someone else did.” And airlines do this. I’m going to push back on your point of saying that you shouldn’t use it for financial transactions.                     Even bitcoin which of course is little bit more complicated but it’s totally possible to do it and it’s how it is done. I just don’t want people to get the wrong impression that, “Oh, it’s not good enough for financial, therefore it’s not good enough for me,” because I think that the theory behind it, you just find other implementations. You decide that customer pain is worth x dollars and if it costs you $8 to pay someone to solve a problem that takes an hour to solve in data entry, is that greater or less than the cost of the hardware to make it more consistent?Joe:               The point is there are definitely applications you have to think through, you have the things in your application that said, “Is this the right mix of the CAP theorem?” You don’t wanna get a notification two days later saying, “Oh sorry, we actually didn’t abort the launching of the nuclear missiles when you hit cancel.”AJ:                Consistency is important.Mark:            I do want to say thank you to AJ for bringing this up though because I do think I have to be academically honest and say that, yeah, you’re dealing with maybe a nuclear launch code or something very theoretically and mathematically precise, you’re not going to want to do this. I want to blow the conversation back way further on the CAP theorem to what its actual problem is. The CAP theorem is not a computer science problem, it’s a physics problem. Again, like I mentioned before, in a distributed system, in a peer to peer system, when there’s multiple machine talking to each other, it takes time for information to travel, that is a fundamental limit of physics. Thank you Einstein. When we look at the stars at night time, if you’re fortunate enough to be in the city that you can actually see the stars, what we are seeing is an eventually consistent state of the universe around us as it takes hundreds and millions of light years for that starlight to reach us. When we look out, the star that we’re seeing might no longer exist. The problem is a physics problem. The universe itself fundamentally takes time for information to transfer. It is fundamentally and eventually consistent system. When we try and build databases that are strongly consistent, that pretend to have a global or neptunium view of the universe, we are literally fighting with the physics of the system and that’s why there winds up being so many problems and failure cases that Cal Kings points out.                     I do think it is better to say, “Hey, okay, our actual problem like AJ said is customer satisfaction, not mathematically, necessarily sound theories.” I’m trying to make sure that you guys know that GUN underneath is, that’s why I’m being academically honest, not to use it for banking, but at the end of the day, customer satisfaction is probably going to be a lot cheaper than having a perfect system.                     Thank you AJ for bringing that up and also to that other point. Yes, if you’re dealing with mathematically precise things like nuclear bombs then you should probably upside more on the strongly consistent side. Aimee, did that answer your question?Aimee:          Yes.Mark:            Wonderful. Any other questions? Follow up on that?Aimee:          None for me.Charles:        Not directly on CAP theorem but I am curious, what kind of applications then can or should you be using GUN for?Mark:            Yeah. Pretty much other than banking but cough cough, what AJ said, that’s a little scary for me to say. GUN is designed for collaborative web apps. That is what I was originally building.Charles:        So for my Google docs or something kind of thing?Mark:            Google docs is a great solution whether that’s Facebook, Google docs, Twitter, Instagram, anything that has a strong web presence and you need to sync data between a bunch of peers who are all online wanting to see your latest Facebook addictions or tweet streams.                     GUN basically provides that out of the box, you in a browser like with Firebase, you just write your app. When somebody writes a tweet, it saves and that save auto syncs it to other people that are on the network.                     For instance if your friend wants to see any updates to your tweets, they’re probably subscribed to your key and then that key can load in a document which is your profile and then your profile can probably have a property on it, which is tweets, which is a table of tweets and that table of tweets have tweet records in them and those tweet records might have references back to your profile or references to other tweets.                     That’s where GUN I think is really ideal, you’re building a browser based game, you’re building a social network, you’re building inventory management, you’re building anything that’s very online and real time driven. We’re also starting to get into a lot of IOT stuff to which goes more into the node JS side than necessarily the browser side.Charles:        You keep comparing it to Firebase and I think of Firebase as more of a back end as a service as opposed to a database.Mark:            Yeah, that is a beautiful point. Because Firebase is Master/Slave, and GUN is Master/Master, when you wind up running GUN in your own infrastructure, it only takes about five minutes to get up and started or less if you know node JS and NPM. Because it is so fault tolerant, you don’t have to really worry about managing your back end. Or even better yet, you can use some of the community GUN servers, don’t trust them for any actual data but they’re just out there for you to prototype and play with.                           You can use the GUN as a back end service with somebody’s community server and then when you are ready to move off of that and run GUN as your own, because the core of GUN is syncing, syncing data, you just sync all the data off of the other server onto your server and now you are running your own GUN server.                     That’s a good point because, yes, Firebase is a back end as a service. GUN’s goal is to make it so easy and so fault tolerant that you just throw up a server and it basically manages itself. There’s been a Heroku, GUN server running for two years, I just handled like hacker news traffic and stuff like that. It’s still running version 0.1 which has a bajillion bugs in it, so I need to update it because if everybody goes running after and loading the server, there might be some problems. It’s been running pretty much fine for the last two years and I haven’t had to think about it. It’s in stark contrast to the MongoDB server that I was running that crushed in the middle of the night, that I had no clue why and has now taking me like two or three years just to figure out what’s going on and then another of couple years to build a solution for it.Charles:        Alright. If people wanna get started with GUN or check it out, fiddle with it, whatever, is there a place to go do that?Mark:            Yes. I think the best place is to do the five minute tutorial, it’s an interactive five minute tutorial right in your browser. Just go togun.js.org/think.html. It’ll allow you to create a cliché to do app and it should only take you 5 minutes or 15 minutes. From there, you can start exploring the rest of the site,gun.js.org is the website.                     If you want to get more information, my github username @mark/gun sogithub.com/mark/gun. We have 2,000 stars, please star us, it really helps, download us on the NPM. If you’re coming from an NPM perspective, there’s a little script, all you have to do is run.                     People can probably do it while they’re listening to me. All you have to do is, npminstallguncdnodemodule/gunnodeexample/hp.js.adad and it’s that simple. NPM install GUN cd, node, modules, GUN, NPM start should work.Charles:        Let’s go ahead and do some picks. Aimee, do you want to start us off with picks?Aimee:          Sure. Sounds good. I found this repo with a bunch of older talks on it. There are couple of talks in there by someone named Dan North who, maybe people know who he is, this was actually the first time that I had heard of him but he had a talk on deliberate learning that, this is a short talk, I don’t know, it’s kind of common sense but it was a good reminder. Basically, he just talks about all these coding errors stuff like that people do. It’s one thing, programming, I just like the point he made because I feel like programming is less about practicing, it’s not just like a memorization thing that you have to do. It involves thinking and learning.                     The talk is just a reminder that really when you’re practicing, because it is important to practice, you’re practicing learning new things, not practicing just doing the same thing over and over again because that’s not what we do in our day to day. That’s my first pick.The second pick, I can’t remember if I picked this or not in the past, I don’t think I have, but if I did forgive me. A lot of people listen to Pandora or Spotify, there’s another thing out there called 8tracks, it’s like the number 8tracks. I feel like it’s something different if you’re sick of Pandora or Spotify and want to try something different which is just like some good music to run to or program to or just have on the background and that’s it for me.Mark:            Those are great points you’ll hear especially on learning.Aimee:          Yeah. Like I said, it’s common sense but I feel like we hear this, so many people saying, “Oh you need to practice, you need to practice.” But what are we really practicing? Because programming is not playing a musical instrument or something where just practicing the same thing.Mark:            Programming is, by definition, solving new problems because it’s already solved, well, then you use that library or that module and you don’t think about it.Aimee:          Yeah. We need to practice learning and exploring and things like that.Charles:        Alright. AJ, what are your picks?AJ:                I’ve got two good things to pick today. One is Netflix is always on top of it. They put Pokémon Indigo League up and I’ve been going through the first hundred episodes or whatever it is because the whole Pokémon Go craze, it just reminds you of your childhood or maybe your adulthood or maybe your nowhood and it’s good just to go back. Of course, there’s a lot of life lesson that you can learn from the Pokémon TV show. Bye bye butterfree, is everything that I ever needed to know about love that I didn’t know that I needed to know about love.                     There’s a couple of other episodes like that and I also love all the little nods they give to other shows that I never caught when I was a kid. All these meme things that they do that you just may not have noticed. Get on Netflix, go watch Pokémon Indigo League, just feel that heart warming sense of your childhood swell inside of you.                     Also, I am going to do a little self-pick, if you got the warm fuzzies when Mark was talking about all that decentralizing the internet and making information and knowledge available to the masses and power to the people and all that stuff, then you’ll probably also be interested in Daplie which is my company. We now have pre orders open for our product which is called Cloud. What we’re doing is taking back the internet. Whereas right now, a lot of you are just now for the first time, starting to get those notices from Microsoft and from Google and from Dropbox and from other Cloud services letting you know that if you don’t pay their ransom, that they’re going to delete your data because you’ve now reached your limit and they want you to pay for the rest of your life.                     We believe that people should be able to own their own data and now that we all have broadband connections and we’re not on dial up anymore, it turns out that you can do that. Among other things, data storage is one of the focuses of our in home product, Cloud. Check out our website and pre order and we also have some special things in store for some of the early bird people. That’s it.Charles:        I’ve had broadband that felt like dial up.AJ:                It’s okay. The point is it’s always on.Charles:        Yeah. Joe, what are your picks?Joe:               Alright. I got two picks. Gene Wilder died, an amazing actor. I wanna pick Gene Wilder, love his shows, many of them, one of them is one of my favorites, Young Frankenstein. If you’ve never seen that Young Frankenstein, take some time right now, not at this exact second since you’re maybe driving in your car but take some time, go watch Young Frankenstein. Awesome show.Joe:               Yup. My second pick is a card game that I played recently called Mystic Vale, it’s a deck building game, it has a very unique mechanic where you start with a card that’s basic and you slowly add things to the card by sliding things into its sleeve. It’s just a really cool mechanic, it is a very fun game.                     It does take almost an hour to basically punch out the things out of the, when it comes in the box because there’s a bunch of card and they all have a film on them, you have to peel the film off each card and it takes forever. It’s a super cool and super fun game so I highly recommend that you check it out, maybe pick you up a copy, Mystic Vale, of course the link’s in the show notes.Charles:        Alright. I’m going to throw a couple of picks out there. First of all, since this is a JavaScript podcast, I have React Remote Conf. coming up. I mentioned that. I’m probably going to be posting the sign up for talks and things for JS Remote Conf. so if you wanna speak or if you want to attend then go check those out, justjsremoteconf.com andreactremoteconf.com.                     Lately, I’ve just been playing a lot of games on my phone and one that I’ve picked up lately is Super Cropsies, it’s based on the farm game kinda like Candy Crush but it’s a little bit different. Anyway, I’m gonna pick that. I’ll put a link to it in the show notes. Those are my picks. Mark, what are your picks?Mark:            I want to pick the people who are listening. I just wanna encourage them again, tackle ambitious problems, solve them with an outsider’s mentality, dismantle elitism and democratize it to the masses, invite others to take credit for your victories, celebrate together as a community, and haters are going to hate so build things you care about. Don’t let trends, peer pressure you and understand what you do. Have a great week until you hear your next JavaScript Jabber.Charles:        Alright. Thanks for coming, we’ll wrap it up. We’ll catch everyone next week.

Sign up for the Newsletter

Join our newsletter and get updates in your inbox. We won’t spam you and we respect your privacy.