Final transcripts (ch 9 and 10) for course.

transcripts/ch10-conclusion/1.txt

@@ -0,0 +1,23 @@
+00:00 There it is, the finish line!
+00:03 That's right, you've made it all the way to the end of this course,
+00:05 I hope you found it super interesting and you've learned a lot,
+00:08 because I believe you now have enough to build production ready applications
+00:13 and deploy them based on MongoDB.
+00:15 So really, the big question you need to be asking yourself is
+00:17 what are you going to build now,
+00:19 you have this amazing new power, this amazing new database,
+00:22 and way of writing data driven applications, what are you going to build?
+00:24 I hope you take what you learned in this course,
+00:26 and you go build something amazing.
+00:28 Now, before you do leave, and you go build that thing,
+00:31 let's talk about a few wrap up details;
+00:33 first of all, make sure you get the materials from the github repository,
+00:36 if you haven't already, go to
+00:38 github.com/mikeyckennedy/mongodb-for-python-developers,
+00:42 the url is there at the bottom, and star this, and consider also forking it
+00:46 so you have a permanent version for yourself.
+00:49 As far as I know, the git materials are entirely finished and published,
+00:54 there is a chance that somebody will find a small bug
+00:57 throughout the course and I'll amend that,
+00:59 so very likely what you see at this github repository is the final materials,
+01:04 it's certainly what you saw me create online during these videos.

transcripts/ch10-conclusion/2.txt

@@ -0,0 +1,18 @@
+00:01 Before we put the wraps on this course
+00:04 let's do a quick lightning review of each chapter that we've covered.
+00:06 We're certainly not going to cover everything that we covered in the chapter,
+00:09 this is just a really quick review, but maybe the main takeaway from each chapter.
+00:13 So we began the course by talking about what is NoSql,
+00:17 and I think there's a little bit of a misunderstanding
+00:20 or maybe multiple definitions of what NoSql means
+00:23 sometimes people say it's not only sql,
+00:26 sometimes you people say it means that there's no sql, the language involved in this.
+00:31 Well what we saw is looking at the history back in 2009,
+00:34 this concept of NoSql came about by a meeting of people
+00:39 working on horizontal scales type of databases,
+00:42 like what trade-offs do they make against relational databases,
+00:45 so that they are more easily horizontally scalable,
+00:48 and basically cluster friendly databases.
+00:50 That world it's not whether or not there's no sequel
+00:53 or there is sequel in the language, it's really about the style of databases
+00:57 and the trade-offs around how they work with that data.

transcripts/ch10-conclusion/3.txt

@@ -0,0 +1,33 @@
+00:01 The MongoDB shell and native query syntax;
+00:03 we saw that the MongoDB shell which you start by typing the word 'mongo'
+00:07 and it just runs the shell, tries to talk to the local one,
+00:10 there's all the different ways to get it to connect to different servers as we've seen.
+00:13 So once it starts you get this little greater than prompt
+00:16 and you write Javascript so we interact with MongoDB at the lowest level
+00:22 in Javascript in a textual way
+00:24 and actually this is converted to bson a binary extended version of json.
+00:29 So here we type something like db so this is the database we have active
+00:33 and book would be the collection name
+00:35 or table if you're still thinking relationally, but the collection name,
+00:38 and we say things like find or count or sort, or things like this
+00:42 and what we give it is this prototypical json object
+00:45 and what we get back are all the things that match the elements of that prototype.
+00:50 So here you can see we got two records back
+00:53 and they both had the same title as the title we indicated here.
+00:56 So it's very much about passing these prototypical json documents,
+01:00 however sometimes we have to do more than just say
+01:04 I want basically equality in my search,
+01:07 I would like to express things like greater than.
+01:09 So this query here that we have written
+01:12 is actually doing a couple of very interesting things,
+01:14 maybe the thing that stands out the most is this greater than operator,
+01:17 so the dollar gte is indicating, the dollar indicates an operator,
+01:20 and gte is the name the greater than or equal to operator,
+01:23 so instead of just saying ratings.value is nine,
+01:26 we're saying I'd like all the ratings where the value is either equal to or greater than nine.
+01:30 The other powerful and interesting thing here is
+01:33 we're actually traversing this hierarchy of the document
+01:36 we're going to find the ratings array which is a list of subdocuments
+01:39 which has a value as an integer,
+01:41 so we're actually reaching down inside that document
+01:44 and we're doing this query with this operator.

transcripts/ch10-conclusion/4.txt

@@ -0,0 +1,63 @@
+00:00 Next step we worked with— PyMongo.
+00:02 So we put our Javascript away, we said all right enough with the Javascript stuff,
+00:05 we're going to write in Python basically for the rest of this course.
+00:08 So the lowest level way to talk to MongoDB from Python is with PyMongo.
+00:13 So let's look at a couple of the crud operations here.
+00:16 We'll start of course by importing the package, import PyMongo,
+00:20 and if you don't have it just pip install it;
+00:22 and then we need to create a Mongo client by passing a connection string,
+00:26 I believe if you actually get a hold of the PyMongo connection
+00:29 you can use it directly, but you should not, because the Mongo client handles
+00:34 reconnects and connection pulling is stuff like that
+00:36 whereas the connection itself wouldn't do those kinds of things.
+00:39 Then if we want to work with the database,
+00:42 we have this sort of interesting highly dynamic api,
+00:46 we go to the client and we just say . (dot) the name of the database
+00:49 so we say client.the_small_bookstore, and we assign that to db
+00:54 so it looks like the rest of the shell stuff that we have been doing,
+00:57 but technically that's optional.
+00:59 This database doesn't even have to exist,
+01:02 we could create the database in this style just by doing our first insert into it.
+01:05 Whether or not it exists, we get all the database
+01:08 and now we can operate on the collections.
+01:11 Let's imagine that in that database there's a collection called books
+01:15 and we want to know how many of them are,
+01:17 we would just say db.books.count
+01:20 and that would actually go there and do this operation.
+01:22 If it happens to be that either the database of the collection doesn't exist,
+01:25 it doesn't crash, you get zero.
+01:27 We could also do a find_one, this line here is notable
+01:31 because in the Javascript api is findOne
+01:34 and they've made a pythonic version here, so find_one
+01:39 just be aware that it's not always a one to one
+01:42 exact verbatim match from the native query syntax over to PyMongo.
+01:46 We can also do an actual search,
+01:50 before we said find_one I basically got the first
+01:54 here we're going to say I want to find a book by isbn, I want to pass it over,
+01:57 here we use Python dictionaries
+01:59 which play the role of those prototypical json objects.
+02:01 We also insert new data, so here we're going to say
+02:06 insert this thing which is a dictionary, it has a title called new book
+02:10 and an isbn of whatever is written there and we get back this result,
+02:15 the result will have this object id in the field inserted _id,
+02:20 we can go requery it and do all sorts of stuff with it.
+02:23 Basically when we say insert one, we get this result
+02:26 which, if it succeeds has the inserted id.
+02:29 Now these are the straightforward crud operations,
+02:31 we can also use our fancy in place operators,
+02:34 so here let's just insert this book, so we see what we get,
+02:36 and we grab a hold of the inserted id,
+02:38 and now suppose we want to add a field called favorited_by,
+02:43 and this is going to be a list, and we want the list to be basically distinct
+02:47 we're adding the ids of the customers or people visiting our site
+02:50 who have favorited in this book, and we'd like to put them in there
+02:54 but there's no reason to have them in there twice,
+02:56 that can cause all sorts of problems.
+02:58 We're going to use the dollar add to set, so we run this,
+03:01 run it again for 1002, and hey we could run it a second time for 1002,
+03:05 and what we'll end up with is an object that looks like this,
+03:08 the two things we inserted, the generated_id
+03:11 and his favorited_by list which has 1001 and 1002.
+03:15 Definitely keep in mind these in place operators
+03:19 because they're very powerful and they leverage some of the special properties
+03:23 of the way MongoDB treats documents atomically.

transcripts/ch10-conclusion/5.txt

@@ -0,0 +1,48 @@
+00:01 Next up was document design.
+00:03 Some of the concepts and ideas of relational databases still apply here,
+00:07 you still are modeling data, you still put it into a database,
+00:10 but many of the techniques fall down,
+00:13 this whole concept of third normal form
+00:15 doesn't make nearly as much sense as it does in a relational database.
+00:18 What more we focus on often is really
+00:21 how do we make relationships either between documents or within documents.
+00:25 We saw the primary question, not the only one, but the most challenging one,
+00:30 the one you have to think most carefully about is to embed or not to embed,
+00:34 and I gave you a few rules or tips to help you guide this decision.
+00:38 One— is the embedded data wanted and you use it 80 percent of the time or more,
+00:44 most of the time when you get that containing document?
+00:48 If that's true, you probably want to embed,
+00:51 if that's false, maybe consider that as a warning sign not to.
+00:54 How often do you want the embedded document without the outer containing document?
+00:59 If often what you really want to get access to is these little inside pieces,
+01:03 there's a lot of overhead and it really kind of complicates the way
+01:07 you access it through your application,
+01:09 if you want to get them most of the time, or frequently, on their own.
+01:13 Is the embedded data abounded set?
+01:16 Remember, these documents can only be sixteen megabytes or larger,
+01:19 the number is way higher than you really want it to be,
+01:22 if this is an unbounded set you're going to continue to add to it,
+01:25 it very easily could outgrow the actual size that you're allowed to store.
+01:28 Really for a performance reason though, is it abounded set and is that set small?
+01:34 Because if you put huge amounts of data in there,
+01:36 you're going to really slow down your read time
+01:38 for these database operations that involve this document.
+01:41 These are the four main rules here,
+01:43 you also want to consider how your application accesses this data,
+01:47 it might be really easy to answer these four questions
+01:50 because there's a very constrained and small set of queries
+01:53 you run against your database;
+01:55 or it could be that you ask all sorts of questions in a highly varied ways
+01:59 in which case it's harder to answer those questions,
+02:02 the more types of queries you have the harder it is to know
+02:05 whether most of the time you want the embedded data for example.
+02:08 The more varied your queries are, the more you'll trend
+02:11 towards third normal form, relational style and less embedding.
+02:15 One of the situations where you have lots of varied queries is
+02:18 if you have this thing called an integration database,
+02:21 which we talked about sort of sharing a database across different applications,
+02:24 versus having one dedicated to a particular application
+02:27 where you can understand these questions very clearly.
+02:30 So when you're designing these documents
+02:33 you want to really think most carefully about do you want to embed this data
+02:36 or create a soft foreign key type of relationship.

transcripts/ch10-conclusion/6.txt

@@ -0,0 +1,72 @@
+00:00 After we talked about document design
+00:03 and we talked about the raw access from PyMongo
+00:05 we said let's take this up a level of abstraction,
+00:08 let's actually build classes and map those over ORM style into MongoDB.
+00:14 We saw a really nice way to do that is with the ODM called MongoEngine.
+00:19 Let's review the main way that we sort of define classes
+00:23 and add constraints and things like that.
+00:25 Over here we are going to create this car object, this is our dealership example
+00:30 and we are going to store the car in the database.
+00:33 The way we create something that MongoEngine can manage
+00:37 in MongoDB as a top level document,
+00:40 is that we're going to derive from mongoengine.document.
+00:43 And then every field is going to be one of these fundamental field types,
+00:46 like StringField, IntField, FloatField and so on.
+00:50 And we can have some of them required, the first three required,
+00:53 we can have some of them with basic default values, like mileage defaults to zero
+00:59 but we can also have interesting functions,
+01:01 for example the vin number is automatically generated
+01:04 and we're based in this on the uuid4 random alphanumeric thing,
+01:08 so what we have here so far is really sort of equivalent
+01:11 to what you might have in a traditional relational database,
+01:15 there's entry and there is a flat set of what you would call columns,
+01:19 this is only part of the story,
+01:21 remember we can have nested documents,
+01:24 we can have actually a rich hierarchy of nested objects.
+01:27 One thing we might want to store in the car is an engine
+01:30 and the engine itself is a special type,
+01:33 here in the field it's going to be an embedded document field
+01:36 an engine derives from mongoengine.EmbeddedDocument,
+01:40 not document, embedded document.
+01:42 These we're never going to directly insert into the database,
+01:44 in fact, we're going to always put them into a car,
+01:48 so this is like a strong relationship between a car and its engine,
+01:51 we can even mark it as required.
+01:53 Now going a little further than that,
+01:55 our service history actually contains a list of subdocuments,
+01:58 each one modeled by the service record.
+02:00 The service record has things like the customer satisfaction,
+02:03 what service was performed and so on.
+02:06 Now if we take this, put some appropriate data into it and store it,
+02:10 we'll get something looking along the lines of this,
+02:12 in our document database in MongoDB,
+02:15 so here we have the first few elements that are just the flat fields
+02:18 and then we have the nested engine, one of them,
+02:21 we have the nested array of nested items for the service histories,
+02:24 and this really gets at the power of MongoDB,
+02:28 this nesting and these strong relationships
+02:31 where you get this aggregate object the car,
+02:34 that always contains everything we need to know about it.
+02:37 How about queering— we're not going to write now in the low level api,
+02:42 we're going to use basically the properties of these objects.
+02:46 Here's the function that we wrote where we wanted to ask the question
+02:49 what percentage of cars have bad customer rating,
+02:53 that would be average or below,
+02:56 so we're going to go to the car and we say objects,
+02:58 we could do lots of these objects.filter.filter.filter
+03:02 but if you just have one query you can just stick it in object,
+03:04 so as the objects service_history, now we can't say dot here,
+03:08 because service_history . customer_rating
+03:10 would not be a valid variable name or parameter name in Python,
+03:13 so we're going to traverse a hierarchy with a double underscore.
+03:17 We also might want to apply one of the operators,
+03:18 in this case we're going to say less than 4,
+03:21 so we're going to use again this double underscore,
+03:24 but in this case it's going to say on the left is the name of the target
+03:28 and on the right is the operator we're going to apply to it.
+03:31 You don't put the dollar again, that wouldn't be valid in Python,
+03:34 but double underscore __lt, and then we can ask
+03:38 things like count, or go and get the first one, or things like that.
+03:42 We can even do paging by slicing on that result.
+03:45 This syntax lets us use almost the entire spectrum of the way of creating MongoDB
+03:50 really straightforward and in a way that ties back to the car object that we defined.

transcripts/ch10-conclusion/7.txt

@@ -0,0 +1,46 @@
+00:01 At this point, we pretty much had MongoDB
+00:03 doing everything we needed it to do,
+00:05 and we'd heard MongoDB was fast,
+00:07 but it turned out it didn't really seem to be behaving as quickly as maybe we hoped,
+00:11 we put a ton of data from our dealership in there,
+00:14 and we were getting query times of like one second, 700 milliseconds, stuff like that.
+00:17 It was okay, but really, we saw it can do much better.
+00:20 What levers and knobs do we have to turn to make this faster?
+00:24 The most important one, even more important than in relational databases,
+00:28 are the indexes, we'll see MongoEngine as well as PyMongo in the shell
+00:33 all have really good ways to deal with this.
+00:35 Document design is really important, mostly around this embedding question
+00:39 but there are many ways to think about document design,
+00:42 there's a lot of really non intuitive and powerful patterns,
+00:45 design patterns you can apply here.
+00:48 What is your query style, maybe one query is better than another
+00:51 and using projections to only pull back a subset of responses,
+00:56 suppose we have a car that has a ton of those service histories
+00:59 and we don't care about them for a particular query
+01:02 we could suppress returning those from the database
+01:04 which saves us a lot of bandwidth on the network,
+01:07 disks reads on the database server and deserialization processing on our side.
+01:11 We also saw there is some network apology things we can do,
+01:15 replication and sharding, and those are both interesting and powerful
+01:19 but not part of this course, so go check that out on your own if you're interested.
+01:23 For indexes, we took an example like our car
+01:27 and we said let's suppose we have make here
+01:30 that we're interested in querying by a service history,
+01:32 and if you look below how service history is defined as the service record objects
+01:36 and they have a description and a customer rating
+01:39 and things like this, price for example,
+01:41 so our goal is to query these things, the make, the service history and stuff, quickly,
+01:45 so we saw adding an index which really a powerful way to do that,
+01:48 so all we've got to do is go to our meta object, our meta element here
+01:52 and say these are the index as an array
+01:55 now these indexes can simply be the name of the thing,
+01:58 like make that's super straightforward,
+02:01 they could traverse the hierarchy using the Javascript style, using the dot,
+02:05 so we'll service_history.customer_rating
+02:08 and that would go down and let us do queries deep into these cars
+02:12 and say let's find the ones that are either good or low customer ratings
+02:17 and we can even do composite indexes,
+02:19 so here we're having a composite index on price and description,
+02:22 within the service history, so we do that by having this fields dictionary thing
+02:27 and the fields are an array, so you can use the simple version
+02:29 or if you need to, you can get a more complex definition of the index there.

transcripts/ch10-conclusion/8.txt

@@ -0,0 +1,41 @@
+00:00 After we had everything we needed for our database up and working,
+00:02 our code was working, we said time to put this puppy in the cloud
+00:06 and let people access it, so we talked about deployments.
+00:09 Now, there's a couple of things we could do,
+00:11 if you go to the MongoDB website and you pick the Linux deployment,
+00:14 you pick your distribution, it actually has a lot of really clear steps,
+00:17 like these are the steps that takes to use your package manager on Linux
+00:20 to get MongoDB installed, and I recommend you to use the package manager
+00:24 because then you get automatic updates, and things like that, it's really nice.
+00:27 However, we also talked about the ways in which MongoDB is
+00:31 maybe going to put you at risk,
+00:33 let's say if you don't know what you're doing about configuring it,
+00:37 so if you configure it to just listen on the open internet without say authentication,
+00:40 you are just asking for some sort of punishment,
+00:43 so there's a couple of things that we went through,
+00:46 a very detailed set of here is how you limit network access on Ubuntu,
+00:50 here is how you enable encryption, here's how you enable authentication, and so on,
+00:55 so the checklist we went through was,
+00:57 first thing to do is limit network exposure.
+01:00 That was a couple of things, one we set up the firewall on Ubuntu,
+01:04 if you want to use a cloud provider that's fine as well,
+01:06 so we set up the firewall, we actually listened on a non default port
+01:10 which we blocked by the firewall, and then we let the few servers in the world
+01:16 that needed to talk to it back in by explicitly allowing in those ip addresses.
+01:19 We enabled access control by creating an account
+01:23 and go into the configuration and enforcing authentication, say it's required,
+01:26 we added encrypted communication by creating
+01:30 a self signed ssl certificates and then adding that in there,
+01:34 you may consider adding encryption at rest as well,
+01:37 so like the actual stuff on disc is encrypted,
+01:41 we didn't go to that it wasn't really necessary for what we were doing.
+01:44 You could audit what's happening on your server,
+01:49 we didn't talk about that but it's pretty straightforward,
+01:51 we also talked about how you can run backups,
+01:54 I mentioned that you can do replication and some of these live backups
+01:56 but you can also use Mongodump for reasonably small data,
+02:00 not terabytes type of data but gigabytes, and that works pretty well as well,
+02:05 we saw that we can even do that over our ssh, so back up, back up, back up.
+02:09 Here's the whole security checklist that we talked about
+02:13 you can go through and read all the ways do it,
+02:15 or just go back and look at the various steps in the previous chapter's video.

transcripts/ch10-conclusion/9.txt

@@ -0,0 +1,8 @@
+00:00 That's it! I want to say thank you, thank you, thank you,
+00:03 I really appreciate you taking my course,
+00:05 I hope you learned a lot, and I hope you found it valuable and enjoyable,
+00:08 and generally just had a great time.
+00:10 I also hope you go build something amazing with MongoDB,
+00:12 if you do, send me a message, either here on twitter at @mkennedy,
+00:16 or visit the website and send us an email and tell us all about it.
+00:19 Until then, take care and thanks again

transcripts/ch9-deployment/1.txt

@@ -0,0 +1,80 @@
+00:01 You've learned almost everything you need to know about MongoDB
+00:04 to work with it, to make it fast, to access it from things like MongoEngine.
+00:07 The last thing is to actually put it into production,
+00:11 to use MongoDB in production to set up our applications,
+00:14 to talk to a secured version of MongoDB, all of those things.
+00:18 So we're going to focus on two main areas,
+00:21 one deploy MongoDB for production servers, two, doing that safely.
+00:25 So far, what we've been doing is we've been running our web app,
+00:30 or script, our little test app, whatever it is that we're going to be building,
+00:34 a little thing we even playing with, and that's the blue thing here,
+00:37 and we've been running it and talking to MongoDB on our local machine,
+00:40 probably our laptop, and we've been just on the local loop back,
+00:44 127.0.0.1, talking to MongoDB, and I have gone on and on
+00:49 about how you must not listen on another ip address on your local dev machine,
+00:55 take that thing to a coffee shop or worse, to like a big hotel
+00:59 where there's a tech conference, god forbid, black hat or something going on
+01:04 and that thing will be in some serious, serious trouble right away.
+01:08 By the way, did you know on your Mac that the firewall is off by default?
+01:12 That's right off by default, that's crazy, but it is, so just another thing to consider,
+01:18 layers and layers and layers, but assuming we're only listening
+01:21 on local loopback we're pretty safe like this,
+01:24 but we have been running without encryption and running without authentication,
+01:29 MongoDB gave us a little warning when we connected
+01:32 but you have to connect, you have to care, if you connect with code
+01:35 and not with a shell, there is no warning it just works,
+01:38 we're going to set up an entirely different thing
+01:41 we're going to have a set of web front ends, fake web front ends,
+01:44 we're not really going to write a website, but what would stand in for our website
+01:49 and we're going to have our production MongoDb server,
+01:51 and these things are going to talk to each other over
+01:55 at least the file computing data center connection,
+01:57 potentially farther than that, so we're going to have to open this thing up
+02:01 and that means we need to add things like encryption,
+02:04 we need to add authentication, firewall sorts of rules and things like that.
+02:07 That's what we're going to talk about in this chapter.
+02:10 This is a bit of a chore, right, this is not the default
+02:14 unfortunately this is not the falling into the pit of success type of thing,
+02:18 you have to work to get this set up correctly
+02:21 so let me just give you some very recent warnings
+02:24 this is not to tell you not to use MongoDB,
+02:27 I use MongoDB for my production stuff, I love MongoDB
+02:29 but you must use it carefully, it's like a sharp knife.
+02:33 What I am about to show you is not meant to dissuade you in any way
+02:36 but just to make sure you really have this burnt your mind
+02:39 that you have to be careful when you're deploying stuff to production with MongoDB.
+02:43 That said, check this out—  here are some recent headlines
+02:47 and you want your company and your data to be nowhere near them,
+02:51 MongoDB databases are being hacked for ransom using ransomware
+02:56 notice this is 2017, here's a little example, look at this,
+03:00 show dbs, please read, use please read, show collections,
+03:04 please read me do a little find on it and you get
+03:08 your database is been hacked and encrypted
+03:10 you have to send a bitcoin here to get it back
+03:13 and chances are they will probably just throw away your data
+03:15 and they'll take your bitcoin and yeah, good job,
+03:17 okay, so here's another one, how about this
+03:20 massive ransomware attack takes out 27 thousand MongoDB servers
+03:24 terabytes and terabytes of data, the petabytes of data we're lost to the world,
+03:28 so these are all not good things, right, you've lost your data
+03:31 here is one more, two million recordings of families
+03:34 imperiled by cloud connected toys crappy MongoDB
+03:38 you don't want that anywhere near your business,
+03:41 so this is a little bear thing here, that with a light on it
+03:43 it connects to the internet and it connects to a service
+03:47 and then the parent and the kid can like talk to each other
+03:50 over the internet through the bear,
+03:52 that was basically all that data in that exchange was done
+03:55 entirely on a wide open MongoDB database with no account at all,
+03:59 just hey here's the port, connect to it, go have fun.
+04:03 All right, so the problem here is that MongoDB lets you listen
+04:07 on the wide open internet without a usnername and password
+04:10 and without encryption and that's the default.
+04:13 What we're going to do it is we're going to add all those things
+04:16 but you just wanted to be really clear
+04:19 like this is not an optional step, we really need to do this
+04:22 unless you're running this, like say you're running a website
+04:24 and the database server is running the same machine as the web server
+04:28 and it just talks over a local host
+04:30 any other scenario use to be extremely careful
+04:32 and we're going to go through all the steps necessary to get this working just right.

transcripts/ch9-deployment/10.txt

@@ -0,0 +1,19 @@
+00:01 Let's review how we added encryption.
+00:03 Somehow we got a hold of an ssl certificate and a private key
+00:06 so the way we did this is we just went into the location
+00:10 where we typically store those analytics
+00:13 and we ran open ssl to generate a self signed certificate,
+00:16 the only change that I made here from the recommendation from MongoDB
+00:20 is I added, I made it a ten year certificate,
+00:23 because look as long as it's not trusted let's not trust it for a long time.
+00:26 Anyway, we did that and then we combined the private key and the certificate
+00:30 into this pem file, which we point MongoDB at
+00:33 we restarted, I first paused on the left, go to the right,
+00:36 we added this ssl section, we added the mode to require ssl
+00:39 and here's the file to do that, and then we were able to connect to MongoDB
+00:44 but only if we say --allow invalid certificates and --ssl,
+00:49 all of this is documented in that url below
+00:52 manual, tutorial, configure -ssl, so you can check that out and like I said,
+00:56 copy the details from there, not by typing them in from watching me do it.
+01:00 Alright, so a really nice step and important step to enabling ssl
+01:04 and secure communication on our MongoDB server.

transcripts/ch9-deployment/11.txt

@@ -0,0 +1,76 @@
+00:00 So we've encrypted our MongoDB,
+00:03 we've got it hidden behind a firewall and
+00:05 listening on a non standard port, let's get into it.
+00:08 Here we are connected to our Mongo,
+00:10 there is really nothing going on yet, it's just empty,
+00:12 we haven't added our data or anything like that, but nonetheless here it is,
+00:16 notice there was no user name or password required to get in,
+00:19 that's what we're going to fix next.
+00:21 So the first thing to do is we're going to run this db.create user command.
+00:25 We want to create a user to admin entire database server
+00:30 like all of MongoDB not just wherever we happen to be,
+00:34 which is called test, not an amazing name.
+00:38 So we're going to say use admin and now you can see db is admin
+00:44 so we can show collections, see what's here
+00:46 and it's basically empty, but now we can run these db commands
+00:50 focused on creating users against admin which means kind of global.
+00:54 So we're going to run this command here, paste it so I don't get it wrong
+00:59 because these roles have to be just so, it's very touchy,
+01:02 go with this for the db admin, that's probably fine,
+01:07 or mongodb admin, you can take your pick
+01:10 and the password is probably little wimpy, let's try to fix that.
+01:14
+01:18 Let's go over here and run pt python and import uuid, okay,
+01:24 and then let's do something like this, print
+01:28
+01:37 we'll call uuid that uuid4, call that, there we go,
+01:43 what do you think is that a decent password?
+01:47 I would say so, that's going to definitely slow down some dictionary attacks.
+01:51 Now over here, we got to delete this,
+01:55 sadly you can't paste over a selection in MacOS,
+01:58 alright, so we're going to run this user, this password
+02:01 and now we have to specify the roles
+02:04 we could create like multiple users
+02:06 that have certain restricted access to different databases
+02:10 and that's probably not a bad idea, but for this example
+02:12 we're just going to say this thing can admin read databases,
+02:17 admin any databases or clusters
+02:19 by the way just because you are an admin for a database
+02:22 does not mean you can read and write to it
+02:25 you could just create users and things like that, so you need them all.
+02:28 Let's try this, boom, successfully created.
+02:32 Now, did magic happen when we did this?
+02:34 Let me copy this real quick, if I exit and I go over here
+02:42 and I try to connect without any authentication,
+02:44 no, nothing happened; why, if we come over here and we check out our config,
+02:52 down here at the security, this puupy is wide open
+02:57 so we need to go down and say authorization is enabled;
+03:05 now, if we do that and we restart MongoDB,
+03:07 so service mongo d restart, probably a good idea to ask for status,
+03:12 also not happy, again, what have we done, let's have a look.
+03:18 I think it might help if I spelled this right,
+03:20 not very forgiving these computers are they,
+03:23 all right, everything is running that's all good,
+03:26 and if we try to connect to it again, now it's going to come over here and say
+03:31 hello you are connected right, db.version for example, like this,
+03:37 right so we're connected to it, we can sort of interact with it
+03:42 but watch this, show dbs, failure you cannot show dbs,
+03:47 in fact, you can't do anything other than basically log in.
+03:51 So I can come over and say db, I say this use admin db.auth
+03:58 and I could set the username and password, so I could say
+04:04 user is this, password is whatever we want to put here,
+04:10 you have to forgive me if I don't want to type that again, copy and paste that,
+04:14 pwd rather not password, so we could log in this way, as you'll see
+04:22 now I can say show dbs, use test and so on, show collection,
+04:30 so I am basically authenticated at this point, right,
+04:34 so I can log in this way and do this, but you probably don't want to do this,
+04:38 you probably don't want to do it that way,
+04:40 instead you probably want to say user is this,
+04:44 it says pwd, I think it might be, is this
+04:49
+04:52 oh one more thing, I forgot, so we have the username and the password
+04:56 but we also have to add the authentication database being admin
+04:59 there we go, okay, notice our warning about running without authentication is gone
+05:08 and I can now show dbs straight away, I don't have to go do this like
+05:12 switch to admin, do the auth and so on.
+05:15 So this is really handy for our scripts here that we're going to use later.

transcripts/ch9-deployment/12.txt

@@ -0,0 +1,19 @@
+00:01 Now let's see about enabling access control,
+00:03 we're going to connect to the shell like we have been,
+00:05 again specifying the extra things like ssl, import and so on,
+00:08 we're going to say use the admin database
+00:11 and then we want to create a user, I set the user to pwd
+00:13 and do not forget the roles this is very important.
+00:17 Once we've done this, this doesn't magically make authentication work,
+00:20 we have to go over to our Mongo config
+00:22 and say security authorization is enabled, then we want to talk to it,
+00:27 we can now pass the port, the, ssl stuff,
+00:32 the users, the user name, the password- p password
+00:35 and authentication database is admin, don't forget that, it doesn't work without it.
+00:40 At this point, you've basically set up your MongoDB in a safe way
+00:43 the final thing that you might consider,
+00:46 and it depends on how you want to run your database and so on,
+00:49 and you might set up a replica set to have failover
+00:52 and multi machine redundancy and things like that,
+00:55 that's certainly a next step that you could take
+00:58 but it's beyond the scope of this course so check out the docs.

transcripts/ch9-deployment/13.txt

@@ -0,0 +1,53 @@
+00:01 I think our big moment has arrived.
+00:03 We're finally ready to make this thing actually listen on the network
+00:06 and do what it is supposed to do.
+00:08 So with everything set up the ssl right here, security right here,
+00:15 also not default port right there, firewall in place, all these things
+00:20 let's change that to the glorious 0.0.0.0
+00:24 now, let's restart the server, see if it liked it,
+00:30 excellent it definitely liked it;
+00:33 see if we can still connect to it here on our local machine— we can,
+00:38 everything seems to be working well.
+00:42 It's now listening at least on local host, let's go over here to this one,
+00:45 now here's all the connection info we got built up,
+00:49 port, certificates, username, password, auth db, let's go add one more thing,
+00:57 because obviously local host is not where this server is running.
+01:01 So we are going to add host and put the ip address of the Mongo server,
+01:08 the moment of truth—we're in, look at that
+01:13 now, that's pretty cool, we could even do stuff on it,
+01:17 let  me go over you and copy this and let's try one more thing,
+01:20 maybe we've screwed up somehow,
+01:22 maybe something super bad is happening here and this is just wide open,
+01:26 let's try to connect to it, notice I am on my Mac book,
+01:29 I'm not on the web server or the Mongo server, I'm on my Mac book
+01:34 and timing, timing, it's timing out, ta-da, that's awesome, that's what you want,
+01:40 no connection possible, we couldn't connect to this
+01:44 because why— only that one server magically gets to connect to it, beautiful.
+01:48 And of course, we saw that we have to pass this auth stuff right,
+01:52 for some reason we don't pass that.
+01:55
+02:00 We still can connect as you saw, but we can't do anything whatsoever
+02:04 so I am not sure if I like that, I kind of would prefer
+02:07 that you can't even connect unless you just go through the authentication step
+02:11 but I guess more or less the same thing.
+02:15 So exit out and now we're back, working,
+02:18 because I ran the one that passed username and password,
+02:21 so this configuration of this little fake web server
+02:25 and this not fake Mongo server is running.
+02:30 Let's do one more thing, let's say pip install glances
+02:34 but we don't have pip, so apt install glances, let's skip that,
+02:41 like this, it's going to take a moment,
+02:45 so glances is pretty heavyweight, you may or may not want to install it
+02:47 but it's a really cool way to look at our server,
+02:50 so if we come over here and look around, it will show us things like
+02:54 how much memory our server is using, right now 15 percent.
+02:57 If we want to know how much cpu it's using, not much at all,
+03:02 right now we're sorting by cpu and here you can see
+03:05 Mongo is here just kind of hanging out that's 0.6 cpu
+03:08 like it must be doing some maintenance on itself,
+03:11 you can sort by memory and it will almost always put MongoDB at the top
+03:15 so over here you can see it's using really in terms of resident memory
+03:20 only six percent, that's not much but it has no data in it.
+03:24 So we'll come over here and we'll use this glances program
+03:27 to have a look at Mongo, maybe later we could load it up
+03:31 with this big dealership database that has the 1.5 million records or so in it.

transcripts/ch9-deployment/14.txt

@@ -0,0 +1,136 @@
+00:01 Let's go back to our little play-around service app,
+00:04 I'll go and run this for you, it probably looks familiar,
+00:07 remember our service central version 2.0, this is demo edition
+00:11 let me actually change that little header to prod edition, not that it much matters,
+00:16 but we're going to set this up to run in our production environment.
+00:19 If I try to do things like list the car, it will show me my local cars
+00:24 because I'm running on my Mac, however if I push this up to the server
+00:28 and I put it onto that fake web server server,
+00:32 it's going to try to talk to local host and have a big fall,
+00:35 right that's not going to work well.
+00:38 So instead, what we need to do is we need to do is
+00:41 we need to go and adjust our little connection bit here.
+00:45 Now, let me go and actually add some parameters to this
+00:50 we're going to add a password, say user, a password,
+00:57 a port, a server, use ssl, and I think that'll do, okay.
+01:07 So now I want to use those things to actually connect,
+01:10 so we're going to have like this dual mode thing going on,
+01:15 and I'll do it like this, we'll say if user or password,
+01:20 so if either of those are set we're going to do something else,
+01:24 we're going to just fall back and just do this little simple bit or right here,
+01:30 here I'll do a print, registering dev connection,
+01:35 so go like this, and it is not going to love it,
+01:40 so let's go over here and give this some defaults, so none for all of these;
+01:46 default here's to 27017, server=local host and use_ssl = false
+02:01 actually let's go and default that to true.
+02:06 Okay so now I should be able to run this here and list cars
+02:11 actually up here we'll see registering dev connection
+02:16 and let's put a little kind of indicator, something to the say
+02:22 hey this is like an extra thing, so go over here and we'll say registering prod connection
+02:32 and I want to give it some extra info, and let's hold off
+02:36 and what goes there just for a second,
+02:39 okay so we want to gather this up, we actually have to pass more information than this
+02:42 and just to make sort of recording how we connected a little bit easier,
+02:46 I'm going to create this dictionary where we set all these,
+02:51 so username is this, password, server, port, authentication sources, admin
+02:55 authentication mechanism is SCRAM-SHA-1,
+02:59 ssl is, use ssl and we have to say ignore the self signed certificate
+03:04 if we don't do this, it will say your certificate is not valid.
+03:09 Now PyCharm warns that this thing is basically missing from ssl
+03:15 but it's not, just ignore that.
+03:17 So we're going to come over here, and we're going to do this as well
+03:22 let's go and say, actually let me change the order real quick,
+03:29 so we're going to say all of these are keyword arguments for this method
+03:32 so we can just say **data
+03:35 and that's going to basically set username= user, password = password and so on,
+03:39 why did I put it like this— because I'd like to capture all those details.
+03:42 So let me just do really quick data of password equals this,
+03:52 and then I'll just print out this dictionary here
+03:56 so registering, production, connection with those details.
+03:59 Okay, so if you pass a username or password in
+04:03 it's going to work differently, let's just make sure everything still runs
+04:06 can I list the cars, see the dev connection, yeah, excellent.
+04:11 So things are still working good on of the dev side of the story.
+04:14 The next thing we've got to do is come over here where we're calling this,
+04:18 and let's just go ahead and pass in all the details here.
+04:25 We wanted to use ssl that defaults to true, so that's all good.
+04:31 Now if I run this, you're going to see not amazing stuff
+04:35 so like list is probably going to time out, it takes a while for it to time actually,
+04:40 let's try to renegotiate the connection and it really doesn't want to crash
+04:43 but eventually this is going to timeout,
+04:46 we already saw we can't connect to the server here.
+04:49 So let me push this up to the git repository
+04:55 and then we'll get it on to the server and production and make sure everything works.
+05:00 I pushed those changes to github and let's go over to the web server
+05:05 see I am already here, I'm just in my home directory/root
+05:10 so what I want to do is I want to go and get that code over here,
+05:14 so we're going to go and go to the github repository for our code here
+05:20 notice when I do a refresh, you should see
+05:22 that I just added now with production capabilities,
+05:26 so let's copy this, and let's say git clone this, its a public repository
+05:31 so I don't need any credentials or any of that business.
+05:34
+05:39 Okay, so things are good, we'll go to Mongo and notice there's a source
+05:44 and I have 09 deploys, so if we look in here,
+05:50 we've got service central deploy and service starter,
+05:55 server central deploy is the starter obviously it's what we started with,
+05:58 the service central deploys is the one that we just changed;
+06:03 so for example, if we look at this
+06:08 you can see it's using this complicated version here,
+06:12 if we look at this one, you can see we're setting a MongoDB just the way we like.
+06:18 Okay, so now what we have to do is run it and let's go over here
+06:24 connect to the MongoDb server and say show dbs,
+06:29 hey there's nothing here, so let's go and run this,
+06:33 so we've got our service deploy, so we'll say it Python 3
+06:36 we didn't use a … or change its execution states.
+06:41 Now one thing we need is we need to install Mongoengine of course
+06:44 so let's do this, we'll just let Python do it,
+06:47 so we'll save Python 3 -m venv to create a virtual environment,
+06:56 here we need to apt install Python 3 -venv, try again,
+07:08 so now we'll source activate this and our prompt changes.
+07:15 Okay good, so now we should be able to run our Python 3 thing again,
+07:19 oh yeah, well it's active, we still need to pip install Mongoengine
+07:23 and that'll take PyMongo along with it.
+07:27 I believe that failed building a wheel because set up tools is out of date,
+07:32 anyway, it should still work. Let's give this another shot,
+07:36 now we have Mongoengine registered in a virtual environment,
+07:39 a virtual environment is active, our code is here
+07:42 a lot of deployment stuff, let's go.
+07:45 Oh, look at that, so now we're registering the production connection,
+07:48 I mean, you probably don't want to print this out all the time
+07:51 but notice the hosts, authentication, everything,
+07:54 it seemed to take it like the register worked
+07:56 we haven't tried to talk to the database yet, let's try to list the cars.
+07:59 There are no cars, did that make in a dent?
+08:04 No, no dent yet. Let's add a car this is going to be an F40,
+08:10 it's going to be built in 2010, that didn't crash, let's try to list the cars,
+08:17 look at that, let's add a service record, service that car.
+08:24 The vin number is that, the price of the service is a thousand dollars
+08:30 and this is going to be a new tires, the customer is extremely happy, loved it.
+08:36 Now we've got our new tires, so look at this, show dbs,
+08:40 use demo dealership, show collections, db.cars.find.pretty
+08:52 bam, look at that, we were able to make our Python code
+08:58 on the right machine with all the right settings,
+09:00 and all the farewell rules and everything,
+09:04 go over and talk to the MongoDB server.
+09:06 This is pretty excellent, we can go add another car
+09:10 obviously like at this point once you see it creating some documents
+09:13 and working to some degree everything is going to work, right,
+09:16 there's really nothing to it, so this is excellent,
+09:19 let me just go create one more car so we have two things,
+09:22 this is going to be Enzo and this was build very recently
+09:29 let's list the cars and add a service record for it.
+09:33 The Enzo needs some work, so for a 100 dollars that will be oil change,
+09:40 pretty happy, yeah, one more, the same car,
+09:47 this is going to be 250 tire rotation moderately happy,
+09:53 so let's go over here and do this again.
+09:56 There we go, we've got our Enzo with two service histories
+09:59 our F40 with one service history and so on.
+10:02 Okay excellent, so it looks like this is working,
+10:05 I added this other record here so we have a little bit of data
+10:08 because the next thing that we want to look at is how do we manage this,
+10:11 how do we connect our database management tools
+10:14 and backup things and what not to.
+10:16 As far as Python goes, this baby is rocking.
+10:19 I guess maybe connect one more time, boom, list the cars,
+10:23 there they are, yeah looks good to me.

transcripts/ch9-deployment/15.txt

@@ -0,0 +1,26 @@
+00:01 Let's review how we connected to our production server
+00:03 from Python and Mongoengine, here's how we connected before,
+00:07 we just said the alias is core and the name is dealership,
+00:12 and that was it, we just let everything else be the default
+00:15 and that worked fine when it was a wide open server on our local machine.
+00:18 It didn't work so well for production,
+00:23 so we saw that we actually added this function that
+00:25 there is a whole bunch of different things here
+00:28 so it takes the password, the port, the server, whether or not to use ssl,
+00:31 the username, as well as the alias and db
+00:34 and I kind of broke this into two different functions
+00:37 that you can use for multiple connections in this example,
+00:42 but you could jam it all into one like I did in the project.
+00:45 So I created this dictionary and we set additional things
+00:47 like the authentication source and mechanism
+00:49 and to tell it to ignore the ssl certificate,
+00:51 I put it in the dictionary so it's easy to print out
+00:54 like in my log here is I am connecting to the database server
+00:56 so you know which machine you're talking to,
+00:59 how you're talking to it, what user you're talking as, things like that.
+01:03 So if you want to just put all that data straight into register connection, fine,
+01:06 you could do that but I find this to be valuable for sort of historical purposes,
+01:10 so here's how we connected and in my real example I said
+01:14 we're going to use the old version if you don't pass in
+01:17 a user name of password, or other things,
+01:19 but if you do then I'll switch to this more advanced version here.

transcripts/ch9-deployment/16.txt

@@ -0,0 +1,95 @@
+00:01 It's great that we have our MongoDB running in production
+00:03 we've got our web server and a MongoDB server
+00:06 and they're entirely locked down, we saw before
+00:08 if we try to connect to that Mongo server,
+00:10 even though it's on a different port with ssl and authentication,
+00:13 we can't talk to it because the Ubuntu firewall is blocking access
+00:17 from everywhere in the world except for that one fake web server thing.
+00:21 So we can't talk to it, we can't for example manage it with Robomongo,
+00:24 which would be totally sweet, right,
+00:27 but we can't even connect to it via the shell, can we?
+00:30 Well, we tried that and we saw it failed and if I do it again it will fail;
+00:32 but I can ssh into the Mongo server like this, we've seen that,
+00:37 so that's cool, now what can we do with this?
+00:41 It turns out we can set up an ssh tunnel using that mechanism
+00:44 so here if we run this command -f to run in the background
+00:49 ssh over here, map the local port, 10001,
+00:53 on the remote machine say the local host 10001, like that.
+00:59 So if we run this code, it takes a second
+01:03 and it creates a background task of ssh tunneling one port locally over there;
+01:08 now, what if we try this— we're going to run the same command
+01:13 we saw working in production with authentication being this,
+01:16 here is the password, the admin and so on, notice there's no host up here
+01:19 we have the port 10001, what is the host if we don't put one— local host,
+01:24 but local host 10001 really means the Mongo server 10001.
+01:29 Let's do it. Check that out, it's working okay,
+01:34
+01:39 we can ask it how are you doing, how many things you got going on here,
+01:45 what is your host, this is what I was looking for, your host is the Mongo server,
+01:49 we're connected to the Mongo server, that's really cool.
+01:52 Now we can say things like show dbs,
+01:56 we could come over here and say use that,
+02:00 we could even do our pretty find here so cars.find.pretty
+02:05 and there's our data, okay so we can access this.
+02:09 And just like we could in the shell.
+02:12 Well if we can get to it this way, maybe,
+02:15 just maybe something magical could happen with better tools.
+02:18 And yes, yes it can, we'll create, it's going to be a direct connection,
+02:23 I'll call this themngoserver, connect on the local host 10001, that part is good,
+02:32 authentication database is good, copy these over, paste that in,
+02:41 you can see this here, a mechanism is good, so this is all set,
+02:47 come over and say use ssl, I've not tried to do anything else,
+02:54 let's try this, let's test it— all right, under ssl we say use a self signed certificate,
+03:02 there we go, good, alright, so we have themongoserver,
+03:07 I didn't test it, but let's go ahead and give it a shot anyway.
+03:14 Authentication failure, okay let's go and edit that again,
+03:21
+03:26 oh look at that, have a little space right there, how frustrating,
+03:29 couldn't somebody give me a trim,
+03:32 connecting, authorized, yes! That is so awesome.
+03:35 Okay, save the connection, now let's go over here, double click
+03:40 it's a little bit slow because hey, it's going over tunnels
+03:43 but look at that, if we go over here we got our cars, we can view the documents
+03:47 we have everything that you could have done before with Robomongo,
+03:51 you can do now, here's the two documents
+03:54 you saw me create in that Python section,
+03:57 oil change, tire rotation, Enzo Ferrari and so on.
+04:01 And we can do things like maybe we had the service_history.price as an index
+04:07 well, add an index, it's going to be service history price,
+04:14 and down here we'll say { 'service_history.price' :1 } like that, save
+04:22 and now how about that, we could even do a little thing
+04:26 come down here say  service_history.price is let's say 100,
+04:31 this should return just one record, and it does
+04:37 and if we say explain, all the stuff we were doing, does it work— you bet it does.
+04:41 It's using that index that we just created remotely using Robomongo,
+04:48 so this is super cool, last thing let's see about doing a backup.
+04:53
+04:56 The next thing that I want to show you which I don't think we've done before,
+05:01 let's go to our desktop here and we'll say make a directory called backtest
+05:06 cd the backup, notice it's there on the back up, nothing is in it,
+05:12 so the last thing I want to do is show you how to use Mongodump
+05:16 so you can go to help and see all of the things that this does
+05:22 but we're going to use Mongodump with basically all the same settings down to here
+05:28 we're going to go to demo dealership as we've named it
+05:33 and the output is going to be into the working folder which is this.
+05:37 Because we're tunneled into the production machine
+05:41 we can go and grab that data from there and back it up locally, let's try.
+05:46 Boom, we wrote two, we're whopping two documents
+05:54 but over here, we have this, now the data comes out in this binary json
+05:58 but you can't really look at, we could technically look at this
+06:01 but the point is this worked, we got our two documents,
+06:04 now you might wonder like ok that's cool for two documents that kind of works,
+06:09 can you really do this for like actual data— yes, yes you can.
+06:13 So I do something like this for Talk Python To Me and the training site, all these things,
+06:18 and I can back them all up in one giant script that does things along these lines
+06:23 and it will back up to six million of records, six million documents,
+06:28 I would say it probably takes less than a minute and a half
+06:33 over my pretty standard connection, and I'm on the West Coast of the US,
+06:37 and that server is on the East Coast in Virginia,
+06:40 so it's not like I'm right next door, that's why it works.
+06:42 So this actually works better than I expected it to work I guess,
+06:46 and it really is quite nice, so using this ssh tunnel means
+06:51 we never have to open up that port,
+06:53 but we can go through ssh and still work with our server, with all of our cool tools.
+06:59 Over here, come back, which one do you want to work with—
+07:08 local or remote, remote one of course.

transcripts/ch9-deployment/17.txt

@@ -0,0 +1,45 @@
+00:01 We've seen that we can use our ssh as a tunnel
+00:04 to give us access to our production MongoDB server
+00:07 without exposing too much of it.
+00:10 So we can run this ssh command to the Mongo server
+00:13 and say map locally the port 10001 over to the machine called local host over
+00:18 in your area Mongo server to port 10001,
+00:23 which means we basically can connect our local host
+00:25 and we connect to the Mongo server on that side of things.
+00:28 Once we do that, we can go to things like Robomongo,
+00:32 and say I'd like to connect to here local host 10001,
+00:36 and the user name is whatever it is, the password is whatever it is,
+00:40 make sure you check perform authentication and also use ssl, check that,
+00:47 you want to use the self signed certificate if that's the way you did it
+00:53 you saw that it doesn't work, kind of blocked me when I said certificate
+00:56 because it is like no, no this is an invalid certificate we won't talk to the server,
+01:00 you decide how you do the certificate management
+01:02 but if you followed along exactly,
+01:04 you want to make sure you use self signed certificate
+01:05 and then ta-da, you are connected to the server just as if it was local
+01:13 there's a slight latency but like I said, once it spins up and starts moving
+01:17 it can actually ship a lot of data over that connection pretty quickly.
+01:20 The other thing you might want to do is connect with the Mongo shell,
+01:23 we've already seen how to do that, here's the command to do it once again
+01:26 remember, this is using a port 10001 on my local dev machine
+01:31 tunneling through the ssh tunnel back to the server;
+01:34 same type of thing you put on the server, basically goes right here
+01:38 as long as you have that ssh tunnel running.
+01:40 We can also use Mongodump and Mongodump is one of the ways
+01:44 in which you can back up a database
+01:46 so same commands exactly except we said --db dealership output is local folder,
+01:52 that will dump out all that data, and like I said
+01:55 this actually works pretty well over that ssh tunnel, for large amounts of data.
+02:00 There you have it, we have our production MongoDB server up and running
+02:04 we installed it through aptitude which means
+02:07 if there's update, security or otherwise
+02:09 it will automatically apt upgrade— boom take care of it for us,
+02:13 that is really super nice, we've got over the wire encryption,
+02:17 we've got non default ports, we've got authentication
+02:22 and we even saw how we can use ssh to still use our local dev machine
+02:27 we even saw how we can use ssh tunnels
+02:30 and our local database management tools,
+02:33 Robomongo, the Mongo shell and things like that
+02:36 to get back into the production server and manage it,
+02:39 without punching holes in the firewall for that purpose.
+02:42 So all in all, I'd call this a pretty good setup.

transcripts/ch9-deployment/2.txt

@@ -0,0 +1,50 @@
+00:00 Let's go through the MongoDB security checklist.
+00:03 Now, most of these come from MongoDB,
+00:05 but also from me personally, from my experience running the server.
+00:09 I've run professional commercial websites
+00:12 using MongoDB for many years, 5 or 6 years
+00:14 and we've never had any problems, but you have to follow the rules.
+00:17 Some of the rules include things like limit network exposure,
+00:20 so this is always a good idea for databases
+00:23 or anything else that listens on the internet,
+00:26 if something doesn't need to talk to it, don't allow it to have an open connection.
+00:30 Enable access control, that means add users and require them to authenticate,
+00:36 this should really be the default, if I were king of MongoDB,
+00:41 and I'm not, but if I were a king of MongoDB,
+00:43 I would make a decree, a new rule that says
+00:46 MongoDB is not allowed to listen on anything other than local host,
+00:50 unless it has access control enabled, period.
+00:54 That's not the way it works right now though,
+00:56 by default you can just say listen on 0000
+00:59 and it will, even if that's wide open, so that can be problematic,
+01:04 so we're going to change that,
+01:07 encrypt the communication, so certainly what goes on the wire
+01:09 should be ssl style encrypted,
+01:12 but there is an option to encrypt the data at rest,
+01:15 I think you have to get the enterprise version of MongoDB which is the paid version
+01:20 this is not something I worry too much about
+01:22 but if it's something that you were about,
+01:24 you can encrypt the data at rest using the wire tiger engine.
+01:27 You can audit system activity, this is easy enough
+01:32 turn on what's called a caped collection and turn on auditing
+01:36 so it will keep some standard amount
+01:39 obviously set up logging, all those types of things.
+01:42 Back up, obviously you want to back up your data
+01:46 this is production data, so back up, back up, back up, back up,
+01:49 set up some way to back up and we'll see that there's two options
+01:52 we can run a tool called Mongo dump which will just take a complete backup
+01:56 and for a certain size of data, that could be pretty large actually
+02:00 but for certain size that works fine,
+02:03 at some point if you truly are working
+02:05 with tremendous amounts of data that doesn't work so well
+02:07 so there's various ways to set up replicas
+02:09 that are like delayed or right on time, things like this
+02:13 but back up, back up, back up, an important thing to do.
+02:16 So you can find all the details on how to do this here
+02:20 at mongodb.com/manual/administration/security-checklist
+02:23 now you're welcome to go over there and check that out
+02:25 and I encourage you to do so,
+02:28 but we're going to go through each of the yellow steps here
+02:30 we're not going to talk about auditing, but everything else pretty much
+02:32 and encryption and rest, we're also not going to do that,
+02:35 everything else we're going to do as part of this chapter.

transcripts/ch9-deployment/3.txt

@@ -0,0 +1,133 @@
+00:01 To deploy our database and set up our production environment
+00:04 I'm going to use Digital Ocean,
+00:06 I've run web applications in MongoDB, in AWS, EC2,
+00:11 I've done it in Azure, and various other places I'll talk about some of those,
+00:15 and I found something like Digital Ocean really to be just such a nice service,
+00:22 simple, extremely fast, extremely affordable compared to the other options.
+00:25 We're going to use Digital Ocean, but what I'm going to show you
+00:29 is not specific Digital Ocean for the most part
+00:31 you can use any other server that lets you basically set up vms,
+00:36 and in a single data center.
+00:39 We're going to use this and if we come down here we'll look at the various options,
+00:46 we'll see that we can basically choose different machines,
+00:49 now it turns out for reasonable amounts of data
+00:53 I'll describe what I think but reasonable is,
+00:56 certainly ten dollars a month is absolutely fine,
+00:59 we've got 30 gigs of disk space on an ssd disc,
+01:03 we've got tons of bandwidth and I don't even know
+01:05 if it counts within data center bandwidth,
+01:07 1GB is not a ton a memory, but it is enough,
+01:10 this is really nice and cheap, the five dollar one,
+01:13 but it's going to put you up against memory limits pretty quickly,
+01:15 if you have lots of data, so what do I mean by a lot,
+01:18 so right now I'm running most of my websites
+01:20 using a shared MongoDB server, separate databases, but shared server
+01:24 and it's running on one of these ten dollar machines
+01:27 and it's got about six million documents in there,
+01:31 something around six million documents, and it takes about
+01:35 let's say 30 percent of the memory of 1GB, so about 300MB resident,
+01:41 if I had lots more than six million things, than probably I'd need to move up.
+01:47 Also if I wanted to run a replica set on the same machine,
+01:51 all these kinds of things, but this is probably a totally decent starting point,
+01:55 unless you really have quite a bit of data.
+01:58 Anyway, we'll get started with Digital Ocean.
+02:00 The first thing we're going to do, what we're going to do in this video, in this lecture is
+02:03 we're just going to create two servers,
+02:06 one that's going to be our web server that's just the thing
+02:09 that's going to try to access MongoDB our app
+02:11 and the one that is the deployment production server
+02:14 that we've kind of locked down and hardened,
+02:16 so let's switch over here for now, and we're going to go and create a droplet.
+02:23 I've done a tiny amount of work in advance,
+02:25 I've created a certificate that I'm going to use to ssh in,
+02:28 I'll show you where the step is and there's a button you can click
+02:31 and it basically says type this, put the contents here, you're good.
+02:35 When we come in here there's a couple of options,
+02:37 the first thing is to choose an image,
+02:40 so we could choose all these different versions of Ubuntu,
+02:42 I'll just take the default, I'm tempted to take the new one
+02:45 but it will take the long term to support one.
+02:47 If you wanted to use another distribution, you totally could,
+02:49 also they've got this one click apps thing that is pretty interesting
+02:52 and I could come down here and even click MongoDB
+02:55 but I don't want to assume that using Digital Ocean you have this button
+02:58 I want to show you how to set up a fresh Linux machine
+03:01 running MongoDB in the end basically.
+03:03 So I'm not going to click this, but this is a totally reasonable option to click this
+03:06 and it has ability to upgrade basically through apt update.
+03:10 So for this, let's go with the ten dollar one, it's charged by the hour
+03:16 we actually pay for this course, it's going to be quite quite low,
+03:20 I'm not going to leave it running for months at a time.
+03:22 We're going to do this, 10 dollars a month, standard Ubuntu,
+03:25 I don't care about block storage, I'm on the West Coast of the US,
+03:28 so let's pick something somewhat nearby
+03:30 but you see there is other data centers,
+03:33 probably you want monitoring, this allows you to go back and do a droplet
+03:36 and get graphs of like cpu, disk, memory, over time that's kind of cool,
+03:40 maybe private networking, but again we're not going to do that here,
+03:42 I have already set up the ssh key,
+03:45 so I'm going to pick this Digital Ocean course test key,
+03:48 which doesn't want for anything but this test bit that I'm doing right here,
+03:52 we also create a new ssh key and there's a little help button you can click,
+03:55 and I'll just show you how to create and store one of these here.
+03:58 Alright, so last thing we want to go over here,
+04:03 we got to give it a name, this Ubuntu name not the most amazing,
+04:06 we'll call it the mongo server, that seems decent right,
+04:11 it doesn't like this, so we'll just go like—
+04:14 so this is all looking good, we've got our ssh key
+04:19 we just need one of these types of things and we click go.
+04:23 I'll let this go in real time, so not sure how long it's going to take today
+04:33 but I'm not going to speed up this part,
+04:36 you can see this is just a real time creation here.
+04:39
+04:45 And we're good, it says happy coding, I love it.
+04:48 Alright, so let's copy this, let's go ahead, I think my ssh is already registered
+04:53 if not I might have to add that, let's go,
+04:56 so we're going to go here like this,
+04:59 and it says you've never connected to this server, no it's brand new.
+05:04 Apparently I have not added that, so go down here, add ssh-add
+05:12 at the k is added to my keychain, like so, so it's added
+05:16 now if I ssh again, do this one, how about the one we're actually working with.
+05:22 Okay, look at that, we are connected,
+05:27 so I had generated my key but I hadn't added it to this user profile,
+05:31 so this is great, and it should also be somewhat concerning
+05:34 that there are 16 security updates right now,
+05:37 so first thing we are going to do, we're going to apt update,
+05:40 go refresh the possible updates and a real quick upgrade,
+05:46 and we'll be back in a minute.
+05:51 Okay, everything is done, now let's exit out real quick and just come straight back
+05:57 and notice, there's no more updates, but a restart is required
+06:01 in order to make this basically finalize those changes,
+06:06 something deep down in the guts was updated,
+06:09 so we'll just do a quick reboot and just to show you the time in here
+06:12 I will not speed this part up either.
+06:15
+06:19 Usually it takes about ten seconds,
+06:21 but with that many updates it might take a little bit longer;
+06:25
+06:29 let's be optimistic give it a shot, and we're back,
+06:33 so really quickly we updated our system, we rebooted
+06:36 so we've got Ubuntu 16.04.2 long term support, and it's all up to date.
+06:43 This is great, this is our Mongo server, let's do this one more time
+06:47 let's go and do this for here, go back to the other stuff in a minute,
+06:55 let's do this for the fake web app that we're going to have talk to this.
+06:59 We'll come down here and pick Ubuntu, five dollar one
+07:03 we don't need block storage, sfo 1, same data center as before
+07:06 that's very important for latency;
+07:09 go ahead and add monitoring, use this ssh key,
+07:16 and we'll call this the web server,
+07:21 and go— good, these are the same data center
+07:27 and we'll do the same thing, I'll ssh into here I'll do apt update, apt upgrade
+07:34 and give it a good reboot, and then we'll have two fresh up to date machines
+07:39 and we'll start configuring them afterwards,
+07:41 let's just double check this one,
+07:44
+07:51 so it's alive, but make those a little bit quick, there we go,
+07:54 now it took a moment just to turn on, excellent, everything is good here
+07:59 let's say apt update, it says there's no packages
+08:03 but I'm not so sure, it's basically running that right now
+08:09 so let's come back in a second,
+08:12 oh look, there's a whole bunch of stuff that we got to do
+08:15 so apt upgrade and we'll do this, I'll kind of shorten the video here
+08:21 you've gone through this before, and we'll just let it do all the upgrades
+08:25 and then we'll come back and talk about installing MongoDB on the Mongo server.

transcripts/ch9-deployment/4.txt

@@ -0,0 +1,150 @@
+00:01 It's time to install MongoDB on our cloud server.
+00:04 One thing I'd like to point out is you don't have to necessarily go down this path
+00:07 to run your own MongoDB server, you maybe don't want to deal with it,
+00:10 maybe you don't have enough experience with it things like that,
+00:13 so a couple of options just that I want to point out,
+00:16 but I definitely want to show you how to run your own MongoDB server,
+00:19 how to do it safely so that you can because in a lot of cases it really is the best.
+00:22 So MongoDB has this thing called MongoDB Atlas
+00:26 which is database as a service,
+00:28 and basically what you do, I haven't used this really,
+00:31 but I have looked at it, is you create a AWS account
+00:33 it was on web service, EC2 account,
+00:35 you give them access to make machines and manage those machines on your behalf
+00:40 and they can create a replica sets and things like that that are cure for you,
+00:44 this is like a service that will manage more or less
+00:48 your EC2 machines upon your behalf, so this is decent.
+00:52 Another one is you can go to something like mlab over here
+00:55 and you can check out the plans on pricing,
+00:58 they have a free, sandbox free half a gig,
+01:01 just cool when you do a shared one with light production work
+01:06 for up to eight gigs of data, but it isn't a replica set with fail over and things like that,
+01:12 so this is a pretty nice service, it's really expensive in my opinion
+01:15 but it is pretty much turnkey, push the button
+01:19 you get what you want, you get what you need.
+01:22 I found it to be decent, but also it seems like it's added a lot of latency
+01:28 to some of the apps that we moved off of our own servers on to mlab,
+01:32 so I guess it probably depends, one on how much you pay,
+01:36 and two, how close your machine is to their machine,
+01:39 but they do claim to do a lot for half a million MongoDB deployments,
+01:43 on the major cloud providers.
+01:46 I just want to put it that like you can go and just get MongoDb as a service.
+01:49 Now, if you're still with me, I'm assuming that you want to install
+01:53 understand how to create your own MongoDB servers,
+01:56 so let's go over here to download,
+01:59 and we're going to go through a few interesting steps,
+02:01 so I would like to do the Linux download, I know I'm on MacOS,
+02:05 but I'm configuring this over here, now notice, I could hit the tarball
+02:10 and this would do a thing, I could install this, I could run it,
+02:14 but it wouldn't give me the ability to say automatically upgrade my server.
+02:18 Right now it says instructions for installing with yum,
+02:23 but I want to do this on this x64 version of Linux of Ubuntu,
+02:30 16.04 that's the one I got, I think you can just take the same instructions
+02:36 and apply them for 17.04 as well.
+02:39 Now here's what we really want to go, we could click this,
+02:41 we could get the binary, but this is better,
+02:44 so we're going to come down here, and there's just going to be some copy paste action,
+02:49 now look, it says what we can do is
+02:52 you can just use aptitude to install this, so let's try that.
+02:56 Before we actually go over here, tell me, which one of these is a MongoDB server,
+03:03 I don't know, I don't remember either, so let's take just a moment
+03:06 and step back and give these names, and I want to give them
+03:09 the exact same name as they believe their machine names are,
+03:12 so this one, the web server, it refers to itself as a web server,
+03:17 this one its local machine name is this, themongoserver
+03:22 so let me open this one up, now we should probably
+03:25 enable floating ips for real production,
+03:28 but this is not real production this is me playing around
+03:32 so I'm not going to mess with that.
+03:35 We could also enable what are called cloud firewalls,
+03:38 but again, this is the Digital Ocean specific thing,
+03:41 you do this in EC2 differently, you do this in Azure differently and so on,
+03:43 so I didn't want to show you how to just use the Linux tools to do that
+03:46 but it may be better actually to do this,
+03:49 here you can see some of the monitoring kicking in
+03:52 so I'd like to be able to say ssh root@ themongoserver, right
+03:55 and sadly, it doesn't work, so let's tell this machine,
+04:00 let's do a sudo, and I'll run visual studio code again, it's /etc/host
+04:08 here you can see I have hacked a few things together already,
+04:12 and we're going to go and put this, the Mongo server in here
+04:16 and what is its ip address, of course like I said,
+04:19 give it a floating ip and use that one possibly
+04:22 but we're going to go like this, if you want to give it a real domain name
+04:27 feel free to go ahead and do so, but this will work for,
+04:30 there's probably no reason to give your MongoDB like a public dns name
+04:35 so I'm going to suggest that maybe you don't do that.
+04:38 Let's go here and get the web server,
+04:41
+04:50 okay, so I save that, now let's try that again,
+04:54 we've never connected to a machine called themongoserver with this key
+04:58 so it's fine, and now we're back,
+05:02 so now we can say connect to root@themongoserver,
+05:06 and at thewebserver, that's what I called it right, thewebserver.
+05:10 This will make things easier and you can see even on that machine,
+05:14 it believes it's called this, for some of the tricks we do later with tunneling ssl,
+05:18 it turns out that makes our life a little bit easier.
+05:22 Okay, so we're on the mongo server, that was our goal
+05:25 maybe a little bit long to get here but that was our goal.
+05:28 The next thing to do is we're going to go down this list that they gave us here,
+05:33 so we are going to do is ssh in here and play this,
+05:36 so it says what you can do is to install MongoDB
+05:42 is you can install this aptitude package
+05:45 and then you can actually install smaller pieces,
+05:47 like we could install say just the server, right
+05:50 or maybe just the sharding deamon things like that,
+05:53 but if you install this you kind of get all of it,
+05:55 and it's going to be amazing except for that it's not,
+05:58 it's not amazing it all, it's not there,
+06:00 because this comes from one of mongoDB's own app update servers,
+06:04 so we got to go down here and go through the steps,
+06:07 so the first thing we have to trust, trust MongoDB
+06:10 we're going to stall the software as root,
+06:13 I guess we're going to have to trust it anyway aren't we.
+06:16 Then all seem to come out alright,
+06:19 be careful here, I always screw this up,
+06:22 even though I clicked on install for 16.04, it gives me all the options here
+06:28 so don't do that, that's 12, 14, 16, that's done.
+06:35 Now the next thing to do is run apt update,
+06:39 I'm already in root, so I don't need sudo
+06:43 so we needed to do that basically to pull from that list we just added there,
+06:50 so now let's see what it's asking about, it's all good.
+06:58 Alright, so now we can go do that apt install mongodb.org,
+07:04 and what happens— magic is going to happen, that's what.
+07:09 Notice when I said this one, it's like a metapackage,
+07:12 it's really installing those four, it just said
+07:15 hey here's an empty package with these four dependencies.
+07:18 Alright, that was quite quick, and notice we have 3.4.5
+07:25 so that is quite a recent one here,
+07:28 and it's even created a mongodbuser for us
+07:31 that is the one I believe that runs the deamon process
+07:33 so it is not running his route, that's pretty awesome,
+07:36 it's another thing you'd have to do
+07:38 if you just downloaded the tarball and tried to set it up.
+07:40 Very cool, and now next time I come over here
+07:42 and I run apt update and then upgrade
+07:45 that could potentially install 3.4.6 or whatever is next.
+07:51 Do we have MongoDB— I could type Mongo and something happens
+07:54 except fail, no you cannot connect; why— because it's not running,
+07:59 it will run though if we just say service mongod start
+08:04 then we can ask status, and notice, it's running this process,
+08:13 it's running the server in quiet mode, especially important using this configuration file
+08:19 so we're going to be able to use that basically to configure or to adjust to the server
+08:24 all the changes we need to make to MongoDB
+08:26 we're just going to edit that file and it'll be golden,
+08:29 but for now, let's just try to see if we can connect it.
+08:31 Wow, we can, we get a few warnings
+08:34 like you really should not put this on the open internet, you really shouldn't do that
+08:38 and some other stuff that we might consider about changing our file system around
+08:43 so we can say things like show db's and it just has the local stuff
+08:49 but we're connected and it's all good.
+08:52 So, that's installing MongoDB on our cloud server.
+08:56 However, you want to be extremely careful about changing this
+09:04 we're not ready to open this up, not even close,
+09:07 we can see here's the log file if we want to go and get the log
+09:12 here's where the data is stored,
+09:15 you generally don't need to go in there and mess with the data
+09:18 you don't need to mess with the files directly,
+09:20 we'll use the tools or replication or something to back up and configure the data
+09:24 but that's where it lives and you can change where it is if you need to, right
+09:28 so those two things are interesting, we're going to go change this stuff
+09:31 but we need to make a few configuration changes before we go and do that.

transcripts/ch9-deployment/5.txt

@@ -0,0 +1,20 @@
+00:01 We saw that we can basically leverage the package manager on Ubuntu
+00:04 to install MongoDB, but it won't work by default, we've got to add some things.
+00:10 Now notice this link at the bottom, you shouldn't be typing in
+00:12 at least not the first two lines at all,
+00:16 you can just copy them from down there, right.
+00:19 So go to the install on Ubuntu, or if you have a different distribution pick that one,
+00:23 and it will show you how to do it with your package manager most likely,
+00:26 so here we are going to add the MongoDb key,
+00:29 so we trust the list, the file list we're going to set up,
+00:33 then we're going to basically set up this mongodb.org file list here
+00:37 and then in order to actually have the stuff in the list available
+00:41 we need to run a quick update, so it pulls it down
+00:44 and then we can say apt install mongodb-org to install everything
+00:47 or you saw that there's subpackages you can choose
+00:51 we're going to manage the server via the just editing etc/mongod.conf
+00:57 and then make changes, restart the service
+01:00 and then it will just pick up those changes which is really nice.
+01:04 Of course, this doesn't mean it's running,
+01:06 it's just ready to run next time you reboot the server
+01:08 so you just say service mongod start and you'll be golden.

transcripts/ch9-deployment/6.txt

@@ -0,0 +1,23 @@
+00:01 One of the most important things we can do to make our MongoDB server safe
+00:03 even if we screw up the configuration, the authentication, the encryption, all those things
+00:08 is to make sure nobody can talk to it.
+00:11 So we're going to do two simple things right away to lock down our server.
+00:15 Obviously our web app, or whatever app, our service
+00:20 whatever we're building that uses MongoDB should be able to talk to it,
+00:23 and it's this probably within a data center
+00:26 we could possibly get to it from our local machines,
+00:28 but well do things like ssl tunnels and so on to do that,
+00:31 so we won't open up any extra ports for this.
+00:34 However, there's always something out there lurking,
+00:37 I showed you that super scary warning at the beginning
+00:40 and they're out there looking, they are saying
+00:43 hey I would love to talk to the server on port 27017 the default port
+00:46 or maybe 1.8 or 1.9, or 20, depending on the service you're running.
+00:50 So we want to block those guys, we want to block them with the firewall
+00:53 and a couple of other things.
+00:56 That's what we're going to do next.
+00:58 We're going to do this like I said, in Linux itself, in Ubuntu itself,
+01:01 we could the cloud computing stuff like Digital Ocean just announced
+01:06 this cloud firewall thing that is really probably easier
+01:09 and if you're using Digital Ocean have a look at that,
+01:12 but we'll do it here and it works just fine.

transcripts/ch9-deployment/7.txt

@@ -0,0 +1,80 @@
+00:01 Alright, so on the left here we're logged into our MongoDB server
+00:03 and let's go to the web server, we're logged in here,
+00:12 now on the web server, just for now, I'm going to set up the Mongo shell
+00:16 so that we can sort of simulate talking to this
+00:20 from the web application, our little fake web application in Python
+00:24 which we haven't gotten to yet, but we'll do that later in this chapter.
+00:26 And we already added the list here, so we're going to install, apt install this,
+00:31
+00:38 ok so let's go Mongo, you're going to run something great, not the right one,
+00:42 okay, so before we do anything let's see if we can get to our Mongo server,
+00:48 and the answer will be no,
+00:52 so here this is the host name of the Mongo server,
+00:57 right now if I try to connect to it, it's going to say no,
+01:00 if I come over here and I type mongo it connects, what is going on?
+01:05 Remember this, remember it's listening only on local host. 01:14 So we're going to want to change this, but not before we make it safe,
+01:17 so we don't want to just tell it to listen on the open internet right away
+01:22 so let's first block access to all of these ports
+01:27 and everything basically except for initially ssh,
+01:31 so what we're going to use is we are going to use something built into Ubuntu
+01:36 called uncomplicated firewall.
+01:40 The first thing that we're going to do is say ufw default deny incoming.
+01:45 By default we're blocking all of the ports.
+01:51 Now, we're going to say allow outgoing,
+01:55 so by default allow our server to get back out, that's cool.
+01:58 The other thing that we want to allow, unless this is going to be
+02:02 the very last time we see the server,
+02:04 we're going to need to allow ssh back to this server.
+02:07 Not default, just allow ssh.
+02:14 Okay, great, it updated for ipv4 and ipv6, that's pretty sweet.
+02:19 Now the last thing is a moment of truth, we're going to enable it,
+02:23 we could ask the status, it's not enabled,
+02:28 it says you know, if you are blocking ssh, you're going to be done for; we're not.
+02:34 And let's just verify, just connect, reconnect, okay, we're good.
+02:40 So at least now nothing can talk to any port except for 22 ssh, at all on this server.
+02:47 The one final thing to do, let's go over here and say ping the web server,
+02:58 so this, that's the ip address of the web server,
+03:04 what I want is to allow the web server to get to the Mongo server,
+03:09 so one more thing I'll say ufw allow from here,
+03:14 so uncomplicated firewall allow from this to any port
+03:18 and we're going to give it a port here and normally you would type this,
+03:23 27017,  that's the default port,
+03:29 but the very next thing we are going to do is say
+03:32 running MongoDB on the default port probably is a stupid idea,
+03:35 everyone is scanning the wide open internet for 27017
+03:38 and then seeing what kind of havoc they can wreak upon that.
+03:41 So even though we think our firewalls are blocking the wide open internet
+03:45 for everything except for ssh— let's go ahead and change the port,
+03:51 so we're going to say 100001 is the port we're going to run Mongo,
+03:55 so we're going to allow that thing to come back to 10001,
+03:57 where MongoDB is going to be listening.
+04:01 Okay, rule added. So it is running,  it's listening on just that port.
+04:07 Next thing to do is we're going to want to go and change the port here,
+04:12
+04:15 like this, and change this port, 10001.
+04:19
+04:23 Excellent, okay, so MongoDB, we're going to have to go do a service restart,
+04:30 now if I type Mongo fail, but if I say --port, like that, we're good.
+04:36 So it looks like everything is working over here.
+04:39 It's still not going to listen to us,
+04:45 because we're still not listening on the public internet,
+04:49 we're just listening on local host.
+04:52 Okay, but this is one step in the right path,
+04:55 we've got basically the firewall here restricting access to everything,
+05:00 except for wide open ssh and MongoDB
+05:05 on a default port only from the web server.
+05:08 Let's while we're over here go ahead and do this as well.
+05:11 Just assuming that you're treating this as your web server,
+05:14 let's go ahead do the same thing.
+05:18 So by default we're going to do deny incoming allow outgoing,
+05:23
+05:28 allow ssh, and let's say allow 80 and 443 to simulate this being the web server,
+05:37 we're not actually going to run a website, like I said,
+05:40 but that is what I would do, and then we would do an enable.
+05:44 It says are you sure you want to do this, we'll exit one more time,
+05:46 make sure we can get back, and we can, fabulous.
+05:49 So now, we've got that server sort of foul lock down just to play along,
+05:54 this one is like actually lock down and this thing can talk to it,
+05:57 but this one is not listening.
+05:59 I don't want to make that one listen, until we go through a few other steps,
+06:01 so you are going to have to hold off on having this whole connection thing working.

transcripts/ch9-deployment/8.txt

@@ -0,0 +1,32 @@
+00:01 Limiting network exposure in concepts, so what do we do?
+00:04 First of all, I said listening on the default port is just crazy
+00:07 because people are going to be scanning that like nobody's business,
+00:10 they may scan every port on your machine, connect to it,
+00:13 somehow distinguish it's a MongoDB port,
+00:15 but chances are that's not going to happen,
+00:18 chances are people are just going to check the few ports
+00:21 and just move on to scanning millions or billions of other ip addresses,
+00:24 even if they do connect, we're going to have
+00:27 some additional layers of security there,
+00:29 hopefully the firewall makes all of this redundant.
+00:32 But still, it's a good idea to just have layers of security
+00:35 so here we have a port that is non default, 10001.
+00:39 Now, we're also going to turn on our firewall
+00:43 so in fact it's very unlikely anyone can get to that from outside of our data center
+00:48 other than the apps or the servers that we said explicitly they can get to it.
+00:53 So by default, deny all incoming connections, allow all outgoing connections
+00:57 allow ssh so that we can get back in
+01:00 or this is going to be the last time we ever see this server,
+01:03 so we're going to allow ssh and then we're going to enable it,
+01:06 that's the moment of truth, it says are you sure
+01:08 I suggest doing this right away before you have lots of important data on the server.
+01:12 And then we're also going to allow from the ip address
+01:15 that is the application that's based upon MongoDB,
+01:19 and then to any port this one here.
+01:23 We've got our farewell set up, we've got MongoDB set up
+01:26 to be non default of firewall rules, reflect that non default port.
+01:30 So this is the web app address, this is the configured MongoDB port
+01:36 this, we're not ready for listening on the internet yet.
+01:40 Two more things, encryption of the connection,
+01:43 which within the same data center may be it doesn't matter
+01:45 but we're going to add it anyway and authentication.

transcripts/ch9-deployment/9.txt

@@ -0,0 +1,68 @@
+00:00 For our MongoDB server we want to add
+00:02 communication level encryption, basically ssl.
+00:05 Now we could go get a proper ssl certificate, we could even use let's encrypt,
+00:10 but because this is only talked to from our few servers
+00:14 we can actually just do an auto generated one, so a self signed certificate.
+00:18 Let's go over here to etc/ssl, let's see what's here—
+00:22 not so much, alright, so the next thing that we want to do
+00:26 is we want to run open ssl to generate this key.
+00:29 Now, I'm going to give you a link that you can copy this from
+00:32 so don't worry about trying to type this in,
+00:34 so notice it's good for 365 days,
+00:36 we could put way more on here if we really wanted,
+00:43 save yourself some trouble,
+00:46 and it's going to be output into these two a public and private key. Let's go.
+00:53 Then you can basically put whatever you want, I'll go in and put some stuff here
+00:58
+01:03 okay, so I entered some, sort of, kind of accurate data,
+01:07 and now we have our two keys, out two MongoDB public and private keys,
+01:11 the next thing is to generate a pem file
+01:15 which is really just the combination of the public and private key
+01:19 and we could do that with a cat command like this,
+01:23 so we run this, and now we've got the private key and the certificate there, okay great.
+01:32 Now, the next thing to do is actually tell MongoDB
+01:36 hey, I would like you to use encryption
+01:39 and I would like you to use this particular key
+01:43 so notice, we're over here in the etc/ssl,
+01:46 and we're going to get that mongodb.pem we just got,
+01:50 so let's edit the config here, we'll go under security
+01:57
+02:02 oh actually sorry, it's not under security, not yet, we're going to be there in a minute,
+02:07 we want to go to network here, and we're going to say ssl
+02:10 say mode is require ssl like so, not model, mode
+02:18 and the pem key file like this is going to be /etc/ssl/mongo.pem
+02:30 Okay, so make sure we save that, and then we just have to restart mongo
+02:35 so service mongod restart, let's see if that went well.
+02:44 It doesn't look so great, does it? Well, why is that?
+02:49 let me grab our little log file here, there's our log file
+02:57
+03:00 ah so, it says here's the error, etc/ssl/mongo.pem file not found
+03:05 now I can just edit this out of the video right and we would skip it,
+03:07 but I kind of want to show you like oh jeez,
+03:10 what do you do when something goes wrong?
+03:12 Well, you go to look at the log file, first of all you can quickly ask
+03:15 on the status and it'll say crash something bad, go look at the log file
+03:21 and then go from there, maybe you want to tail it in a real production thing.
+03:26 So we are just going to edit this again and say you know what, you're right,
+03:30 I believe that's mongodb, so we'll restart it
+03:38 ask for the status and hey, look, a running process, super, that is so much better.
+03:44 Okay, so let's try to connect to this on the same machine here
+03:48 so we tried Mongo, and it said no, no, no you can't find it there
+03:52 so we did the port 10001, and it said I can't connect to this,
+03:58 this is not so good, I'm not sure what this error message is
+04:01 but we need to basically say one more thing,
+04:05 we need to say allow invalid ssl certificates
+04:08 because it doesn't trust itself and use ssl;
+04:11 there we go, so you can see this network error while attempting to run is master
+04:16 basically said I tried to run an unencrypted command on an encrypted connection
+04:21 and I got junk back— yeah, because it was encrypted.
+04:24 Now we're kind of talking to the server on its non default port
+04:28 using its non valid ssl certificate,
+04:31 you can generate valid ones if you want, you can use other things lets encrypt,
+04:34 you can buy them, whatever, but like I said it's probably fine to use this.
+04:39 We're very close to coming over here,
+04:41 and coming down and changing this to 0000 which will allow our web app to talk
+04:51 so we have the encryption of a communication that's good,
+04:54 but still, this is not good enough, what we need to be able to do is
+04:58 restrict this to only people with username and password
+05:02 and because we're doing this over ssl that password exchange is relatively safe.