Want to learn more? Take the full course at at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. ---
Hello again. Excellent work on the exercises! In this last video of this chapter, we're going to talk about cloud computing. You've probably already heard people use this term before. Data engineers are heavy users of the cloud. In this video, we'll explain why.
Let's take data processing as an example. You've seen in the previous video that data processing often runs on clusters of machines.
In the past, companies that relied on data processing owned their own data center. You can imagine racks of servers, ready to be used. The electrical bill and maintenance were also at the company's cost. Moreover, companies needed to be able to provide enough processing power for peak moments. That also meant that at quieter times, much of the processing power remained unused.
It's this waste of resources that made cloud computing so appealing. In the cloud, you use the resources you need, at the time you need them. You can see that once these cloud services came to be, many companies moved to the cloud as a way of cost optimization.
Apart from the costs of maintaining data centers, another reason for using cloud computing is database reliability. If you run a data-critical company, you have to prepare for the worst. Don't ask yourself the question "will disaster strike?" but rather ask yourself "when will disaster strike?"
For example, a fire can break out in your data center. To be safe, you need to replicate your data at a different geographical location. That brings along a bunch of logistical problems of its own.
Out of these needs, companies specializing in these kinds of issues were born. We call these companies "cloud service providers" now.
In this slide, we'll talk about three big players in the cloud provider market. First and foremost, there's Amazon Web Services or AWS. Think about the last few websites you visited. Chances are AWS hosts at least a few of them. Back in 2017, AWS had an outage, it reportedly 'broke' the internet. That's how big AWS is. While AWS took up 32% of the market share in 2018, Microsoft Azure is the second big player and took 17% of the market. The third big player, is Google Cloud, and held 10% of the market in 2018.
So we talked about the big players. However, what do they provide? We'll discuss three types of services these companies offer: Storage, Computation, and Databases.
First, storage services allow you to upload files of all types to the cloud. In an online store for example, you could upload your product images to a storage service. Storage services are typically very cheap since they don't provide much functionality other than storing the files reliably. AWS hosts S3 as a storage service. Azure has Blob Storage, and Google has Cloud Storage.
Second, computation services allow you to perform computations on the cloud. Usually, you can start up a virtual machine and use it as you wish. It's often used to host web servers, for example. Computation services are usually flexible, and you can start or stop virtual machines as needed. AWS has EC2 as a computation service, Azure has Virtual Machines, and Google has Compute Engine.
Last but not least, cloud providers host databases. We already talked about databases in the previous video, so you know what they are. For SQL databases, AWS has RDS. Azure has SQL Database, and Google has Cloud SQL.
That's it for this video, good luck with the exercises.
0 Comments