(I am an Amazon employee, but the viewpoints expressed here are my personal opinion.)
The inevitable happened, Hundreds of millions of users moved their data from local computers, mobile devices, and on-premise storage arrays to the cloud. This mass-scale movement has created a seminal moment in history with large distributed data lakes operating on the cloud. Because of this diversity, we are at the cusp of witnessing next-generation of finely tuned innovation throughout the data storage eco-system. From component suppliers like NAND to device manufacturers building HDD, SSD, Tapes to data storage software providers, new opportunities abound to build innovative category-defining products.
One may question why now? The answer lies lurking under the covers. When we talk about cloud data storage we tend to discuss convenience, scale, and IT savings (at least in the short run). But somehow we forget to talk about incredible insights gained from owning a large customer base. Storing tens of dozens of exabytes (EB) of data from an incredibly diverse set of customers allows an incredible view into nature i.e. the characteristics of the stored data – life cycle, access patterns, compressibility, performance, and much more.
The insights naturally lend itself to bucketing data logically into many differentiated Classes. Each class with its own set of characteristics. Now, what really sets apart the cloud providers compared to anyone else are two very important attributes. The number of Classes and the EB’s of data pooled into each class. It is really important to appreciate why these are truly important and decisive. First, the increasing number of classes allows the industry an opportunity to think and invent new types of storage media and devices, each customized and cost-optimized to the characteristic of that data class. Secondly, which is supercritical is that EB’s stored in each class translates directly to the hundreds of millions of dollars of new revenue for the downstream ecosystem. It incentivizes and opens the door for the industry to fund research and commercialization of class-based storage products.
Cloud providers like AWS, continue to create new storage classes. There is classification not only in terms of the block (EBS) and object storage (S3), but further segregation based on the characteristics like performance and access frequency of the data with S3-Infrequent access, S3-Glacier, and S3-Glacier Deep Archive classes. Each class benefiting from the unique characteristics of the underlying storage media, devices, and systems. In my opinion, the trend is clear and will continue to grow in the future, creating an opportunity for customers to reduce their storage costs and for the storage eco-system suppliers to create more compelling cost targeted storage products.
We are uniquely placed at the edge of an innovative age for storage.