In the era of on-premise systems, data-warehouses had grown to prominence with their excellent capabilities to store and process massive data. As they use high-end servers and storage devices to process and store data, these systems are huge and expensive. Often they claim scalability, but the scalability is limited to the ability to scale up data warehouses in one-off exercise to add more machines in architecture. In other words, it is not a run-time activity. So, the set-up needs to be at least sufficient to the maximum load on the warehouse, leaving a large portion of the infrastructure unused even when load is not at its peak.
Cloud has solved the problem of massive infrastructure set-up and high cost to quite an extent. One can pay for infrastructure as per use, avoiding requirement of large fixed cost, need to estimate the future requirements (as per growth) and paying for idle infrastructure. Hence it’s no wonder many are moving to cloud from on-premise set-ups. But cloud also poses many limitations. Even with cloud infrastructure, the requirement has to be clear in advance wherein the processing or storage will be limited to the capability of what has been rented. Though, this infrastructure can be increased or decreased as per need by asking the IaaS provider, but there are limitations on doing it on the fly.
There is another breed of data warehouses which solve this problem. These new type of DW are being named as Elastic data warehouses or Dynamic warehouses, provided by companies like Snowflake. They have an entirely new architecture wherein interfaces, cloud service management, database servers and storage are all kept in different tiers making each tier scalable without impacting other tiers. They provide the flexibility to scale-up or scale-down as per the need and on the fly. So, if estimated storage requirement is 20 TB, but it increases to 21 TB, there is no optimization or compromise needed to restrict the data size to 20 TB. The elastic DW will automatically increase the storage allocated on the fly. Similarly, if more processing power is required for some complex queries or to handle peak-day load, additional infrastructure is added on the fly to increase the overall processing capabilities.
Elastic DWs are offering a very tempting alternative for enterprises to move to cloud by removing some of the critical limitations of cloud data-warehouses. Commercially too, it makes all the sense to move to such systems as they can be instrumental in saving cost. However, the issue of data security, confidentiality still remains a concern, despite most of the providers claiming policies and practices to maintain data privacy and confidentiality.