Amazon Redshift is the preferred and quickest cloud knowledge warehouse, providing seamless integration together with your knowledge lake and different knowledge sources, as much as 3 times quicker efficiency than another cloud knowledge warehouse, automated upkeep, separation of storage and compute, and as much as 75% decrease value than another cloud knowledge warehouse. This publish explores totally different architectures and use circumstances that concentrate on maximizing knowledge availability, utilizing Amazon Redshift because the core knowledge warehouse platform.

Within the trendy data-driven group, many knowledge analytics use circumstances utilizing Amazon Redshift have more and more advanced to imagine a important enterprise profile. These use circumstances are actually required to be extremely resilient with little to no downtime. For instance, analytical use circumstances that after relied solely on historic knowledge and produced static forecasts are actually anticipated to repeatedly weave real-time streaming and operational knowledge into their ever-updating analytical forecasts. Machine studying (ML) use circumstances that relied on in a single day batch jobs to extract buyer churn predictions from extraordinarily massive datasets are actually anticipated to carry out those self same buyer churn predictions on demand utilizing each historic and intraday datasets.

This publish is a component one in every of a sequence discussing excessive resiliency and availability with Amazon Redshift. On this publish, we focus on a various set of standard analytical use circumstances which have historically or maybe extra just lately assumed a important enterprise profile. The purpose of this publish is to indicate the artwork of the doable with excessive resiliency use circumstances. For every use case, we offer a quick description and discover the explanations for its important enterprise profile, and supply a reference structure for implementing the use case following greatest practices. Within the following part, we embody a quick point out of a number of the complimentary excessive resiliency options in Amazon Redshift as they apply for every use case.

Within the ultimate part of this publish, we broaden the scope to debate excessive resiliency in a knowledge ecosystem that makes use of Amazon Redshift. Specifically, we focus on the Lake Home Structure within the excessive resiliency context.

Half two of this sequence (coming quickly) offers a deeper look into the person excessive resiliency and availability options of Amazon Redshift.

Now let’s discover a number of the hottest use circumstances which have historically required excessive resiliency or have come to require excessive resiliency within the trendy data-driven group.

Knowledge analytics as a service

Many analytical use circumstances give attention to extracting worth from knowledge collected and produced by a corporation to serve the group’s inner enterprise and operational targets. In lots of circumstances, nonetheless, the info collected and produced by a corporation can itself be packaged and supplied as a product to different organizations. Extra particularly, entry to the info collected and produced together with analytical capabilities is often supplied as a paid service to different organizations. That is known as knowledge analytics as a service (DaaS).

For instance, think about a advertising company that has amassed demographic info for a geographic location corresponding to inhabitants by age, revenue, and household construction. Such demographic info usually serves as an important enter for a lot of organizations’ choices to establish the perfect location for enlargement, match their merchandise with doubtless patrons, product choices, and lots of different enterprise wants. The advertising company can provide entry to this demographic info as a paid service to a mess of shops, healthcare suppliers, resorts, and extra.

A number of the most important points of DaaS choices are ease of administration, safety, cost-efficiency, workload isolation, and excessive resiliency and availability. For instance, the advertising company providing the DaaS product wants the power to simply refresh the demographic knowledge on a daily cadence (ease of administration), guarantee paying clients are capable of entry solely approved knowledge (safety), decrease knowledge duplication to keep away from runaway prices and maintain the DaaS competitively priced (cost-efficiency), guarantee a constant efficiency profile for paying clients (workload isolation), and guarantee uninterrupted entry to the paid service (excessive availability).

By housing the info in a number of Amazon Redshift clusters, organizations can use the service’s knowledge sharing capabilities to make such DaaS patterns doable in an simply manageable, safe, cost-efficient, and workload-isolated method. Paying clients are then capable of entry the info utilizing the highly effective search and aggregation capabilities of Amazon Redshift. The next structure diagram illustrates a generally used reference structure for this state of affairs.

The next diagram illustrates one other reference structure that gives excessive resiliency and availability for inner and exterior customers of the info.

Whereas an in-depth dialogue of the info sharing capabilities in Amazon Redshift is past the scope of this publish, consult with the next assets for extra info:

Recent forecasts

As the ability of the trendy knowledge ecosystem is unleashed, analytical workloads that historically yielded point-in-time stories primarily based solely on historic datasets are evolving to include knowledge in real-time and produce on-demand evaluation.

For instance, occasion coordinators which will have needed to rely solely on historic datasets to create analytical gross sales forecasts in enterprise intelligence (BI) dashboards for upcoming occasions are actually ready to make use of Amazon Redshift federated queries to include dwell ticket gross sales saved in operational knowledge shops corresponding to Amazon Aurora or Amazon Relational Database Service (Amazon RDS). With federated queries, occasion coordinators can now have their analytical workloads operating on Amazon Redshift question and incorporate operational knowledge corresponding to dwell ticket gross sales saved in Aurora on demand in order that BI dashboards mirror probably the most up-to-date ticket gross sales.

Organising federated queries is achieved by creating exterior tables that reference the tables of curiosity in an RDS occasion. The next reference structure illustrates one easy solution to obtain federated queries utilizing two totally different variations of Aurora.

Whereas an in-depth dialogue of federated question capabilities in Amazon Redshift is past the scope of this publish, consult with the next assets for extra info:

ML-based predictions

The multitude of ML-based predictive use circumstances and in depth analytical capabilities supplied throughout the AWS ecosystem has positioned ML in an ever-prominent and demanding function inside data-driven organizations. This may very well be retailers trying to predict buyer churn, healthcare insurers trying to predict the variety of claims within the subsequent 30 days, monetary companies organizations working to detect fraud or managing their market danger and publicity, and extra.

Amazon Redshift ML offers seamless integration to Amazon SageMaker for coaching ML fashions as usually as obligatory utilizing knowledge saved in Amazon Redshift. Redshift ML additionally offers the power to weave on-demand, ML-based predictions instantly into Amazon Redshift analytical workloads. The benefit with which ML predictions can now be utilized in Amazon Redshift has paved the trail to analytical workloads or BI dashboards that both use or focus on ML-based predictions, and which are relied on closely by operations groups, enterprise groups, and lots of different customers.

For instance, retailers could have historically relied on ML fashions that had been skilled in a periodic cadence, maybe weekly or another prolonged interval, to foretell buyer churn. Quite a bit can change, nonetheless, throughout these coaching intervals, rendering the retailer’s capability to foretell buyer churn much less efficient. With Redshift ML, retailers are actually capable of practice their ML fashions utilizing their most up-to-date knowledge inside Amazon Redshift and incorporate ML predictions instantly within the Amazon Redshift analytical workloads used to energy BI dashboards.

The next reference structure demonstrates the usage of Redshift ML features in numerous analytical workloads. With ANSI SQL instructions, you should utilize Amazon Redshift knowledge to create and practice an ML mannequin (Amazon Redshift makes use of SageMaker) that’s then made accessible by means of an Amazon Redshift operate. That operate can then be utilized in numerous analytical workloads.

Whereas an in-depth dialogue of Redshift ML is past the scope of this publish, consult with the next assets for extra info:

Manufacturing knowledge for dev environments

Getting access to high-quality take a look at knowledge is likely one of the commonest challenges encountered within the growth course of. To take care of entry to high-quality take a look at knowledge, builders should usually overcome hurdles corresponding to excessive administrative overhead for replicating knowledge, elevated prices from knowledge duplication, extended downtime, and danger of dropping growth artifacts when refreshing take a look at environments.

The knowledge sharing characteristic permits Amazon Redshift growth clusters to entry high-quality manufacturing knowledge instantly from an Amazon Redshift manufacturing or pre-production cluster in an easy, safe, and cost-efficient strategy that achieves a extremely resilient posture.

For instance, you possibly can set up a knowledge share on the Amazon Redshift manufacturing cluster that securely exposes solely the schemas, tables, or views applicable for growth environments. The Amazon Redshift growth cluster can then use that knowledge share to question the high-quality manufacturing knowledge instantly the place it’s endured on Amazon Easy Storage Service (Amazon S3), with out impacting the Amazon Redshift manufacturing cluster’s compute capability. As a result of the event cluster makes use of its personal compute capability, the manufacturing cluster’s excessive resiliency and availability posture is insulated from long-running experimental or growth workloads. Likewise, growth workloads are insulated from competing for compute assets on the manufacturing cluster.

As well as, querying the high-quality manufacturing knowledge through the manufacturing cluster’s knowledge share avoids pointless knowledge duplication that may result in greater storage prices. Because the manufacturing knowledge adjustments, the event cluster mechanically positive aspects entry to the newest high-quality manufacturing knowledge.

Lastly, for growth options that require schema adjustments, builders are free to create customized schemas on the event cluster which are primarily based on the high-quality manufacturing knowledge. As a result of the manufacturing knowledge is decoupled from the event cluster, the customized schemas are situated solely on the event cluster, and the manufacturing knowledge will not be impacted in any approach.

Let’s discover two instance reference architectures that you should utilize for this use case.

Manufacturing knowledge for dev environments utilizing current-generation Amazon Redshift occasion sorts

With the native Amazon Redshift knowledge sharing accessible with the present technology of Amazon Redshift occasion sorts (RA3), we are able to use a comparatively easy structure to allow dev environments with the freshest high-quality manufacturing knowledge.

Within the following structure diagram, the manufacturing cluster takes on the function of a producer cluster, as a result of it’s the cluster producing the info of curiosity. The event clusters tackle the function of the buyer cluster as a result of they’re the clusters taken with accessing the produced knowledge. Observe that the producer and shopper roles are merely labels to make clear the totally different function of every cluster, and never a proper designation inside Amazon Redshift.

Manufacturing knowledge for dev environments utilizing previous-generation Amazon Redshift occasion sorts

Once we mentioned this use case, we relied solely on the native knowledge sharing functionality in Amazon Redshift. Nevertheless, should you’re utilizing the earlier technology Amazon Redshift occasion varieties of dense compute (DC) and dense storage (DS) nodes in your manufacturing environments, you must make use of a barely totally different implementation of this use case, as a result of native Amazon Redshift knowledge sharing is obtainable just for the present technology of Amazon Redshift occasion sorts (RA3).

First, we use a snapshot of the dense compute or dense storage manufacturing cluster to revive the manufacturing surroundings to a brand new RA3 cluster that has the newest manufacturing knowledge. Let’s name this cluster the dev-read cluster to emphasise that this cluster is just for read-only functions and doesn’t exhibit any knowledge modifications. As well as, we are able to arise a second RA3 cluster that merely serves as a sandbox for builders with knowledge shares established to the dev-read cluster. Let’s name this cluster the dev-write cluster, as a result of its most important objective is to function a learn/write sandbox for builders and broader growth work.

The next diagram illustrates this setup.

One of many key advantages of getting a separate dev-read and dev-write cluster is that the dev-read cluster may be swapped out with a brand new RA3 cluster containing brisker manufacturing knowledge, with out wiping out the entire potential growth artifacts created by builders (saved procedures for debugging, modified schemas, elevated privileges, and so forth). This resiliency is a vital profit for a lot of growth groups which may in any other case considerably delay refreshing their growth knowledge just because they don’t need to lose their testing and debugging artifacts or broader growth settings.

For instance, if the event workforce desires to refresh the manufacturing knowledge within the dev-read cluster on the primary of each month, then each month you might rename the present dev-read cluster to dev-read-old, and use the newest manufacturing snapshot to create a brand new dev-read RA3 cluster. You additionally should reestablish the info share setup between the dev-write and dev-read clusters together with the dev-read cluster swap, however this process may be automated pretty simply and shortly utilizing a lot of approaches.

One other key profit is that the dev-read cluster doesn’t exhibit any load past the preliminary snapshot restoration, so it may be a easy two-node ra3.xlplus cluster to attenuate value, whereas the dev-write cluster may be extra appropriately sized for growth workloads. In different phrases, there’s minimal extra value with this setup vs. utilizing single growth cluster.

Whereas an in-depth dialogue of Amazon Redshift’s knowledge sharing capabilities is past the scope of this publish, consult with the next assets for extra info:

Streaming knowledge analytics

With integration between the Amazon Kinesis household of companies and Amazon Redshift, you will have a straightforward and dependable solution to load streaming knowledge into knowledge lakes in addition to analytics companies. Amazon Kinesis Knowledge Firehose micro-batches real-time streaming messages and hundreds these micro-batches into the designated desk inside Amazon Redshift. With a number of clicks on the Kinesis Knowledge Firehose console, you possibly can create a supply stream that may ingest streaming knowledge from tons of of sources to a number of locations, together with Amazon Redshift. Ought to there be any interruptions in publishing streaming messages to Amazon Redshift, Kinesis Knowledge Firehose mechanically makes an attempt a number of retries, and you’ll configure and customise that retry habits.

You can too configure Amazon Kinesis Knowledge Streams to transform the incoming knowledge to open codecs like Apache Parquet and ORC earlier than knowledge is delivered to Amazon Redshift for optimum question efficiency. You possibly can even dynamically partition your streaming knowledge utilizing well-defined keys like customer_id or transaction_id. Kinesis Knowledge Firehose teams knowledge by these keys and delivers into key-unique S3 prefixes, making it simpler so that you can carry out high-performance, cost-efficient analytics in Amazon S3 utilizing Amazon Redshift and different AWS companies.

The next reference structure reveals one of many easy approaches to integrating Kinesis Knowledge Firehose and Amazon Redshift.

Whereas an in-depth dialogue of Kinesis Knowledge Firehose and integration with Amazon Redshift are past the scope of this publish, consult with the next assets for extra info:

Change knowledge seize

Whereas Amazon Redshift federated question permits Amazon Redshift to instantly question knowledge saved in an operational knowledge retailer corresponding to Aurora, there are additionally occasions when it helps for a few of that operational knowledge to be solely replicated to Amazon Redshift for a mess of different analytical use circumstances, corresponding to knowledge refinement.

After an preliminary replication from the operational knowledge retailer to Amazon Redshift, ongoing change knowledge seize (CDC) replications are required to maintain Amazon Redshift up to date with subsequent adjustments that occurred on the operational knowledge retailer.

With AWS Database Migration Service (AWS DMS), you possibly can mechanically replicate adjustments in an operational knowledge retailer corresponding to Aurora to Amazon Redshift in an easy, cost-efficient, safe, and extremely resilient and accessible strategy. As knowledge adjustments on the operational knowledge retailer, AWS DMS mechanically replicates these adjustments to the designated desk on Amazon Redshift.

The next reference structure illustrates the easy use of AWS DMS to copy adjustments in an operational knowledge retailer corresponding to Amazon Aurora, Oracle, SQL Server, and so forth. to Amazon Redshift and different locations corresponding to Amazon S3.

Whereas an in-depth dialogue of AWS DMS is past the scope of this publish, consult with the next assets for extra info:

Workload isolation

Sharing knowledge can enhance the agility of your group by encouraging extra connections and fostering collaboration, which permits groups to construct upon the work of others somewhat than repeat already present processes. Amazon Redshift does this by providing you with on the spot, granular, and high-performance entry to knowledge throughout Amazon Redshift clusters while not having you to manually copy or transfer your knowledge. You have got dwell entry to knowledge so your customers can see probably the most up-to-date and constant info because it’s up to date in Amazon Redshift clusters.

Amazon Redshift parallelizes queries throughout the totally different nodes of a cluster, however there could also be circumstances if you need to permit extra concurrent queries than one cluster can present or present workload separation. You should utilize knowledge sharing to isolate your workloads, thereby minimizing the chance {that a} impasse scenario in a single workload impacts different workloads operating on the identical cluster.

The normal strategy to excessive resiliency and availability is to deploy two or extra similar, unbiased, and parallel Amazon Redshift clusters. Nevertheless, this design requires that every one database updates be carried out on all Amazon Redshift clusters. This introduces complexity in your general structure. On this part, we show the right way to use knowledge sharing to design a extremely resilient and accessible structure with workload isolation.

The next diagram illustrates the high-level structure for knowledge sharing in Amazon Redshift.

This structure helps totally different sorts of business-critical workloads, corresponding to utilizing a central extract, remodel, and cargo (ETL) cluster that shares knowledge with a number of analytic or BI clusters. This strategy offers BI workload isolation, so particular person BI workloads don’t impression the efficiency of the ETL workloads and vice-versa. You possibly can scale the person Amazon Redshift cluster compute assets in accordance with the workload-specific necessities of value and efficiency.

Amazon Redshift Spectrum is a characteristic of Amazon Redshift that allows you to run queries towards exabytes of unstructured knowledge in Amazon S3, with no loading or ETL required. You should utilize your producer cluster to course of the Amazon S3 knowledge and unload the ensuing dataset again to Amazon S3. Then arrange as many Amazon Redshift shopper clusters as it’s essential question your Amazon S3 knowledge lake, thereby offering excessive resiliency and availability, and limitless concurrency.

Extremely accessible knowledge ecosystem utilizing Amazon Redshift

On this part, we delve slightly deeper into the Lake Home Structure, which achieves a variety of greatest practices whereas offering a number of excessive resiliency and availability advantages that complement Amazon Redshift.

Within the trendy knowledge ecosystem, many data-driven organizations have achieved great success using a Lake Home Structure to course of the ever-growing quantity, velocity, and number of knowledge. As well as, the Lake Home Structure has helped these data-driven organizations obtain larger resiliency.

As the next diagram reveals, the Lake Home Structure consists of a knowledge lake serving as the one supply of reality with totally different compute layers corresponding to Amazon Redshift sitting atop the info lake (in impact constructing a home on the lake, therefore the time period “lake home”).

Organizations can use a knowledge lake to maximise knowledge availability by centrally storing the info within the sturdy Amazon S3 layer however get hold of entry from a number of AWS merchandise. Separation of compute and storage affords a number of resiliency and availability benefits. An information lake offers these identical benefits however from a heterogeneous set of companies that may all entry a standard knowledge layer. Utilizing Amazon Redshift with a Lake Home Structure reinforces the lake home’s excessive resiliency and availability. Moreover, with the seamless integration of Amazon Redshift with the S3 knowledge lake, you should utilize Redshift Spectrum to run ANSI SQL queries inside Amazon Redshift that instantly reference exterior tables within the S3 knowledge lake, as is usually accomplished with chilly knowledge (knowledge that’s occasionally accessed).

As well as, there are a mess of easy companies corresponding to AWS Glue, AWS DMS, and AWS Lambda that you should utilize to load heat knowledge (knowledge that’s regularly accessed) from an S3 knowledge lake to Amazon Redshift for larger efficiency.

Conclusion

On this publish, we explored a number of analytical use circumstances that require excessive resiliency and availability and supplied an outline of the Amazon Redshift options that assist fulfill these necessities. We additionally offered a number of instance reference architectures for these use circumstances in addition to a knowledge ecosystem reference structure that gives a variety of advantages and reinforces excessive resiliency and availability postures.

For additional info on excessive resiliency and availability inside Amazon Redshift or implementing the aforementioned use circumstances, we encourage you to succeed in out to your AWS Options Architect—we sit up for serving to.

Concerning the Authors

Asser Moustafa is an Analytics Specialist Options Architect at AWS primarily based out of Dallas, TX, USA. He advises clients within the Americas on their Amazon Redshift and knowledge lake architectures and migrations, ranging from the POC stage to precise manufacturing deployment and upkeep.

Milind Oke is a Knowledge Warehouse Specialist Options Architect primarily based out of New York. He has been constructing knowledge warehouse options for over 15 years and focuses on Amazon Redshift. He’s centered on serving to clients design and construct enterprise-scale well-architected analytics and resolution assist platforms.