Tuesday, November 29, 2022
HomeBig DataHow GoDaddy constructed an information mesh to decentralize knowledge possession utilizing AWS...

How GoDaddy constructed an information mesh to decentralize knowledge possession utilizing AWS Lake Formation

This can be a visitor put up co-written with Ankit Jhalaria from GoDaddy.

GoDaddy is empowering on a regular basis entrepreneurs by offering all the assistance and instruments to succeed on-line. With greater than 20 million prospects worldwide, GoDaddy is the place individuals come to call their thought, construct knowledgeable web site, entice prospects, and handle their work.

GoDaddy is a data-driven firm, and getting significant insights from knowledge helps them drive enterprise choices to thrill their prospects. In 2018, GoDaddy started a big infrastructure revamp and partnered with AWS to innovate quicker than ever earlier than to fulfill the wants of its buyer progress all over the world. As a part of this revamp, the GoDaddy Knowledge Platform workforce wished to set the corporate up for long-term success by making a well-defined knowledge technique and setting targets to decentralize the possession and processing of knowledge.

On this put up, we focus on how GoDaddy makes use of AWS Lake Formation to simplify safety administration and knowledge governance at scale, and allow knowledge as a service (DaaS) supporting organization-wide knowledge accessibility with cross-account knowledge sharing utilizing an information mesh structure.

The problem

Within the huge ocean of knowledge, deriving helpful insights is an artwork. Previous to the AWS partnership, GoDaddy had a shared Hadoop cluster on premises that varied groups used to create and share datasets with different analysts for collaboration. Because the groups grew, copies of knowledge began to develop within the Hadoop Distributed File System (HDFS). A number of groups began to construct tooling to handle this problem independently, duplicating efforts. Managing permissions on these knowledge belongings grew to become tougher. Making knowledge discoverable throughout a rising variety of knowledge catalogs and techniques is one thing that had began to grow to be a giant problem. Though the price of storage today is comparatively cheap, when there are a number of copies of the identical knowledge asset obtainable, it makes it tougher for analysts to effectively and reliably use the information obtainable to them. Enterprise analysts want sturdy pipelines on key datasets that they rely on to make enterprise choices.

Resolution overview

In GoDaddy’s knowledge mesh hub and spoke mannequin, a central knowledge catalog accommodates details about all the information merchandise that exist within the firm. In AWS terminology, that is the AWS Glue Knowledge Catalog. The information platform workforce gives APIs, SDKs, and Airflow Operators as parts that completely different groups use to work together with the catalog. Actions corresponding to updating the metastore to replicate a brand new partition for a given knowledge product, and infrequently operating MSCK restore operations, are all dealt with within the central governance account, and Lake Formation is used to safe entry to the Knowledge Catalog.

The information platform workforce launched a layer of knowledge governance that ensures finest practices for constructing knowledge merchandise are adopted all through the corporate. We offer the tooling to help knowledge engineers and enterprise analysts whereas leaving the area consultants to run their knowledge pipelines. With this strategy, we’ve got well-curated knowledge merchandise which can be intuitive and simple to grasp for our enterprise analysts.

A knowledge product refers to an entity that powers insights for analytical functions. In easy phrases, this might seek advice from an precise dataset pointing to a location in Amazon Easy Storage Service (Amazon S3). Knowledge producers are chargeable for the processing of knowledge and creating new snapshots or partitions relying on the enterprise wants. In some instances, knowledge is refreshed each 24 hours, and different instances, each hour. Knowledge shoppers come to the information mesh to eat knowledge, and permissions are managed within the central governance account by Lake Formation. Lake Formation makes use of AWS Useful resource Entry Supervisor (AWS RAM) to ship useful resource shares to completely different client accounts to have the ability to entry the information from the central governance account. We go into particulars about this performance later within the put up.

The next diagram illustrates the answer structure.

Solution architecture illustrated

Defining metadata with the central schema repository

Knowledge is barely helpful if end-users can derive significant insights from it—in any other case, it’s simply noise. As a part of onboarding with the information platform, an information producer registers their schema with the information platform together with related metadata. That is reviewed by the information governance workforce that ensures finest practices for creating datasets are adopted. We’ve automated a few of the commonest knowledge governance evaluation objects. That is additionally the place the place producers outline a contract about dependable knowledge deliveries, also known as Service Stage Goal (SLO). After a contract is in place, the information platform workforce’s background processes monitor and ship out alerts when knowledge producers fail to fulfill their contract or SLO.

When managing permissions with Lake Formation, you register the Amazon S3 location of various S3 buckets. Lake Formation makes use of AWS RAM to share the named useful resource.

When managing sources with AWS RAM, the central governance account creates AWS RAM shares. The information platform gives a customized AWS Service Catalog product to simply accept AWS RAM shares in client accounts.

Having constant schemas with significant names and descriptions makes the invention of datasets simple. Each knowledge producer who’s a site skilled is chargeable for creating well-defined schemas that enterprise customers use to generate insights to make key enterprise choices. Knowledge producers register their schemas together with extra metadata with the information lake repository. Metadata consists of details about the workforce chargeable for the dataset, corresponding to their SLO contract, description, and speak to info. This info will get checked right into a Git repository the place automation kicks in and validates the request to verify it conforms to requirements and finest practices. We use AWS CloudFormation templates to provision sources. The next code is a pattern of what the registration metadata seems like.

Sample code of what the registration metadata looks like

As a part of the registration course of, automation steps run within the background to deal with the next on behalf of the information producer:

  • Register the producer’s Amazon S3 location of the information with Lake Formation – This enables us to make use of Lake Formation for fine-grained entry to manage the desk within the AWS Glue Knowledge Catalog that refers to this location in addition to to the underlying knowledge.
  • Create the underlying AWS Glue database and desk – Based mostly on the schema specified by the information producer together with the metadata, we create the underlying AWS Glue database and desk within the central governance account. As a part of this, we additionally use desk properties of AWS Glue to retailer extra metadata to make use of later for evaluation.
  • Outline the SLO contract – Any business-critical dataset must have a well-defined SLO contract. As a part of dataset registration, the information producer defines a contract with a cron expression that will get utilized by the information platform to create an occasion rule in Amazon EventBridge. This rule triggers an AWS Lambda operate to look at for deliveries of the information and triggers an alert to the information producer’s Slack channel in the event that they breach the contract.

Consuming knowledge from the information mesh catalog

When an information client belonging to a given line of enterprise (LOB) identifies the information product that they’re focused on, they submit a request to the central governance workforce containing their AWS account ID that they use to question the information. The information platform gives a portal to find datasets throughout the corporate. After the request is authorized, automation runs to create an AWS RAM share with the buyer account masking the AWS Glue database and tables mapped to the information product registered within the AWS Glue Knowledge Catalog of the central governance account.

The next screenshot reveals an instance of a useful resource share.

Example of a resource share

The buyer knowledge lake admin wants to simply accept the AWS RAM share and create a useful resource hyperlink in Lake Formation to begin querying the shared dataset inside their account. We automated this course of by constructing an AWS Service Catalog product that runs within the client’s account as a Lambda operate that accepts shares on behalf of shoppers.

When the useful resource linked datasets can be found within the client account, the buyer knowledge lake admin gives grants to IAM customers and roles mapping to knowledge shoppers throughout the account. These shoppers (utility or consumer persona) can now question the datasets utilizing AWS analytics companies of their alternative like Amazon Athena and Amazon EMR primarily based on the entry privileges granted by the buyer knowledge lake admin.

Day-to-day operations and metrics

Managing permissions utilizing Lake Formation is one a part of the general ecosystem. After permissions have been granted, knowledge producers create new snapshots of the information at a sure cadence that may differ from each quarter-hour to a day. Knowledge producers are built-in with the information platform APIs that informs the platform about any new refreshes of the information. The information platform robotically writes a 0-byte _SUCCESS file for each dataset that will get refreshed, and notifies the subscribed client account through an Amazon Easy Notification Service (Amazon SNS) subject within the central governance account. Shoppers use this as a sign to set off their knowledge pipelines and processes to begin processing newer model of the information using an event-driven strategy.

There are over 2,000 knowledge merchandise constructed on the GoDaddy knowledge mesh on AWS. Day-after-day, there are literally thousands of updates to the AWS Glue metastore within the central knowledge governance account. There are a whole lot of knowledge producers producing knowledge each hour in a wide selection of S3 buckets, and hundreds of knowledge shoppers consuming knowledge throughout a wide selection of instruments, together with Athena, Amazon EMR, and Tableau from completely different AWS accounts.

Enterprise outcomes

With the transfer to AWS, GoDaddy’s Knowledge Platform workforce laid the foundations to construct a contemporary knowledge platform that has elevated our velocity of constructing knowledge merchandise and delighting our prospects. The information platform has efficiently transitioned from a monolithic platform to a mannequin the place possession of knowledge has been decentralized. We accelerated the information platform adoption to over 10 traces of enterprise and over 300 groups globally, and are efficiently managing a number of petabytes of knowledge unfold throughout a whole lot of accounts to assist our enterprise derive insights quicker.


GoDaddy’s hub and spoke knowledge mesh structure constructed utilizing Lake Formation simplifies safety administration and knowledge governance at scale, to ship knowledge as a service supporting company-wide knowledge accessibility. Our knowledge mesh manages a number of petabytes of knowledge throughout a whole lot of accounts, enabling decentralized possession of well-defined datasets with automation in place, which helps the enterprise uncover knowledge belongings faster and derive enterprise insights quicker.

This put up illustrates using Lake Formation to construct an information mesh structure that allows a DaaS mannequin for a modernized enterprise knowledge platform. For extra info, see Design an information mesh structure utilizing AWS Lake Formation and AWS Glue.

Concerning the Authors

Ankit Jhalaria is the Director Of Engineering on the Knowledge Platform at GoDaddy. He has over 10 years of expertise working in large knowledge applied sciences. Outdoors of labor, Ankit loves mountain climbing, taking part in board video games, constructing IoT initiatives, and contributing to open-source initiatives.

Harsh Vardhan is an AWS Options Architect, specializing in Analytics. He has over 6 years of expertise working within the discipline of massive knowledge and knowledge science. He’s obsessed with serving to prospects undertake finest practices and uncover insights from their knowledge.

Kyle Tedeschi is a Principal Options Architect at AWS. He enjoys serving to prospects innovate, remodel, and grow to be leaders of their respective domains. Outdoors of labor, Kyle is an avid snowboarder, automotive fanatic, and traveler.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments