Introducing AWS Glue for Ray: Scaling your information integration workloads utilizing Python


AWS Glue is a serverless information integration service that makes it easy to find, put together, transfer, and combine information from a number of sources for analytics, machine studying (ML), and utility improvement. As we speak, AWS Glue processes buyer jobs utilizing both Apache Spark’s distributed processing engine for giant workloads or Python’s single-node processing engine for smaller workloads. Prospects like Python for its ease of use and wealthy assortment of built-in data-processing libraries however would possibly discover it tough for purchasers to scale Python past a single compute node. This limitation makes it tough for purchasers to course of giant datasets. Prospects desire a answer that enables them to proceed utilizing acquainted Python instruments and AWS Glue jobs on information units of all sizes, even these that may’t match on a single occasion.

We’re completely happy to announce the discharge of a brand new AWS Glue job sort: Ray. Ray is an open-source unified compute framework that makes it easy to scale AI and Python workloads. Ray began as an open-source challenge at RISELab in UC Berkeley. In case your utility is written in Python, you possibly can scale it with Ray in a distributed cluster in a multi-node surroundings.  Ray is Python native and you’ll mix it with the AWS SDK for pandas to arrange, combine and remodel your information for working your information analytics and ML workloads together. You need to use AWS Glue for Ray with Glue Studio Notebooks, SageMaker Studio Pocket book, or an area pocket book or IDE of your alternative.

This put up gives an introduction to AWS Glue for Ray and exhibits you find out how to begin utilizing Ray to distribute your Python workloads.

What’s AWS Glue for Ray?

Prospects just like the serverless expertise and quick begin time provided by AWS Glue. With the introduction of Ray, we’ve got ensured that you simply get the identical expertise. We’ve additionally ensured that you should use the AWS Glue job and AWS Glue interactive session primitives to entry the Ray engine. AWS Glue jobs are fire-and-forget methods the place buyer submit their Ray code to the AWS Glue jobs API and AWS Glue mechanically provisions the required compute assets and runs the job. AWS Glue interactive session APIs permit interactive exploration of the info for the aim of job improvement. Whatever the choice used, you’re solely billed throughout the compute used. With AWS Glue for Ray, we’re additionally introducing a brand new Graviton2 primarily based employee (Z.2x) which affords 8 digital CPUs and 64 GB of RAM.

AWS Glue for Ray consists of two main elements:

  1. Ray Core – The distributed computing framework
  2. Ray Dataset – The distributed information framework primarily based on Apache Arrow

When working a Ray job, AWS Glue provisions the Ray cluster for you and runs these distributed Python jobs on a serverless auto-scaling infrastructure.  The cluster in AWS Glue for Ray will consists of precisely one head node and a number of employee nodes.  

The pinnacle node is equivalent to the opposite employee nodes with the exception that it runs singleton processes for cluster administration and the Ray driver course of.  The motive force is a particular employee course of within the head node that runs the top-level utility in Python that begins the Ray job.  The employee node has processes which might be chargeable for submitting and working duties.

The next determine gives a easy introduction to the Ray structure.  The structure illustrates how Ray is ready to schedule jobs by way of processes referred to as Raylets.  The Raylet manages the shared assets on every node and is shared between the concurrently working jobs.  For extra info on how Ray works, see

The next determine exhibits the elements of the employee node and the shared-memory object retailer:

There’s a World Management Retailer within the head node that may deal with every separate machine as nodes, just like how Apache Spark treats staff as nodes.  The next determine exhibits the elements of the top node and the World Management Retailer managing the cluster-level metadata.

AWS Glue for Ray comes included with Ray Core, Ray DatasetModin (distributed pandas) and the AWS SDK for pandas (on Modin) for seamless distributed integration into different AWS providers.  Ray Core is the inspiration of Ray and the essential framework for distributing Python capabilities and lessons. Ray Dataset is a distributed information framework primarily based on Apache Arrow and is most intently analogous to a dataframe in Apache Spark. Modin is a library designed to distribute pandas functions throughout a Ray cluster with none modification and is appropriate with information in Ray Datasets. The included AWS SDK for pandas (previously AWS Information Wrangler) is an abstraction layer on high of Modin to permit for the creation of pandas dataframes from (and writing to) many AWS sources akin to Amazon Easy Storage Service (Amazon S3), Amazon Redshift, Amazon DynamoDB, Amazon OpenSearch Service, and others.

You can too set up your individual ARM appropriate Python libraries through pip, both by way of Ray’s environmental configuration in @ray.distant or through --additional-python-modules.

To study extra about Ray, please go to the GitHub repo.

Why use AWS Glue for Ray?

Many people begin our information journey on AWS with Python, trying to put together information for ML and information science, and transfer information at scale with AWS APIs and Boto3. Ray permits you to carry these acquainted expertise, paradigms, frameworks and libraries to AWS Glue and make them scale to deal with huge datasets with minimal code adjustments. You need to use the identical information processing instruments you at present have (akin to Python libraries for information cleaning, computation, and ML) on datasets of all sizes. AWS Glue for Ray permits the distributed run of your Python scripts over multi-node clusters.

AWS Glue for Ray is designed for the next:

  • Process parallel functions (for instance, while you wish to apply a number of transforms in parallel)
  • Dashing up your Python workload in addition to utilizing Python native libraries.
  • Operating the identical workload throughout a whole lot of knowledge sources.
  • ML ingestion and parallel batch inference on information

Resolution overview

For this put up, you’ll use the Parquet Amazon Buyer Evaluations Dataset saved within the public S3 bucket. The target is to carry out transformations utilizing the Ray dataset after which write it again to Amazon S3 within the Parquet file format.

Configure Amazon S3

Step one is to create an Amazon S3 bucket to retailer the reworked Parquet dataset as the top consequence.

  1. On the Amazon S3 console, select Buckets within the navigation pane.
  2. Select Create bucket.
  3. For Bucket identify, enter a reputation to your Amazon S3 bucket.
  4. Select Create.

Arrange a Jupyter pocket book with an AWS Glue interactive session

For our improvement surroundings, we use a Jupyter pocket book to run the code.

You’re required to put in the AWS Glue interactive classes domestically or run interactive classes with an AWS Glue Studio pocket book. Utilizing AWS Glue Interactive classes will enable you comply with and run the collection of demonstration steps.

Consult with Getting began with AWS Glue interactive classes for directions to spin up a pocket book on an AWS Glue interactive session.

Run your code utilizing Ray in a Jupyter pocket book

This part walks you thru a number of pocket book paragraphs on find out how to use AWS Glue for Ray. On this train, we take a look at the client critiques from the Amazon Buyer Assessment Parquet dataset, carry out some Ray transformations, and write the outcomes to Amazon S3 in a Parquet format.

  1. On Jupyter console, below New, select Glue Python.
  2. Signify you wish to use Ray because the engine by utilizing the %glue_ray magic.
  3. Import the Ray library together with further Python libraries:
    import ray
    import pandas
    import pyarrow
    from ray import information
    import time
    from ray.information import ActorPoolStrategy

  4. Initialize a Ray Cluster with AWS Glue.
  5. Subsequent, we learn a single partition from the dataset, which is Parquet file format:
    begin = time.time()
    ds = ray.information.read_parquet("s3://amazon-reviews-pds/parquet/product_category=Wi-fi/")
    finish = time.time()
    print(f"Studying the info to dataframe: {finish - begin} seconds")

  6. Parquet information retailer the variety of rows per file within the metadata, so we are able to get the whole variety of data in ds with out performing a full information learn:

  7. Subsequent , we are able to examine the schema of this dataset. We don’t must learn the precise information to get the schema; we are able to learn it from the metadata:

  8. We will examine the whole dimension in bytes for the complete Ray dataset:
    #calculate the dimensions in bytes of the complete dataset,  Observe that for Parquet information, this size-in-bytes can be pulled from the Parquet
    #  metadata (not triggering a knowledge learn).

  9. We will see a pattern document from the Ray dataset:
    #Present pattern data from the underlying Parquet dataset  
    begin = time.time()
    finish = time.time()
    print(f"Time taken to point out the info from dataframe : {finish - begin} seconds")

Making use of dataset transformations with Ray

There are primarily two varieties of transformations that may be utilized to Ray datasets:

  • One-to-One transformations – Every enter block will contributes to just one output block, akin to add_column(), map_batches() and drop_column() , and so forth.
  • All-to-All transformations – Enter blocks can contribute to a number of output blocks akin to type() and groupby(), and so forth.

Within the subsequent collection of steps we are going to apply a few of these transformations on our resultant Ray datasets from the earlier part.

  1. We will add a brand new column and examine the schema to confirm the newly added column, adopted by retrieving a pattern document. This transformation is simply obtainable for the datasets that may be transformed to pandas format.
    # Add the given new column to the dataset and present the pattern document after including a brand new column
    begin = time.time()
    ds = ds.add_column( "helpful_votes_ratio", lambda df: df["helpful_votes"] / df["total_votes"])
    finish = time.time()
    print(f"Time taken to Add a brand new columns : {finish - begin} seconds")

  2. Let’s drop just a few columns we don’t want utilizing a drop_columns transformation after which examine the schema to confirm if these columns are dropped from the Ray dataset:
    # Dropping few columns from the underlying Dataset 
    begin = time.time()
    ds = ds.drop_columns(["review_body", "vine", "product_parent", "verified_purchase", "review_headline"])
    finish = time.time()
    print(f"Time taken to drop just a few columns : {finish - begin} seconds")

    Ray datasets have built-in transformations akin to sorting the dataset by the desired key column or key operate.

  3. Subsequent, we apply the kind transformation utilizing one of many columns current within the dataset (total_votes):
    #Kind the dataset by complete votes
    begin = time.time()
    ds =ds.type("total_votes")
    finish = time.time()
    print(f"Time taken for type operation  : {finish - begin} seconds")

  4. Subsequent, we are going to create a Python UDF operate that permits you to write personalized enterprise logic in transformations. In our UDF we’ve got written a logic to search out out the merchandise which might be rated low (i.e. complete votes lower than 100).We create a UDF as a operate on pandas DataFrame batches. For the supported enter batch codecs, see the UDF Enter Batch Format. We additionally show utilizing map_batches() which applies the given operate to the batches of data of this dataset. Map_batches() makes use of the default compute technique (duties), which helps distribute the info processing to a number of Ray staff, that are used to run duties. For extra info on a map_batches() transformation, please see the following documentation.
    # UDF as a operate on pandas DataFrame - To Discover merchandise with total_votes < 100 
    def low_rated_products(df: pandas.DataFrame) -> pandas.DataFrame:
        return df[(df["total_votes"] < 100)]
    #Calculate the variety of merchandise that are rated low when it comes to low votes i.e. lower than 100
    # This system is named Batch inference processing with Ray duties (the default compute technique).
    ds = ds.map_batches(low_rated_products)
    #See pattern data for the merchandise that are rated low when it comes to low votes i.e. lower than 100

    #Rely complete variety of merchandise that are rated low 

  5. You probably have complicated transformations that require extra assets for information processing, we suggest using Ray actors utilizing further configurations with relevant transformations. We’ve demonstrated with map_batches() under:
    # Batch inference processing with Ray actors. Autoscale the actors between 2 and 4.
    class LowRatedProducts:
        def __init__(self):
            self._model = low_rated_products
        def __call__(self, batch: pandas.DataFrame) -> pandas.DataFrame:
            return self._model(batch)
    begin = time.time()
    predicted = ds.map_batches(
        LowRatedProducts, compute=ActorPoolStrategy(2, 4), batch_size=4)
    finish = time.time()

  6. Subsequent, earlier than writing the ultimate resultant Ray dataset we are going to apply map_batches() transformations to filter out the client critiques information the place the whole votes for a given product is bigger than 0 and the critiques belongs to the “US” market solely. Utilizing map_batches() for the filter operation is best when it comes to efficiency compared to filter() transformation.
    # Filter our data with total_votes == 0
    ds = ds.map_batches(lambda df: df[df["total_votes"] > 0])
    # Filter and choose data with market equals US solely
    ds = ds.map_batches(lambda df: df[df["marketplace"] == 'US'])

  7. Lastly, we write the resultant information to the S3 bucket you created in a Parquet file format. You need to use totally different dataset APIs obtainable, akin to write_csv() or write_json() for various file codecs.  Moreover, you possibly can convert the resultant dataset to a different DataFrame sort akin to Mars, Modin or pandas.

Clear up

To keep away from incurring future fees, delete the Amazon S3 bucket and Jupyter pocket book.

  1. On the Amazon S3 console, select Buckets.
  2. Select the bucket you created.
  3. Select Empty and enter your bucket identify.
  4. Select Verify.
  5. Select Delete and enter your bucket identify.
  6. Select Delete bucket.
  7. On the AWS Glue console, select Interactive Periods
  8. Select the interactive session you created.
  9. Select Delete to take away the interactive session.


On this put up, we demonstrated how you should use AWS Glue for Ray to run your Python code in a distributed surroundings.  Now you can run your information and ML functions in a multi-node surroundings.

Consult with the Ray documentation for added info and use instances.

Concerning the authors

Zach Mitchell is a Sr. Massive Information Architect. He works throughout the product crew to boost understanding between product engineers and their clients whereas guiding clients by way of their journey to develop information lakes and different information options on AWS analytics providers.

Ishan Gaur works as Sr. Massive Information Cloud Engineer ( ETL ) specialised in AWS Glue. He’s enthusiastic about serving to clients construct out scalable distributed ETL workloads and implement scalable information processing and analytics pipelines on AWS. When not at work, Ishan likes to cook dinner, journey along with his household, or hearken to music.

Derek Liu is a Options Architect on the Enterprise crew primarily based out of Vancouver, BC.  He’s a part of the AWS Analytics discipline neighborhood and enjoys serving to clients remedy large information challenges by way of AWS analytic providers.

Kinshuk Pahare is a Principal Product Supervisor on AWS Glue.


Please enter your comment!
Please enter your name here