HomeBig DataHow NerdWallet makes use of AWS and Apache Hudi to construct a...

How NerdWallet makes use of AWS and Apache Hudi to construct a serverless, real-time analytics platform

This can be a visitor publish by Kevin Chun, Employees Software program Engineer in Core Engineering at NerdWallet.

NerdWallet’s mission is to offer readability for all of life’s monetary selections. This covers a various set of matters: from selecting the best bank card, to managing your spending, to discovering one of the best private mortgage, to refinancing your mortgage. Consequently, NerdWallet provides highly effective capabilities that span throughout quite a few domains, comparable to credit score monitoring and alerting, dashboards for monitoring internet price and money stream, machine studying (ML)-driven suggestions, and lots of extra for hundreds of thousands of customers.

To construct a cohesive and performant expertise for our customers, we’d like to have the ability to use giant volumes of various person information sourced by a number of impartial groups. This requires a powerful information tradition together with a set of information infrastructure and self-serve tooling that allows creativity and collaboration.

On this publish, we define a use case that demonstrates how NerdWallet is scaling its information ecosystem by constructing a serverless pipeline that allows streaming information from throughout the corporate. We iterated on two totally different architectures. We clarify the challenges we bumped into with the preliminary design and the advantages we achieved through the use of Apache Hudi and extra AWS providers within the second design.

Downside assertion

NerdWallet captures a large quantity of spending information. This information is used to construct useful dashboards and actionable insights for customers. The info is saved in an Amazon Aurora cluster. Despite the fact that the Aurora cluster works properly as an On-line Transaction Processing (OLTP) engine, it’s not appropriate for giant, advanced On-line Analytical Processing (OLAP) queries. Consequently, we are able to’t expose direct database entry to analysts and information engineers. The info homeowners have to resolve requests with new information derivations on learn replicas. As the information quantity and the range of information customers and requests develop, this course of will get tougher to take care of. As well as, information scientists largely require information recordsdata entry from an object retailer like Amazon Easy Storage Service (Amazon S3).

We determined to discover options the place all customers can independently fulfill their very own information requests safely and scalably utilizing open-standard tooling and protocols. Drawing inspiration from the information mesh paradigm, we designed a information lake based mostly on Amazon S3 that decouples information producers from customers whereas offering a self-serve, security-compliant, and scalable set of tooling that’s straightforward to provision.

Preliminary design

The next diagram illustrates the structure of the preliminary design.

The design included the next key elements:

  1. We selected AWS Information Migration Service (AWS DMS) as a result of it’s a managed service that facilitates the motion of information from varied information shops comparable to relational and NoSQL databases into Amazon S3. AWS DMS permits one-time migration and ongoing replication with change information seize (CDC) to maintain the supply and goal information shops in sync.
  2. We selected Amazon S3 as the muse for our information lake due to its scalability, sturdiness, and adaptability. You’ll be able to seamlessly improve storage from gigabytes to petabytes, paying just for what you utilize. It’s designed to offer 11 9s of sturdiness. It helps structured, semi-structured, and unstructured information, and has native integration with a broad portfolio of AWS providers.
  3. AWS Glue is a totally managed information integration service. AWS Glue makes it simpler to categorize, clear, remodel, and reliably switch information between totally different information shops.
  4. Amazon Athena is a serverless interactive question engine that makes it straightforward to investigate information immediately in Amazon S3 utilizing normal SQL. Athena scales robotically—working queries in parallel—so outcomes are quick, even with giant datasets, excessive concurrency, and complicated queries.

This structure works tremendous with small testing datasets. Nevertheless, the crew rapidly bumped into problems with the manufacturing datasets at scale.


The crew encountered the next challenges:

  • Lengthy batch processing time and complexed transformation logic – A single run of the Spark batch job took 2–3 hours to finish, and we ended up getting a reasonably large AWS invoice when testing towards billions of information. The core drawback was that we needed to reconstruct the newest state and rewrite your entire set of information per partition for each job run, even when the incremental modifications have been a single file of the partition. After we scaled that to 1000’s of distinctive transactions per second, we rapidly noticed the degradation in transformation efficiency.
  • Elevated complexity with a lot of purchasers – This workload contained hundreds of thousands of purchasers, and one widespread question sample was to filter by single shopper ID. There have been quite a few optimizations that we have been compelled to tack on, comparable to predicate pushdowns, tuning the Parquet file dimension, utilizing a bucketed partition scheme, and extra. As extra information homeowners adopted this structure, we must customise every of those optimizations for his or her information fashions and shopper question patterns.
  • Restricted extendibility for real-time use instances – This batch extract, remodel, and cargo (ETL) structure wasn’t going to scale to deal with hourly updates of 1000’s of information upserts per second. As well as, it might be difficult for the information platform crew to maintain up with the various real-time analytical wants. Incremental queries, time-travel queries, improved latency, and so forth would require heavy funding over a protracted time period. Bettering on this subject would open up prospects like near-real-time ML inference and event-based alerting.

With all these limitations of the preliminary design, we determined to go all-in on an actual incremental processing framework.


The next diagram illustrates our up to date design. To help real-time use instances, we added Amazon Kinesis Information Streams, AWS Lambda, Amazon Kinesis Information Firehose and Amazon Easy Notification Service (Amazon SNS) into the structure.

The up to date elements are as follows:

  1. Amazon Kinesis Information Streams is a serverless streaming information service that makes it straightforward to seize, course of, and retailer information streams. We arrange a Kinesis information stream as a goal for AWS DMS. The info stream collects the CDC logs.
  2. We use a Lambda perform to rework the CDC information. We apply schema validation and information enrichment on the file degree within the Lambda perform. The reworked outcomes are printed to a second Kinesis information stream for the information lake consumption and an Amazon SNS matter in order that modifications may be fanned out to varied downstream methods.
  3. Downstream methods can subscribe to the Amazon SNS matter and take real-time actions (inside seconds) based mostly on the CDC logs. This could help use instances like anomaly detection and event-based alerting.
  4. To resolve the issue of lengthy batch processing time, we use Apache Hudi file format to retailer the information and carry out streaming ETL utilizing AWS Glue streaming jobs. Apache Hudi is an open-source transactional information lake framework that enormously simplifies incremental information processing and information pipeline growth. Hudi permits you to construct streaming information lakes with incremental information pipelines, with help for transactions, record-level updates, and deletes on information saved in information lakes. Hudi integrates properly with varied AWS analytics providers comparable to AWS Glue, Amazon EMR, and Athena, which makes it a simple extension of our earlier structure. Whereas Apache Hudi solves the record-level replace and delete challenges, AWS Glue streaming jobs convert the long-running batch transformations into low-latency micro-batch transformations. We use the AWS Glue Connector for Apache Hudi to import the Apache Hudi dependencies within the AWS Glue streaming job and write reworked information to Amazon S3 repeatedly. Hudi does all of the heavy lifting of record-level upserts, whereas we merely configure the author and remodel the information into Hudi Copy-on-Write desk sort. With Hudi on AWS Glue streaming jobs, we cut back the information freshness latency for our core datasets from hours to beneath quarter-hour.
  5. To resolve the partition challenges for prime cardinality UUIDs, we use the bucketing method. Bucketing teams information based mostly on particular columns collectively inside a single partition. These columns are often called bucket keys. While you group associated information collectively right into a single bucket (a file inside a partition), you considerably cut back the quantity of information scanned by Athena, thereby bettering question efficiency and decreasing price. Our current queries are filtered on the person ID already, so we considerably enhance the efficiency of our Athena utilization with out having to rewrite queries through the use of bucketed person IDs because the partition scheme. For instance, the next code reveals whole spending per person in particular classes:
    FROM "{{DATABASE}}"."{{TABLE}}"

  1. Our information scientist crew can entry the dataset and carry out ML mannequin coaching utilizing Amazon SageMaker.
  2. We preserve a replica of the uncooked CDC logs in Amazon S3 through Amazon Kinesis Information Firehose.


Ultimately, we landed on a serverless stream processing structure that may scale to 1000’s of writes per second inside minutes of freshness on our information lakes. We’ve rolled out to our first high-volume crew! At our present scale, the Hudi job is processing roughly 1.75 MiB per second per AWS Glue employee, which might robotically scale up and down (due to AWS Glue auto scaling). We’ve additionally noticed an impressive enchancment of end-to-end freshness at lower than 5 minutes resulting from Hudi’s incremental upserts vs. our first try.

With Hudi on Amazon S3, we’ve constructed a high-leverage basis to personalize our customers’ experiences. Groups that personal information can now share their information throughout the group with reliability and efficiency traits constructed right into a cookie-cutter answer. This allows our information customers to construct extra refined indicators to offer readability for all of life’s monetary selections.

We hope that this publish will encourage your group to construct a real-time analytics platform utilizing serverless applied sciences to speed up your online business targets.

Concerning the authors

Kevin Chun is a Employees Software program Engineer in Core Engineering at NerdWallet. He builds information infrastructure and tooling to assist NerdWallet present readability for all of life’s monetary selections.

Dylan Qu is a Specialist Options Architect targeted on massive information and analytics with Amazon Internet Companies. He helps prospects architect and construct extremely scalable, performant, and safe cloud-based options on AWS.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments