Data Engineering Vol2 AWS : Data Processing - Spark & Kafka
Introduction - Data, Data Lifecycle & Data Engineering Pipeline
Data Engineering Vol2 AWS : Data Processing - Spark & Kafka
0%
Completed
Introduction to Data Engineering – Vol 2
Introduction - Data, Data Lifecycle & Data Engineering Pipeline
Video - 00:26:28 mins
Data Engineering Volume 2 Course & Projects Overview, Roles in Data
Video - 00:25:19 mins
AWS Resource Cost for the Course
Video - 00:6:26 mins
Section 1 – Big Data Processing
Distributed Compute & Storage, Big Data, MapReduce
Video - 00:34:11 mins
Map & Reduce Tasks, Big Data Ecosystem
Video - 00:30:55 mins
HandsOn : MapReduce using "mrjob" Python Library
Video - 00:27:44 mins
HandsOn : HDFS Commands
Video - 00:10:14 mins
YARN - Architecture & Usage
Video - 00:26:5 mins
ZooKeeper, Big Data File Formats - Parquet & Avro
Video - 00:20:18 mins
Section 2 – Spark Introduction
Introduction to Spark, Batch & Stream Processing
Video - 00:27:22 mins
Spark Ecosystem, Development, Architecture & Execution
Video - 00:23:56 mins
Spark Codebase, JVM, Setup & Configuration, PySpark
Video - 00:34:1 mins
Section 3 – Knowing Spark Up Close: Part 1
Spark Standalone Cluster on Laptop & EC2, Spark UI
Video - 00:29:24 mins
Spark on AWS EMR
Video - 00:21:54 mins
Spark Application Deployment Modes (--deploy-mode cluster|client)
Video - 00:3:11 mins
DataFrames, DAG, PySpark Module
Video - 00:24:22 mins
SparkSession
Video - 00:20:16 mins
DataFrame Reader & Writer
Video - 00:28:39 mins
Define DataFrame Schema - StructType, StructField, inferSchema
Video - 00:21:5 mins
Partition, Split, Task, Executor Relation
Video - 00:17:50 mins
Splits for Smaller Files
Video - 00:17:37 mins
Section 4 – Spark Transformation & Action : Part 1
Introduction to Transformation
Video - 00:15:59 mins
HandsOn : Intro to Transformation
Video - 00:17:59 mins
Step-by-step Transformation
Video - 00:18:26 mins
Chain Transformation
Video - 00:5:47 mins
Transformation - describe, sort, limit, drop, dropDuplicates
Video - 00:19:16 mins
Spark Action & Lazy Loading
Video - 00:8:37 mins
HandsOn : Spark Action & Lazy Loading
Video - 00:15:12 mins
HandsOn : Spark UI - Jobs, Executors, Environment
Video - 00:10:4 mins
HandsOn : load() - Transformation or Action??
Video - 00:13:23 mins
HandsOn : File Performance Comparison - CSV vs Parquet
Video - 00:11:37 mins
Assignment Project 1 - Transformation & Action
Video - 00:10:47 mins
Transformation - agg, groupBy, orderBy, selectExpr, subtract, withColumn, union
Video - 00:7:57 mins
HandsOn : Transformation : agg(), alias(), groupby()
Video - 00:14:8 mins
HandsOn : Transformation : join()
Video - 00:19:00 mins
HandsOn : Function : col()
Video - 00:19:21 mins
HandsOn : Transformation : orderBy(), selectExpr()
Video - 00:9:13 mins
HandsOn : Transformation : subtract()
Video - 00:9:3 mins
HandsOn : Transformation : union(), unionAll(), intersect(), withColumn()
Video - 00:10:10 mins
Shuffle Process & External Shuffle Service (ESS)
Video - 00:23:53 mins
HandsOn : Shuffle Process in groupBy() - Explanation using Spark UI
Video - 00:24:12 mins
HandsOn : Shuffle Process in join() - Explanation using Spark UI
Video - 00:10:13 mins
DO NOT use show() for analysis in Spark
Video - 00:6:10 mins
What should be the value of Shuffle Partition No (spark.sql.shuffle.partitions)
Video - 00:13:21 mins
Assignment Project 2 : Shuffle & Spark UI
Video - 00:19:59 mins
Summary - Transformations & Actions Part 1
Video - 00:3:5 mins
Section 5 – Spark Partition – Input, Shuffle & Output
Spark Input Partitions (spark.sql.files.maxPartitionBytes)
Video - 00:16:31 mins
Output Partitions - repartition(), coalesce()
Video - 00:11:34 mins
partitionBy(), Transformations that change the no. of Partitions
Video - 00:15:1 mins
DataFrame Writer Output Modes
Video - 00:6:45 mins
Section 6 – Knowing Spark Up close: Part 2
Do Dataframes keep occupying Memory ??
Video - 00:9:39 mins
How Spark Applications use Memory CPU & its calculation
Video - 00:12:53 mins
How to decide the size of Spark PROD Cluster
Video - 00:14:9 mins
Explore Spark UI - WholeStageCodegen, MapPartitions, Exchange etc.
Video - 00:22:9 mins
Spark OnHeap (Executor) Memory Mgmt - Reserve, Execution, Storage & User Memory
Video - 00:14:44 mins
HandsOn : Configure no. of Executors, its memory & cpu - applicaton+spark-submit
Video - 00:11:24 mins
HandsOn : Executor, Execution & Storage Memory in Spark UI
Video - 00:10:00 mins
Memory Spilling, Unified Memory Management & Off-heap Memory
Video - 00:11:4 mins
Cache, Persist, Unpersist, ClearCache
Video - 00:7:5 mins
HandsOn : Monitor Memory Spill, Caching & Persist in Spark UI
Video - 00:13:55 mins
Garbage Collection & Kryo Serialization in Spark
Video - 00:10:21 mins
Assignment Project 3.1 - Flight Efficiency Analysis
Video - 00:21:38 mins
Assignment Project 3.1 - Flight Efficiency Analysis Coding
Video - 00:17:16 mins
Assignment Project 3- Airport Operations,Same Route Performance,Freq Flier Prog
Video - 00:7:00 mins
Section 7 – Transformation & Action Part 2 + Spark Functions
Dataframe collect() API
Video - 00:10:35 mins
foreach() & foreachPartition() to apply customer logic on dataframe records
Video - 00:16:51 mins
Loop/Iterate though Dataframe records - toLocalIterator(), df.na.fill()
Video - 00:16:53 mins
Spark Functions - lit(), concat(), expr()
Video - 00:13:16 mins
DATE Functions
Video - 00:15:22 mins
when() & otherwise() Functions
Video - 00:6:25 mins
WINDOW Functions
Video - 00:9:24 mins
HandsOn : WINDOW Functions
Video - 00:7:32 mins
Assignment : Spark WINDOW Functions
Video - 00:18:9 mins
Assignment Project 4 - Layover Analysis, Travel Activity, Environment Impact
Video - 00:36:28 mins
lead(), lag(), nth_value(), first_value(), last_value()
Video - 00:11:3 mins
HandsOn : lead(), lag(), nth_value(), first_value(), last_value()
Video - 00:6:50 mins
Section 8 – Knowing Spark Up Close : Part 3
Spark Catalyst Optimizer & Tungsten Execution Engine
Video - 00:16:57 mins
Spark Internal Joins - Sort Merge Join (SMJ)
Video - 00:15:11 mins
Spark Internal Join - Broadcast Hash Join (BHJ)
Video - 00:9:57 mins
Spark Internal Join - Broadcast Nested Loop Join (BNLJ)
Video - 00:6:6 mins
Spark Internal Join - Shuffled Hash Join (SHJ)
Video - 00:5:56 mins
Spark Internal Join - Shuffled NL Join (SNLJ), Join Hints, Join Performance
Video - 00:5:11 mins
SparkUI Operators - HashAggregate, SortAggregate, BroadCastExchange
Video - 00:12:35 mins
SparkUI Operators - BroadcastHashJoin, BroadcastNLJoin
Video - 00:5:59 mins
SparkUI Operators - ShuffledHashJoin, Limit, CollectLimit, Window, ReusedExchang
Video - 00:7:39 mins
HandsOn : Spark Operators
Video - 00:7:28 mins
Spark Explain & How to Read Explain Output
Video - 00:21:6 mins
Data Skewness or Skewed Data
Video - 00:8:10 mins
HandsOn : Analyse & Find Data Skewness using Spark UI
Video - 00:14:51 mins
HandsOn : Spark Salting - Mitigate Data Skewness and Skew Joins
Video - 00:9:12 mins
HandsOn : Implement Salting step-by-step
Video - 00:16:56 mins
Adaptive Query Execution (AQE), AQE Parameters
Video - 00:17:40 mins
HandsOn : AQE Coalesce Shuffle Partitions
Video - 00:9:41 mins
HandsOn : AQE Optimize Data Skewness & Skewed Join
Video - 00:4:25 mins
Dynamic Partition Pruning
Video - 00:7:24 mins
HandsOn : Dynamic Resource Allocation (DRA)
Video - 00:17:41 mins
SUPER SUMMARY
Video - 00:10:29 mins
Section 9 – Hosting Platforms AWS EMR (Elastic MapReduce), Databricks
EMR Hosting Platform Introduction
Video - 00:23:20 mins
EMR Cluster Creation with IAM Roles
Video - 00:13:46 mins
EMR Considerartions - Spark, YARN Configuration Parameters, Submit Spark code to
Video - 00:17:40 mins
HandsOn : Prepare EMR environment to execute Spark Applications
Video - 00:12:3 mins
HandsOn : Submit Spark application to EMR from Primary Node & Laptop
Video - 00:20:3 mins
HandsOn : S3 vs HDFS as File Source & Target
Video - 00:5:58 mins
HandsOn : EMR Steps
Video - 00:9:41 mins
Relational (RDBMS) Database Sources & Targets
Video - 00:15:23 mins
numPartitions, lowerBound, upperBound
Video - 00:4:50 mins
HandsOn : Write to RDBMS (MySQL) Target
Video - 00:22:27 mins
HandsOn : Read from RDBMS (MySQL) Source
Video - 00:5:32 mins
Data Warehouse Source & Targets - Redshift
Video - 00:13:19 mins
HandsOn : Read & Write from/to Redshift Data Warehouse
Video - 00:16:39 mins
Filter and Aggregate Pushdown in Parquet
Video - 00:7:11 mins
Handling JSON Files
Video - 00:23:19 mins
Jupyter Notebook on EMR - Introduction
Video - 00:9:32 mins
HandsOn : Jupyter Notebook Security Group & IAM pre-requisites
Video - 00:6:47 mins
HandsOn : Create Jupyter Notebook and execute PySpark application
Video - 00:19:48 mins
Section 10 – Spark Core API RDD, Spark Configurations
Spark Configuration Parameters
Video - 00:19:8 mins
Introduction to RDD - map(), filter(), flatMap() APIs
Video - 00:15:56 mins
HandsOn : SELECT & FILTER using RDD
Video - 00:14:1 mins
Section 11 – Spark SQL
Introduction to Spark SQL, SQL on DataFrames
Video - 00:14:27 mins
HandsOn : DataFrame to SparkSQL
Video - 00:13:50 mins
Spark SQL Objects
Video - 00:12:41 mins
HandsOn : Spark SQL Objects - Database, Tables
Video - 00:13:15 mins
HandsOn : Read & Write Spark SQL Objects
Video - 00:15:28 mins
Iceberg ACID
Video - 00:32:58 mins
HANDS-ON - SparkSQL on EMR using Jupyter Notebook
Video - 00:10:21 mins
Section 12 – Datalakehouse Using Open Table Format (OTF)
Lakehouse Introduction & Iceberg Architecture
Video - 00:19:29 mins
HANDS-ON : Iceberg Architecture
Video - 00:13:13 mins
Iceberg Configuration, COW, MOR, Table Operations & Properties
Video - 00:17:28 mins
HandsOn : Iceberg Table Properties
Video - 00:11:21 mins
HANDS-ON : Iceberg Delete files - copy-on-write (COW), merge-on-read (MOR)
Video - 00:9:43 mins
COW or MOR??? Time Travel Queries, Iceberg Table History
Video - 00:16:4 mins
HANDS-ON : Iceberg Metadata, Time Travel Queries
Video - 00:8:7 mins
Iceberg Table Maintenance
Video - 00:7:25 mins
HANDS-ON : Iceberg Table Maintenance
Video - 00:8:7 mins
HANDS-ON : Iceberg Hidden Partition & Partition Evolution
Video - 00:22:55 mins
Medallion & Lakehouse Architecture
Video - 00:13:49 mins
Section 13 – Apache Kafka – The Streaming Injection
Introduction to Kafka
Video - 00:15:35 mins
Create Kafka Cluster using AWS MSK (Provisioned & Serverless)
Video - 00:20:39 mins
Create Kafka Cluster on Laptop
Video - 00:9:39 mins
Kafka Components
Video - 00:9:28 mins
Kafka Components - Topics & Partitions
Video - 00:14:21 mins
Kafka Up Close - Topology, Replication, Message Distribution, Write & Read Data
Video - 00:18:55 mins
Kafka Producer - Application, API, Configuration, Sync-Async Send, Python Library
Video - 00:21:15 mins
HANDS-ON : Kafka Producer Application - Send Records to Topic
Video - 00:19:38 mins
HANDS-ON : Kafka Producer Configuration
Video - 00:5:15 mins
Kafka Consumer - Application, Configuration, Partition Rebalance
Video - 00:15:49 mins
HANDS-ON : Kafka Consuler Application, Partition Ownership
Video - 00:14:44 mins
HANDS-ON : Kafka poll() API
Video - 00:5:58 mins
HANDS-ON : Kafka Offset Management
Video - 00:18:5 mins
HANDS-ON : Auto Offset Reset - Earliest & Latest
Video - 00:9:18 mins
Kafka Cluster (MSK) & Topic Throughput Calculation
Video - 00:14:50 mins
Avro Data Format & Glue Schema Registry
Video - 00:13:11 mins
HANDS-ON : Send AVRO data to Kafka Topic using Glue Schema Registry
Video - 00:21:43 mins
Schema Evolution & Kafka Configuration
Video - 00:15:45 mins
HANDS-ON : Schema Evolution using Glue Schema Registry
Video - 00:6:43 mins
Section 14 – Spark Streaming – Stream Processing Using Spark
Spark Streaming Intro, Trigger, Output Mode
Video - 00:21:47 mins
HANDS-ON : PySpark Streaming using Console Sink
Video - 00:17:3 mins
HANDS-ON : Execute Streaming code from PyCharm, use Spark History Server
Video - 00:10:34 mins
HANDS-ON : JSON data Streaming - Python Producer, Spark Consumer
Video - 00:24:58 mins
HANDS-ON : Spark Streaming OUTPUT Mode
Video - 00:10:5 mins
HANDS-ON : Using Files as Triggers in Streaming
Video - 00:13:53 mins
HANDS-ON : Streaming State Store & Checkpointing with Output Modes
Video - 00:18:44 mins
Introduction to Event Time in Stream Processing
Video - 00:13:59 mins
Event Time - Tumbling Window
Video - 00:14:47 mins
HANDS-ON : Event Time - Tumbling Window
Video - 00:13:57 mins
HANDS-ON : Event Time - Sliding Window
Video - 00:12:13 mins
Watermark - Handle Late Data & Manage State Store
Video - 00:22:35 mins
HANDS-ON : Watermark - Handle Late Data & Manage State Store
Video - 00:12:47 mins
HANDS-ON : Monitor Streaming Applications using Spark UI
Video - 00:8:21 mins
Spark Streaming Conclusion
Video - 00:5:45 mins
Section 15 – AWS Lanbda for Data Processing
Intro to Serverless Compute & AWS Lambda Use Cases
Video - 00:9:49 mins
Lambda Function & its Components
Video - 00:8:1 mins
HANDS-ON : Configure Lambda Function & its Components
Video - 00:16:53 mins
Lambda (Python Code) Execution Model
Video - 00:11:51 mins
HANDS-ON : Write Python Code from Lambda Console & Explore 'Event' data
Video - 00:12:39 mins
HANDS-ON : S3 Trigger -> Read File -> Copy to /tmp -> Print data
Video - 00:9:1 mins
HANDS-ON : S3 Trigger -> Read File -> Process -> Write to S3
Video - 00:10:16 mins
HANDS-ON : Deploy Python code from S3 using ZIP deployment package
Video - 00:9:48 mins
HANDS-ON : Deploy Additional Python Packages & Modules using ZIP in Lambda
Video - 00:10:44 mins
HANDS-ON : S3 Trigger to Database & Concurrent Lambda Execution
Video - 00:8:38 mins
Architecture using Lambda
Video - 00:3:17 mins
(Optional) AWS Essentials
AWS Cloud and EC2 Intro
Video - 00:19:13 mins
EC2 Components & HandsOn 1
Video - 00:21:7 mins
EC2 HandsOn 2
Video - 00:11:26 mins
EBS Theory
Video - 00:19:33 mins
EBS HandsOn
Video - 00:13:25 mins
VPC Introduction & Components
Video - 00:18:36 mins
VPC Components Hands On
Video - 00:23:4 mins
Bastion Host
Video - 00:5:29 mins
Security Groups
Video - 00:15:9 mins
NAT Gateway & VPC Endpoint
Video - 00:20:13 mins
VPC Peering
Video - 00:2:33 mins
AWS IAM Intro & Hands On
Video - 00:26:1 mins
IAM Service Role
Video - 00:19:38 mins
(Optional) Python Essentials
Python Intro - Architecture, PyCharm, Virtual Env
Video - 00:39:33 mins
PyCharm & CLI Walkthrough
Video - 00:8:51 mins
Compiled vs Interpreted
Video - 00:7:42 mins
Everything is Python is Object
Video - 00:12:39 mins
String
Video - 00:10:20 mins
Number
Video - 00:4:2 mins
List
Video - 00:11:36 mins
Tuple
Video - 00:6:16 mins
Set Dict Type Conversion
Video - 00:16:3 mins
Memory Allocation & Operators
Video - 00:10:27 mins
Set up Python interpreter in PyCharm
Video - 00:11:23 mins
Print & Input Functions
Video - 00:16:4 mins
IF Statement
Video - 00:14:36 mins
For & While loops
Video - 00:15:57 mins
Functions Intro
Video - 00:9:54 mins
Function Scoping
Video - 00:15:20 mins
Functions RETURN
Video - 00:7:53 mins
Function Arguments
Video - 00:9:38 mins
Modify Arguments
Video - 00:9:7 mins
Positional & Keyword Arguments
Video - 00:9:42 mins
(Optional) SQL Essentials
SQL Introduction
Video - 00:30:35 mins
Client & Server Setup
Video - 00:12:18 mins
Database Objects Theory
Video - 00:28:13 mins
Database Objects Hands On
Video - 00:29:19 mins
CRUD Operations
Video - 00:21:18 mins
SELECT Operators
Video - 00:24:41 mins
CASE COALESCE
Video - 00:13:19 mins
DATE Functions
Video - 00:5:46 mins
CTAS Cast Concat
Video - 00:14:11 mins
Update Delete Truncate
Video - 00:12:31 mins
HAVING Clause
Video - 00:7:43 mins
Joins
Video - 00:19:27 mins
Union Intersect View
Video - 00:17:35 mins
Materialized View
Video - 00:8:19 mins
Common Table Expression (CTE)
Video - 00:10:48 mins
Window Functions
Video - 00:22:40 mins
MERGE & Summary
Video - 00:10:52 mins
More
Announcements
Reviews
Course Info
Sign in
Sign up
Sign in
Don’t have an account?
Sign up
Remember me
Lost your password?
Sign up
Already have an account?
Sign in
Finish Course Early?
You have not completed all required lessons and assessments.
Your Progress
0%
Go Back to Course
Complete Anyway
