Realtime Change Data Capture Streaming | End to End Data Engineeering Project
In this video, we dive deep into the world of Change Data Capture (CDC) and how it can be implemented for real-time data streaming using a powerful tech stack. You will use the integration of technologies like Docker, Postgres, Debezium, Kafka, Apache Spark, and Slack to create an efficient and responsive data pipeline.
Master Data Engineering by enrolling at datamasterylab.com
Timestamps:
0:00 Introduction
0:35 The system architecture
14:40 Getting live data into postgres db
31:00 Connecting to Postgres with Debezium and Kafka from the UI
34:35 Previewing Debezium data on Kafka
37:25 Getting full data from Postgres with Debezium
39:55 Setting up debezium connector from the terminal
41:00 Handling decimal values on debezium
46:00 Getting the user that changed data on postgres with time
53:17 Creating a more robust data capture on postgres
1:03:31 Outro
Don't forget to SUBSCRIBE, LIKE, COMMENT and SHARE for more exciting videos!
🔹 Connect with Us:
Follow us on Twitter: x.com/@DataMasterylab
Follow us on LinkedIn: https://www.linkedin.com/in/yusuf-ganiyu-b90140107
Tags:
Change Data Capture, CDC, Real-Time Streaming, Docker, Postgres, Debezium, Kafka, Apache Spark, Slack, Data Engineering, Data Pipeline, Tech Tutorial, Software Development, Data Streaming, IT Education, End to End Data Engineering
Hashtags:
#ChangeDataCapture #RealTimeData #Docker #Postgres #Debezium #Kafka #ApacheSpark #Slack #DataEngineering #TechTutorial #SoftwareDevelopment #DataStreaming #ITEducation #bigdataanalytics #EndToEndDataEngineeringProject
-
LIVE
JdaDelete
7 hours ago $3.07 earnedProject Zomboid with the Boys | Season 2 Finale | Fiery Farewell
1,792 watching -
LIVE
Alex Zedra
2 hours agoLIVE! New Game | The Escape: Together
641 watching -
LIVE
Drew Hernandez
13 hours agoTRUMP RALLY LAS VEGAS NEVADA
2,404 watching -
1:38:08
Glenn Greenwald
5 hours agoCNN Town Hall Exposes Kamala's Inability To State Beliefs; Democrats' "Trump Is Hitler" Tactic Failing Again | SYSTEM UPDATE #357
63.6K105 -
1:07:17
Donald Trump Jr.
11 hours agoMore Media Hoaxes, They’re as Desperate as Ever. Interviews with Steve Moore, Lee Smith, and Matt Walsh | TRIGGERED Ep.185
108K152 -
2:43:46
LFA TV
1 day agoFREE SCREENING | WHAT IS TREASON #TRAFFICKED
35.3K39 -
DVR
LFA TV
1 day agoKing David Is Having a Resurgence | Trumpet Daily 10.24.24 9PM EST
19.9K -
LIVE
VOPUSARADIO
3 days agoPOLITI-SHOCK! W/ Special Guests: David "Nino" Rodriguez & Rick Barrett!
295 watching -
1:02:29
The StoneZONE with Roger Stone
6 hours agoGeneral John Kelly lies about Trump...AGAIN | The StoneZONE w/ Roger Stone
40.1K12 -
5:50:50
Right Side Broadcasting Network
1 day agoLIVE REPLAY: President Trump Holds a Rally in Tempe, AZ - 10/24/24
223K124