Published inTowards Data Science·PinnedMember-only10 Fantastic Books For Data Engineering10 Must-Read Books To Make You Successful As A Data Engineer — Introduction To Ten Fantastic Books For Data Engineering Many free resources online teach you how to become a data engineer, but the depth of those online resources context is limited. Reading books from domain experts is a more promising way to gain such in-depth knowledge. Are you looking for suggestions on what books to read for data engineering…Data Engineering10 min readData Engineering10 min read
Published inTowards Data Science·PinnedMember-onlyHere Is What I Learned Using Apache Airflow over 6 YearsA journey with Apache Airflow from experiment to production hassle-free — Apache Airflow is undoubtedly the most popular open-source project for data engineering for years. It gains popularity at the right time with The Rise Of Data Engineer, and the core concept of making code as the first-class citizen instead of drag and drop for data pipeline (aka. ETL) is a…Airflow8 min readAirflow8 min read
Published inTowards Data Science·PinnedMember-onlyDeep Dive into Handling Apache Spark Data SkewThe Ultimate Guide To Handle Data Skew In Distributed Compute — “Why my Spark job is running slow?” is an inevitable question while working with Apache Spark. One of the most common scenarios regarding Apache Spark performance tuning is data skew. In this article, we will cover how to identify whether your Spark job slowness is caused by data skew and…Apache Spark10 min readApache Spark10 min read
Published inTowards Data Science·PinnedMember-onlyAirflow Schedule Interval 101The airflow schedule interval could be a challenging concept to comprehend, even for developers work on Airflow for a while find difficult to grasp. A confusing question arises every once a while on StackOverflow is “Why my DAG is not running as expected?”. This problem usually indicates a misunderstanding among…Data7 min readData7 min read
Published inTowards Data Science·May 17Member-onlyData Engineering: Why It's About Much More Than Just the Tools You UseRethink Data Engineering Than Just Focusing On Tools — There are many tools for data engineering. Data engineering doesn't work solely on Python or SQL. With the era of big data and cloud computing, new data-related open-source projects have been emerging than ever before. Not only do we have multiple options for your tasks, but too many of them…Data Science6 min readData Science6 min read
Published inTowards Data Science·May 8Member-onlyBuilding Better Data Warehouses with Dimensional Modeling: A Guide for Data EngineersData Warehouse Dimensional Modeling Design 101 — Regarding system design for a data-intensive application, it usually comes with two options: write or read optimized. There isn’t a database design that fits and optimizes both writing and reading. Like all system design perspectives, no solution is right or wrong, while only pros and cons. As data professionals who…Data Science9 min readData Science9 min read
Published inTowards Data Science·Apr 20Member-onlyBoosting Spark Union Operator Performance: Optimization Tips for Improved Query SpeedDemystify Spark Performance in Union Operator — The union operator is one of the set operators to merge two input data frames into one. Union is a convenient operation in Apache Spark for combining rows with the same order of columns. One frequently used case is applying different transformations and then unioning them together. The ways of…Spark6 min readSpark6 min read
Published inData Engineering Space·Apr 15Member-onlyWhy R for Data Engineering is More Powerful Than You ThoughtR: Not Just for Data Scientists Anymore — How Data Engineers Can Benefit — Data Engineering is less inclusive for R. Most data engineers' daily work in the industry involves SQL and Python. We occasionally write Scala or Java for Spark and Flink jobs. Rarely do data engineers practice R in their work. …Data Science5 min readData Science5 min read
Published inData Engineering Space·Apr 7Member-only5 Hidden Apache Spark Facts That Fewer People Talk About5 Important Facts to Comprehend When Debugging Apache Spark — Developing Apache Spark can sometimes be frustrating when you hit the hidden facts on it. Those facts that fewer people talk about should be addressed in online courses or books. Until one day, you found the unexpected result and dug into the Apache Spark source code. I want to share…Data Engineering6 min readData Engineering6 min read
Published inData Engineering Space·Updated Apr 2Member-onlyThe Practical Data Engineering ResourceThe Awesome Data Engineering Resource (Keep Updating) — The data engineering space is evolving. Here are the resources I collected for practical data engineering resources. Last Updated: 2023–04–02 Books The Master Book On Distributed System: Designing Data-Intensive Application The SQL Fundamental Book: T-SQL Querying (Developer Reference) OLAP/Data Warehouse Must Read: The Data Warehouse Toolkit (The Definitive Guide to Dimensional…Data Engineering3 min readData Engineering3 min read
