Published inTowards Data Science·PinnedMember-only10 Fantastic Books For Data Engineering10 Must-Read Books To Make You Successful As A Data Engineer — Introduction To Ten Fantastic Books For Data Engineering Many free resources online teach you how to become a data engineer, but the depth of those online resources context is limited. Reading books from domain experts is a more promising way to gain such in-depth knowledge. Are you looking for suggestions on what books to read for data engineering…Data Engineering10 min readData Engineering10 min read
Published inTowards Data Science·PinnedMember-onlyHere Is What I Learned Using Apache Airflow over 6 YearsA journey with Apache Airflow from experiment to production hassle-free — Apache Airflow is undoubtedly the most popular open-source project for data engineering for years. It gains popularity at the right time with The Rise Of Data Engineer, and the core concept of making code as the first-class citizen instead of drag and drop for data pipeline (aka. ETL) is a…Airflow8 min readAirflow8 min read
Published inTowards Data Science·PinnedMember-onlyDeep Dive into Handling Apache Spark Data SkewThe Ultimate Guide To Handle Data Skew In Distributed Compute — “Why my Spark job is running slow?” is an inevitable question while working with Apache Spark. One of the most common scenarios regarding Apache Spark performance tuning is data skew. …Apache Spark10 min readApache Spark10 min read
Published inTowards Data Science·PinnedMember-onlyAirflow Schedule Interval 101The airflow schedule interval could be a challenging concept to comprehend, even for developers work on Airflow for a while find difficult to grasp. A confusing question arises every once a while on StackOverflow is “Why my DAG is not running as expected?”. This problem usually indicates a misunderstanding among…Data7 min readData7 min read
Published inILLUMINATION·Nov 15Member-only5 Lessons I Learned From a Totaled Car AccidentA life-changing accident changed my life — Well, it's still not easy to process everything, but I want to share at this point at the end of the year after the accident happened on April 26th. After a car accident that resulted in my car being totaled, I want to share what I learned with more people. The Accident …Life6 min readLife6 min read
Published inData Engineering Space·Aug 25Member-onlyBidding War on Housing? Let’s Use R For Exploratory Data AnalysisExploratory Data Analysis 101 in R — Exploratory Data Analysis (EDA) is a methodology in data science as the initial approach to gain insights by visualizing and summarizing data. As this article was written in August 2023, the 30-year fixed mortgage is 7.48%. Will this make housing prices go down further? …Data5 min readData5 min read
Published inData Engineering Space·Aug 7Member-onlyVisualizing Data with ggridges: Techniques to Eliminate Density Plot Overlaps in ggplot2Data Exploratory: Avoiding Overlapping Density Plots in R Using ggridges — Problem with Default Density Plot in ggplots When it comes to visualizing data with a histogram and dealing with multiple groups, it can be quite challenging. I have recently come across a useful ggplot2 extension called ggridges that has been helpful for my data exploratory tasks. It can be difficult to display multiple groups in a plot…Ggplot23 min readGgplot23 min read
Published inTowards Data Science·Jul 17Member-onlyUnlocking the Secrets of Slowly Changing Dimension (SCD): A Comprehensive View of 8 TypesDeep Dive Guide for When and How to Use 8 Types of SCD — If you are designing a data warehouse, it is essential to understand the methodology of dimensional modeling. One of the most important concepts in this methodology is the Slowly Changing Dimension (SCD). …Data Science12 min readData Science12 min read
Published inData Engineer Things·Jun 23Member-onlyHow I Built a Tool to Visualize Expense In Sankey DiagramHow to Create a Sankey Diagram to Track Your Personal Finances in R — One year ago, I wrote an article about How to Visualize Monthly Expenses in a Comprehensive Way: Develop a Sankey Diagram in R. It leverages a package called ggalluvial R for building Sankey charts. However, it isn’t interactive, and I didn’t create a UI for the tool. …Data Science5 min readData Science5 min read
Published inTowards Data Science·May 17Member-onlyData Engineering: Why It's About Much More Than Just the Tools You UseRethink Data Engineering Than Just Focusing On Tools — There are many tools for data engineering. Data engineering doesn't work solely on Python or SQL. With the era of big data and cloud computing, new data-related open-source projects have been emerging than ever before. Not only do we have multiple options for your tasks, but too many of them…Data Science6 min readData Science6 min read
