Rethink Data Engineering Than Just Focusing On Tools — There are many tools for data engineering. Data engineering doesn't work solely on Python or SQL. With the era of big data and cloud computing, new data-related open-source projects have been emerging than ever before. Not only do we have multiple options for your tasks, but too many of them…

Data Warehouse Dimensional Modeling Design 101 — Regarding system design for a data-intensive application, it usually comes with two options: write or read optimized. There isn’t a database design that fits and optimizes both writing and reading. Like all system design perspectives, no solution is right or wrong, while only pros and cons. As data professionals who…

Demystify Spark Performance in Union Operator — The union operator is one of the set operators to merge two input data frames into one. Union is a convenient operation in Apache Spark for combining rows with the same order of columns. One frequently used case is applying different transformations and then unioning them together. The ways of…

The Awesome Data Engineering Resource (Keep Updating) — The data engineering space is evolving. Here are the resources I collected for practical data engineering resources. Last Updated: 2023–04–02 Books The Master Book On Distributed System: Designing Data-Intensive Application The SQL Fundamental Book: T-SQL Querying (Developer Reference) OLAP/Data Warehouse Must Read: The Data Warehouse Toolkit (The Definitive Guide to Dimensional…

How To Detect The Best Deals With Web Scraping v2 — Automatically finding and detecting great deals and coupons promptly can save you money and time. slickdeals.net is a community-based website where people share deals information. Slickdeals has the "deal alert" option to detect deal information automatically. However, the choice is limited. It doesn't support if you'd want customized filters like…

The new way of playing Pac-Man, powered by streaming data — Data visualization has always been a delightful area for me to work on as a data professional. Visualizing data is like an art. Many creative ideas sprout in this space. People transform boring number and lengthy summary into beautiful charts. The well-known subreddit Dataisbeautiful is one of those places. Visualizing…

How R and Shiny Can Help You Find the Best Youtube Videos for Your Kids — Data is more engaged with storytelling. As a data professional, I seek less complicated ways to convey the gap between data analysis and communication. A dashboard is traditionally the default way to visualize and share data. It also carries the responsibility for communication. However, I found the limitations of the…

How to Build End-to-End Data Project Exploring New Trending Cocomelon Videos from Scratch Using R — Cocomelon — Nursery Rhymes is the world's second-largest Youtube channel (155M+ subscribers). It is such a popular and helpful channel that it is an inevitable subject for toddlers and parents. I enjoy spending time watching Cocomelon together with my son. After watching Cocomelon videos for a month, I noticed the…

Is Data Streaming Right for Your Business? Key Facts to Consider — Streaming data is an exciting space in the data field, and it has been getting tremendous attraction in recent years. With much excitement, the areas for open-source projects became crowded. Many technologies have made the streaming data process more straightforward than ever: Kafka, Flink, Spark, Storm, Beam, etc., …

SQL Window Function 101 with Examples and Diagram — Many SQL tutorials mark the window function as the advanced topic. Numerous job interviews on SQL often involve at least one question on window function. Understanding window function is critical for anyone that writes SQL daily (as well as SQL like languages like Spark Dataframe and Pandas). …

Comprehend Python Decorators By Visualization — Python decorator is syntactic sugar. You can achieve everything without explicitly using the decorator. However, Using the decorator can help your code be more concise and readable. Ultimately, you write fewer lines of code by leveraging Python decorators. Nevertheless, Python decorator isn't a trivial concept to comprehend. Understanding Python decorators…

Performance Benchmark on Popular Data Analysis Tools — Pandas is no doubt one of the most popular libraries in Python. Its DataFrame is intuitive and has rich APIs for data manipulation tasks. Many Python libraries integrated with Pandas DataFrame to increase their adoption rate. However, Pandas doesn’t shine in the land of data processing with a large dataset…

Write Clear SQL By Comprehend Logical Query Processing Order — You might find writing SQL challenging due to its declarative nature. Especially for engineers familiar with imperative languages like Python, Java, or C, SQL is gear-switching and mind shifts to many people. …

Explore Excellent Options for Data Pipeline Orchestration for R Users — The data pipeline orchestration tool is critical for producing healthy and reliable data-driven decisions. R is one of the popular languages for data scientists. With R’s exceptional packages, the R programming language is great for data manipulation, statistical analysis, and visualization. One pattern that often brings data scientists’ R local…

Gentle Introduction to gganimate with Example on Racing Bar Chart — Have you seen any beautiful racing bar chart data animation on Youtube and wondered how it was built? Data visualization with animations is fun to watch. There are various libraries to create spectacular animations. …

Explore Data Engineering Side Project Ideas On Your Resume — Working on side projects to get experience and show your work is a good idea to practice data engineering. You might be eager to look for side projects as a new and experienced data engineer. Those side projects could initiate impressive discussions to help you land a dream job. …

Introduction to mage-ai as the alternative to Apache Airflow for data engineers — Apache Airflow became the top project in Apache Foundation in January 2019. It is one of the most popular projects for data engineers, and many companies list Airflow as one of the requirements for hiring. Data projects iterate quickly, and some projects have been replaced rapidly. Airflow has been widespread…

A journey with Apache Airflow from experiment to production hassle-free — Apache Airflow is undoubtedly the most popular open-source project for data engineering for years. It gains popularity at the right time with The Rise Of Data Engineer, and the core concept of making code as the first-class citizen instead of drag and drop for data pipeline (aka. ETL) is a…

Suggestions for your work get discovered as data professionals — As data professionals, you are familiar with ways to show your work to leadership, colleague, or someone you know. Many good books (Storytelling with Data by Cole Nussbaumer Knaflic and DataStory: Explain Data and Inspire Action Through Story by Nancy Duarte) covers this topic of telling stories with data. Getting…

How to Get Data Engineering Opportunity in 2023 — If you are looking for a job in the data field, you may have heard of the position: Data Engineer. What is a Data Engineer? Data Engineer is a role that bridges the gap between raw data into information that is ready to be analyzed. The usual workflow for a…