data-engineering-log on Shadowsong's Personal Website

DE Log 9: Extend duckdb beyond the single player mode, by just a little bit

Sat, 28 Dec 2024 00:00:00 +0000

Yet another take on poor man’s lakehouse / or how to NOT contribute to Databricks’ IPO success even though it seems inevitable.

DET Log 1: A first “byte” of Airbyte

Fri, 21 Oct 2022 00:00:00 +0000

In this article, I am sharing my first impression on the open-sourced almighty data ingestion tool - Airbyte.

DE Log 8: Thoughts on Data Engineering Specialisations

Sat, 01 Jan 2022 00:00:00 +0000

Happy New Year everyone! And I am too lazy to change the thumbnail image.

DE Log 7: Migrating from Airflow 1 to 2

Thu, 29 Apr 2021 00:00:00 +0000

Today I have successfully migrated my current Airflow setup from v1.10.14 to v2.0.2. This article will not be a very detailed step by step guide for upgrading, instead I will introduce the general migration step worked very specifically for my setup, and share some of the problems I encountered during the process, and finally some general feelings with Airflow 2.

DE Log 6: From Data Engineering to Meta Data Engineering - the future of Data Engineering

Wed, 19 Aug 2020 00:00:00 +0000

Data engineering jobs are really popular nowadays, mostly contributed by the rising demand of data insights and data driven decision making.

DE Log 5: Thoughts on Analytical Tables

Thu, 09 Jan 2020 00:00:00 +0000

I have taken some Database design course back in university days, though I have skipped almost all of the lectures as I was also self-learning during my first internship on the same matter, in a much more practical manner. I did not even know the word OLAP back then. However, I am not dismissing the importance of data modelling in data engineering. On the contrary, data modelling is one of the important skills if you want to be a data engineer.

DE Log 4: ETL vs ELT

Wed, 01 Jan 2020 00:00:00 +0000

We have all heard of the term ETL. If you are working in the data field, you might have been asked to do some sort of ETL work regardless of your actual job description.

DE Log 3: Amazon Redshift disk space saving tips

Mon, 04 Nov 2019 00:00:00 +0000

Recently Amazon Redshift launched a new console interface, which is pretty nice. It actually gives some valuable optimisation tips. A data warehouse is like a sword, you need to constantly sharpen it so it won’t lose its edge.

DE Log 2.1: Develop Data Science Project in Production

Thu, 17 Oct 2019 00:00:00 +0000

Recently I was working on the deployment of a predictive model built by my dear friend valued ex-colleague. Here I will share some thoughts and challenges I have encountered during its production deployment.

DE Log 1: Idempotency

Sat, 31 Aug 2019 00:00:00 +0000

In linear algebra, an idempotent matrix is a matrix which, when multiplied by itself, yields itself.

DE Log 0.1: Rescheduling of Airflow DAGs

Sat, 22 Sep 2018 00:00:00 +0000

This article briefly records the step to reschedule a DAG on Apache Airflow