Data Engineering
-
Setting up Row-level and Column-level Security in Fabric Lakehouse
This guide walks through how to set up Row-Level Security (RLS) and Column-Level Security (CLS) in the Microsoft Fabric Lakehouse using the new OneLake Security…
·
-
Variable Library in Microsoft Fabric Data Pipeline
Managing connections in Microsoft Fabric pipelines can be messy if you hard-code them for every environment. In this post, I’ll show you how to use…
·
-
Databricks Autoloader vs Structured Streaming vs Delta Live Tables: What’s the Difference?
Confused about when to use Databricks Autoloader, Structured Streaming, or Delta Live Tables? You’re not alone. While all three seem to handle streaming data, they…
·
-
Incremental Loading 101: Timestamp, Watermarking, Hash Comparisons and CDC
Learn four incremental-load strategies — timestamp-based, Change Data Capture (CDC), watermarking, and hash-based comparison — to optimize performance, reliability, and resource utilization in your data…
·
-
Why Your API Retries Keep Failing: Fix It with Exponential Backoff
In the real world of data engineering, we often work with external systems, e.g. APIs, databases, cloud services, etc., that occasionally fail. These failures can…
·
-
12 Tips for Every Data Engineer: A Technical Blueprint for Data Ingestion
Discover 12 essential steps for data engineers to design reliable, scalable ingestion pipelines— from source profiling and schema handling to SLAs, monitoring, and cost optimization.
·
-
Variable Library in Microsoft Fabric Notebooks
Learn how to use the Variable Library in Microsoft Fabric to manage environment-specific configurations across notebooks and pipelines. This feature, now in public preview, helps…
·
-
TIMESTAMP vs TIMESTAMP_NTZ in Delta Tables: Lessons from Real Projects
Struggling with time zone challenges in your data pipelines? Discover the key differences between TIMESTAMP and TIMESTAMP_NTZ in Delta tables, and learn how Databricks and…
·
-
Mastering Slow Changing Dimensions (SCD)
Discover the importance of Slow Changing Dimensions (SCD) in data warehousing. Learn how to track data changes over time with practical examples and a detailed…
·
-
Why Surrogate Keys are Essential in Data Warehousing: When to Use and Why They Matter
Discover why surrogate keys are essential in data warehousing. Learn when to use them, their role in improving performance, handling historical data, and ensuring data…
·
Categories
Tags
Autoloader Best Practices CDC Chatbot ChatGPT Column-level Security CRUD database databricks Data Cleansing Data Ingestion Data Model Data Warehouse DeepSeek Delta Live Table Dictionary Django E-Commerce ETL Exponential Backoff fstring Generative AI Generator Expressions json List List Comprehensions LLM Microsoft Fabric Number Programming Tips Python RAG Reverse String Row-level Security SCD Slicing Software Engineering String Structured Streaming token TOON tstring Tuples Variable Library web