Writing hub
Practical notes from XH / bossruji on data platform architecture, governance, lakehouse operations, and analytics delivery. Full articles are published here on xhverse.co. When a piece is also on Medium, a second link is shown for readers who prefer that surface.
Change Data Capture matters because it turns source-system change into ordered, replayable data contracts, but only when teams design for snapshots, retention, schema drift, deletes, and recovery.
Data architecture is not a stack label, data model, or implementation pattern; it is the operating blueprint that connects ownership, governance, serving contracts, cost, latency, reliability, and change.
A practical way to group PySpark data quality checks by the production failures they prevent: broken contracts, duplicate loads, stale data, invalid business logic, and silent drift.
A practical guide to reading Spark cluster mode as operational contracts: what the driver owns, what the cluster manager allocates, what executors execute, and how those boundaries help debug production pipeline failures.
Delta Lake turns object storage into a governed table system by making every table change explicit in the transaction log, but that reliability still depends on operational discipline.
Materialized views can reduce repeated analytical cost, but only after teams diagnose modeling, query shape, layout, freshness, and workload repeatability.
OneLake works best when teams treat it as a tenant-wide platform contract for ownership, access, shortcuts, discovery, and cross-engine reuse, not just a managed storage layer.
ABAC can make Databricks row and column controls consistent, but only if tags, policies, UDFs, exemptions, performance, and audit readback have named owners.
Serverless removes cluster work, but production data teams still need ownership, eligibility gates, cost attribution, observability, release discipline, and recovery paths.
Data products work when teams treat them as platform contracts: named consumers, explicit owners, quality expectations, access boundaries, and lifecycle rules before the consumer count grows.
Delta, Iceberg, and Hudi are not just file formats. They change who owns metadata, how tables are maintained, and where lock-in appears.
Open formats move cost from one place to another. The winning teams track metadata health, compaction debt, and ownership load as first-class operating costs.
Most teams adopt envelope encryption to scale security, then keep treating key management like an afterthought. This breaks quickly under shared-platform operating pressure.
A practical guide to choosing between wide analytical tables, dimensional models, and the hybrid layer that usually survives production.