================================================================================
PYTHON FOR DATA ENGINEERING — COMPANION SAMPLE DATASETS
Nolan Whitfield / Waskey Press
================================================================================

WHAT THIS PACKAGE CONTAINS
--------------------------

This book's central subject is data itself, so this companion package is
genuinely data-heavy — a real dataset for nearly every major demonstration
in the book, letting you rerun this book's own real, executed comparisons
and validations yourself, on your own machine, rather than only reading
about their results.

  ch02_first_pipeline/
    orders_2026-07-10.csv        Chapter 2's first ETL pipeline exercise.
                                  150 rows, 2 deliberately invalid (a null
                                  order_id, a null amount) so you can watch
                                  clean_orders() correctly drop exactly 2 rows.

  ch03_formats/
    orders_sample.csv            The same 2,000-row dataset, once as CSV,
    orders_sample.parquet        once as Parquet — round-trip both and
                                  confirm CSV loses the datetime column's
                                  type while Parquet preserves it, exactly
                                  as Chapter 3 demonstrated.
    orders_nested.json           20 genuinely nested orders (a customer
                                  object + a variable-length line_items
                                  array) for the pd.json_normalize()
                                  flattening exercise.

  ch04_sql_demo/
    orders_customers.db          A SQLite database with 20 orders and 4
                                  customer rows, deliberately including a
                                  duplicate customer_id=100 (for the join
                                  fan-out exercise) and an orphaned
                                  customer_id=999 referenced by one order
                                  but absent from the customers table (for
                                  the referential integrity exercise).

  ch06_validation/
    orders_with_issues.csv       50 rows with deliberate, realistic
                                  problems: null customer_ids, a negative
                                  amount, a null amount, and one
                                  future-dated order — built specifically
                                  to exercise every row-level validation
                                  rule from Chapter 6.

  ch07_extraction/
    paginated_api_sample/        5 JSON files simulating a paginated API
                                  response (450 total records reported),
                                  with page 3 deliberately empty — the
                                  exact scenario Chapter 7 used to
                                  demonstrate silent, incomplete pagination.
    incoming_export.csv          A file/done-marker pair for Chapter 7's
    incoming_export.csv.done     file-completeness exercise.

  ch08_incremental/
    watermark_state.json         A sample watermark store state.
    run1_extraction.csv          Two snapshots for Chapter 8's boundary-
    full_source_state.csv        problem exercise — order 202 shares a
                                  timestamp with the watermark set by the
                                  first run, and would be permanently
                                  missed by a naive '>' comparison.

  ch10_large_data/
    large_orders.csv             100,000 rows (~3.3 MB) — large enough to
                                  genuinely exercise Chapter 10's dtype
                                  optimization and chunked-processing code
                                  without an unreasonably large download.

  ch12_dimensional/
    fact_sales_sample.csv        A 300-row fact table at a clean, single
                                  grain (one row per sale) for Chapter 12's
                                  dimensional modeling exercises.
    dim_customer_scd2_sample.csv A 5-row dimension table already containing
                                  one genuine SCD Type 2 history (customer
                                  500's move from EMEA to APAC), ready for
                                  the join_as_of() exercise.

  ch15_observability/
    historical_baseline_amounts.csv   ~5,000 rows of stable historical data.
    todays_batch_with_drift.csv       800 rows with the same ~8% silent
                                        currency-conversion drift Chapter 15
                                        demonstrated — run the Kolmogorov-
                                        Smirnov comparison yourself and
                                        confirm the same overwhelming
                                        p-value the chapter reported.

  ch18_capstone_lumen/
    sales_2026-07-10.csv         The exact two source files used to build
    crm.db                       and run the Chapter 18 capstone, Lumen,
                                  end to end — extraction through the
                                  final "revenue by region" query.


HOW TO USE THESE FILES
-----------------------

Every file here is designed to be pointed at directly by the code in this
book, unmodified. For example, to rerun Chapter 3's CSV-vs-Parquet type
preservation check yourself:

  import pandas as pd
  csv_version = pd.read_csv("ch03_formats/orders_sample.csv")
  parquet_version = pd.read_parquet("ch03_formats/orders_sample.parquet")
  print(csv_version["order_date"].dtype)      # object (a plain string)
  print(parquet_version["order_date"].dtype)  # datetime64[ns] (preserved)

Or to rerun Chapter 15's distribution drift test yourself:

  import pandas as pd
  from scipy import stats
  historical = pd.read_csv("ch15_observability/historical_baseline_amounts.csv")
  todays = pd.read_csv("ch15_observability/todays_batch_with_drift.csv")
  ks_stat, p_value = stats.ks_2samp(historical["amount"], todays["amount"])
  print(p_value)  # an extraordinarily small value, confirming genuine drift


AN HONEST NOTE ON HOW THIS PACKAGE WAS BUILT
-----------------------------------------------

Every dataset here is synthetic — generated programmatically with numpy's
random number generator, using a fixed seed for full reproducibility, never
real customer, order, or business data of any kind. Several files were
deliberately constructed to contain a specific, realistic problem (a
duplicate key, an empty API page, a currency-shifted distribution) exactly
matching a demonstration this book actually ran and reported real, measured
results for. Rerunning the book's own code against these exact files should
reproduce results very close to what this book reports — small numeric
differences from randomness are expected and completely normal, since exact
reproduction depends on your own installed library versions matching the
book's, exactly the dependency-pinning concern Chapter 16 discussed directly.


LICENSE
-------

These sample datasets are provided free of charge for personal, educational
use alongside this book. They may be redistributed only as part of this
book's companion package, unmodified.

Questions or corrections: www.waskeypress.com
================================================================================
