Revise README for clarity and detail on SQLCompare functionality and usage

Author: luisggoncalves

README.md

@@ -1,139 +1,229 @@
 # SQLCompare
 
-Professional CLI for comparing datasets across database tables.
+SQLCompare helps you understand how a change impacted your data.
+When you modify logic, filters, or inputs, SQLCompare lets you compare the previous and current versions of a dataset—whether they come from tables, SQL queries, or files.
 
-SQLCompare performs row-level comparisons between a "previous" and "current" dataset using a composite key, materializes a join table in a comparison schema, computes column-level differences, and stores a diff run ID for repeatable analysis.
+You can compare datasets in two complementary ways:
 
-## Key Features
-- Compare tables or dataset queries using composite keys.
-- Persist comparison runs with metadata for later analysis.
-- Generate per-column stats, missing-row checks, and samples of differences.
-- Designed for repeatable CLI workflows in data engineering and QA.
+1) Row-by-row comparison (with an ID): detect missing rows on either side, identify the columns with the most changes, and inspect before/after values for any record.
+2) Statistical comparison: compare column-level statistics such as null counts, distinct counts, and other aggregates to quickly understand overall impact.
 
-## Installation
-Use your project setup and dependency manager of choice. For local development:
+---
+
+## What you get
+
+- **Repeatable checks** for releases, backfills, migrations, and vendor drops
+- **Clear summaries**: missing-row detection + per-column change counts
+- **One workflow** for warehouses *and* local files (DuckDB-powered)
+
+---
+
+## Install
+
+Recommended:
 
 ```bash
-uv sync --extra dev
+uv tool install sqlcompare
 ```
 
-## Quick Start
-Compare two tables on a single key:
+Install optional connector extras as needed:
 
 ```bash
-sqlcompare table analytics.fact_sales analytics.fact_sales_new id
+uv tool install "sqlcompare[<connector>]"
 ```
 
-Compare using a composite key:
+Examples:
 
 ```bash
-sqlcompare table analytics.users analytics.users_new user_id,tenant_id
+uv tool install "sqlcompare[snowflake]"
+uv tool install "sqlcompare[databricks]"
 ```
 
-Use a named connector and custom schema:
+---
+
+## Quick start (tables)
+
+Compare two tables on a key:
 
 ```bash
-sqlcompare table public.orders public.orders_latest order_id -c prod --schema sqlcompare
+export SQLCOMPARE_CONN_DEFAULT="postgresql://<user>:<pass>@<host>/<db>"
+sqlcompare table analytics.fact_sales analytics.fact_sales_new id
 ```
 
-Follow up with analysis:
+That command prints a **diff_id**. Use it for follow-up analysis:
 
 ```bash
-sqlcompare report <diff_id> --stats
-sqlcompare report <diff_id> --column revenue --limit 100
-sqlcompare report <diff_id> --missing-current
+sqlcompare analyze-diff <diff_id> --stats
+sqlcompare analyze-diff <diff_id> --column revenue --limit 100
+sqlcompare analyze-diff <diff_id> --missing-current
 ```
 
-## Command Reference
+---
+
+## Core idea: compare once, analyze many times
+
+SQLCompare does two things:
+
+1. **Compare** two datasets using an index (single or composite key)
+2. Persist comparison results and return a **diff_id** you can use to:
+
+* get overall stats
+* drill into a specific column’s changes
+* list missing rows (previous-only / current-only)
+* pull samples for debugging
 
-### Compare tables
+---
+
+## Usage by use case
+
+### 1) Compare two tables
+
+Best for production validation and regression checks across supported connectors.
 
 ```bash
-sqlcompare table TABLE1 TABLE2 IDS [--connection/-c CONNECTION] [--schema SCHEMA]
+sqlcompare table analytics.users analytics.users_new user_id,tenant_id
 ```
 
-#### Arguments
-- `TABLE1`: Fully qualified name of the "previous" table to compare. Passed directly to SQL (example: `analytics.schema.table`).
-- `TABLE2`: Fully qualified name of the "current" table to compare. Passed directly to SQL.
-- `IDS`: Comma-separated list of columns that uniquely identify a row (example: `id` or `id,sub_id`). Whitespace around commas is stripped.
+Why it’s useful:
 
-#### Options
-- `--connection`, `-c`: Name of a connector profile to use. If omitted, SQLCompare reads the default connector from configuration.
-- `--schema`: Schema used to store the join table created for the comparison. If omitted, it uses the configured comparison schema.
+* Handles **composite keys**
+* Produces a **saved diff ID** for repeatable review
 
-## Configuration
-SQLCompare is configured through environment variables:
+---
+
+### 2) Compare SQL query results
+
+Use this when tables aren’t materialized yet or you want a filtered slice.
+
+Create a dataset config:
+
+```yaml
+previous:
+  select_sql: "SELECT * FROM analytics.orders WHERE order_date < '2024-01-01'"
+  index:
+    - ORDER_ID
+
+new:
+  select_sql: "SELECT * FROM analytics.orders WHERE order_date >= '2024-01-01'"
+  index:
+    - ORDER_ID
+```
+
+Run the compare:
+
+```bash
+sqlcompare dataset path/to/dataset.yaml
+```
+
+Why it’s useful:
+
+* Compare slices without touching production tables
+* Keep an auditable **diff_id**
 
-- `SQLCOMPARE_COMPARISON_SCHEMA`:
-  Schema used for storing comparison tables.
-- `SQLCOMPARE_CONN_DEFAULT`:
-  Name of the default connector profile.
-- `SQLCOMPARE_CONN_{NAME}`:
-  SQLAlchemy URL for a named connector profile. Use with `-c {NAME}`.
+---
 
-## How It Works
+### 3) Compare local CSV / XLSX files (DuckDB)
 
-1) Resolve configuration, connector, and comparison schema.
-2) Parse `IDS` into a list of index columns.
-3) Create a unique comparison name and diff ID.
-4) Use `DatabaseComparator` to:
-   - Verify index columns exist in both tables (case-insensitive match).
-   - Create a `FULL OUTER JOIN` table in the comparison schema.
-   - Count rows present only in either table.
-   - Compute per-column differences for shared columns.
-   - Persist metadata for later analysis.
-5) Emit a summary and a follow-up report command.
+Great for ad hoc QA, one-off deliveries, or vendor drops.
+SQLCompare uses DuckDB under the hood — no DB server required.
 
-## Diff IDs and Metadata
-Each run is recorded in:
+Create a dataset config (supports `{{here}}` for relative paths):
 
+```yaml
+previous:
+  file_name: "{{here}}/previous.csv"
+  index:
+    - id
+
+new:
+  file_name: "{{here}}/current.xlsx"
+  index:
+    - id
 ```
-~/.config/sqlcompare/db_test_runs.yaml
+
+Set a local default connector and run:
+
+```bash
+export SQLCOMPARE_CONN_DEFAULT="duckdb:///:memory:"
+sqlcompare dataset path/to/dataset.yaml
 ```
 
-The record includes:
-- `tables`: mapping of previous, current, and join table names
-- `index_cols`: resolved index columns (as found in the database)
-- `cols_prev` / `cols_new`: column lists for each table
-- `conn`: connector name (or `duckdb` for file-based runs)
+Why it’s useful:
+
+* Same diff workflow as warehouses
+* Fast local comparisons, zero infra
+
+---
+
+### 4) Compare tables inside a DuckDB file
 
-Diff ID format:
+Use this when your data lives in a local `.duckdb` file.
 
+```bash
+export SQLCOMPARE_CONN_LOCAL="duckdb:////absolute/path/to/warehouse.duckdb"
+sqlcompare table raw.customers staged.customers id -c local
 ```
-compare_{table1}_{table2}_{timestamp}_{random}
+
+Why it’s useful:
+
+* Local + fast, without Snowflake/warehouse costs
+* Easy to integrate into lightweight pipelines
+
+---
+
+## Configuration
+
+SQLCompare resolves connectors in this order:
+
+1. default connector
+2. direct URL
+3. environment variables
+4. YAML files
+
+### Environment variables
+
+* `SQLCOMPARE_CONN_DEFAULT` — default connector URL (used when `-c` is omitted)
+* `SQLCOMPARE_CONN_{NAME}` — SQLAlchemy URL for a named connector
+* `SQLCOMPARE_COMPARISON_SCHEMA` — schema for comparison tables (default: `sqlcompare`)
+
+Example:
+
+```bash
+export SQLCOMPARE_CONN_DEFAULT="postgresql://..."
+export SQLCOMPARE_CONN_LOCAL="duckdb:////abs/path/to/db.duckdb"
 ```
 
-Table names are sanitized to alphanumeric plus underscores.
+### YAML connections file (optional)
 
-## Output Summary
-The CLI prints a concise report to stdout (via `data_toolkit.core.log`), including:
-- Missing rows in either table (based on null key columns in the join table).
-- Total number of value differences across common rows.
-- Per-column difference counts (when common columns exist).
-- A sample of the first 10 differences.
-- A follow-up command to inspect the diff (`sqlcompare report <diff_id>`).
+Location:
 
-## Difference Detection
+```
+~/.sqlcompare/connections.yml
+```
 
-`DatabaseComparator` builds a join table with the following shape:
-- For every column in `TABLE1`: `column_previous`
-- For every column in `TABLE2`: `column_new`
+Example:
+
+```yaml
+snowflake:
+  drivername: snowflake
+  username: my_user
+  password: my_password
+  host: my_account
+  database: ANALYTICS
+  schema: PUBLIC
+  query:
+    warehouse: COMPUTE_WH
+```
 
-It uses a `FULL OUTER JOIN` on the provided `IDS`.
+---
 
-Differences are detected when:
-- The row exists on both sides (key columns are not null in both).
-- A common column does not match (or one side is `NULL` and the other is not).
+## Operational notes
 
-Analysis queries are generated on-demand:
-- `get_diff_query`: per-row differences across all common columns
-- `get_stats_query`: counts of differences per column
-- `get_in_current_only_query` / `get_in_previous_only_query`: missing-row checks
+* SQLCompare creates a **physical join table** in the comparison schema. Ensure your connector has `CREATE SCHEMA` and `CREATE TABLE` privileges.
+* Table names are passed through directly to SQL. Provide fully-qualified or quoted names when required by your database.
+* Index columns must exist in both datasets; otherwise the command fails with a clear error.
 
-## Operational Notes
-- The command creates a physical join table in the comparison schema. Ensure the connector has `CREATE SCHEMA` and `CREATE TABLE` privileges.
-- Table names are passed through directly to SQL. Provide fully-qualified or quoted names when required by your database.
-- Index columns must exist in both tables; otherwise the command fails with a clear error that includes a sample of available columns.
+---
 
 ## Development