Add model generation from raw SQL queries

Extended coverage analyzer to bootstrap semantic layers from raw SQL:

Features:
- Generate model definitions from query analysis
- Create rewritten queries using semantic layer
- CLI flag --generate-models to output both models and queries

Usage:
  # Bootstrap from raw queries
  sidemantic coverage --queries raw_queries/ --generate-models output/

This generates:
  - output/models/*.yml - Model definitions (dimensions + metrics)
  - output/rewritten_queries/*.py - Python code using semantic layer

Example included in examples/coverage_analysis/ with 8 sample queries covering:
- Single table aggregations
- Multi-dimensional grouping
- Cross-model joins
- Missing tables (for gap analysis)

Perfect for migrating from raw SQL to semantic layer.

Author: nicosuave

examples/coverage_analysis/README.md

@@ -0,0 +1,100 @@
+# Coverage Analysis Example
+
+This example demonstrates how to bootstrap a semantic layer from raw SQL queries.
+
+## Directory Structure
+
+```
+coverage_analysis/
+├── raw_queries/           # Raw SQL queries from your application
+│   ├── revenue_by_status.sql
+│   ├── customer_demographics.sql
+│   ├── product_performance.sql
+│   ├── monthly_trends.sql
+│   ├── high_value_orders.sql
+│   ├── customer_orders.sql
+│   ├── inventory_analysis.sql
+│   └── cancelled_orders.sql
+└── README.md
+```
+
+## Usage
+
+### Bootstrap Semantic Layer from Queries
+
+Generate model definitions and rewritten queries from your raw SQL:
+
+```bash
+cd examples/coverage_analysis
+
+# Generate models and rewritten queries
+uv run sidemantic coverage --queries raw_queries/ --generate-models output/
+```
+
+This will create:
+- `output/models/` - YAML model definitions for each table
+- `output/rewritten_queries/` - Python code showing how to query using the semantic layer
+
+### Analyze Coverage
+
+If you already have a semantic layer, analyze which queries can be rewritten:
+
+```bash
+# Compare queries against existing semantic layer
+uv run sidemantic coverage models/ --queries raw_queries/
+
+# Show detailed analysis for each query
+uv run sidemantic coverage models/ --queries raw_queries/ --verbose
+```
+
+## What Gets Generated
+
+### Model Definitions
+
+From queries like:
+```sql
+SELECT status, SUM(total_amount), COUNT(*)
+FROM orders
+GROUP BY status
+```
+
+Generates models like:
+```yaml
+model:
+  name: orders
+  table: orders
+  description: Auto-generated from query analysis
+dimensions:
+  - name: status
+    sql: status
+    type: categorical
+metrics:
+  - name: count
+    agg: count
+    sql: '*'
+  - name: sum_total_amount
+    agg: sum
+    sql: total_amount
+```
+
+### Rewritten Queries
+
+Generates Python code to replace raw SQL:
+```python
+# Original query:
+# SELECT status, SUM(total_amount), COUNT(*)
+# FROM orders
+# GROUP BY status
+
+result = layer.query(
+    dimensions=['orders.status'],
+    metrics=['orders.count', 'orders.sum_total_amount']
+)
+```
+
+## Use Cases
+
+1. **Migration** - Bootstrap semantic layer from existing SQL queries
+2. **Discovery** - Find what metrics/dimensions your team actually uses
+3. **Standardization** - Identify inconsistent business logic across queries
+4. **Coverage** - Track how much of your SQL can be replaced with semantic layer

examples/coverage_analysis/raw_queries/cancelled_orders.sql

@@ -0,0 +1,10 @@
+-- Cancelled orders analysis
+SELECT
+    cancellation_reason,
+    COUNT(*) as cancelled_count,
+    SUM(total_amount) as lost_revenue,
+    AVG(total_amount) as avg_order_value
+FROM orders
+WHERE status = 'cancelled'
+GROUP BY cancellation_reason
+ORDER BY cancelled_count DESC;

examples/coverage_analysis/raw_queries/customer_demographics.sql

@@ -0,0 +1,9 @@
+-- Customer demographics analysis
+SELECT
+    region,
+    age_group,
+    COUNT(*) as customer_count,
+    AVG(total_spent) as avg_lifetime_value
+FROM customers
+GROUP BY region, age_group
+ORDER BY customer_count DESC;

examples/coverage_analysis/raw_queries/customer_orders.sql

@@ -0,0 +1,12 @@
+-- Customer order patterns (cross-model query)
+SELECT
+    c.region,
+    c.customer_segment,
+    COUNT(o.order_id) as order_count,
+    SUM(o.total_amount) as total_spent,
+    AVG(o.total_amount) as avg_order_value
+FROM customers c
+JOIN orders o ON c.customer_id = o.customer_id
+WHERE o.status = 'completed'
+GROUP BY c.region, c.customer_segment
+ORDER BY total_spent DESC;

examples/coverage_analysis/raw_queries/high_value_orders.sql

@@ -0,0 +1,11 @@
+-- High value orders analysis
+SELECT
+    status,
+    payment_method,
+    COUNT(*) as order_count,
+    AVG(total_amount) as avg_order_value,
+    MAX(total_amount) as max_order_value
+FROM orders
+WHERE total_amount > 500
+GROUP BY status, payment_method
+ORDER BY avg_order_value DESC;

examples/coverage_analysis/raw_queries/inventory_analysis.sql

@@ -0,0 +1,10 @@
+-- Inventory and sales analysis (table not in semantic layer)
+SELECT
+    warehouse_location,
+    product_category,
+    SUM(quantity_in_stock) as total_inventory,
+    SUM(quantity_sold) as total_sold,
+    AVG(reorder_point) as avg_reorder_point
+FROM inventory
+GROUP BY warehouse_location, product_category
+ORDER BY total_inventory DESC;

examples/coverage_analysis/raw_queries/monthly_trends.sql

@@ -0,0 +1,10 @@
+-- Monthly revenue trends
+SELECT
+    DATE_TRUNC('month', order_date) as month,
+    COUNT(*) as order_count,
+    SUM(total_amount) as revenue,
+    COUNT(DISTINCT customer_id) as unique_customers
+FROM orders
+WHERE order_date >= '2024-01-01'
+GROUP BY DATE_TRUNC('month', order_date)
+ORDER BY month;

examples/coverage_analysis/raw_queries/product_performance.sql

@@ -0,0 +1,13 @@
+-- Product sales performance
+SELECT
+    category,
+    brand,
+    COUNT(DISTINCT product_id) as product_count,
+    SUM(units_sold) as total_units,
+    SUM(revenue) as total_revenue,
+    AVG(price) as avg_price
+FROM products
+GROUP BY category, brand
+HAVING SUM(revenue) > 10000
+ORDER BY total_revenue DESC
+LIMIT 20;

examples/coverage_analysis/raw_queries/revenue_by_status.sql

@@ -0,0 +1,8 @@
+-- Total revenue by order status
+SELECT
+    status,
+    SUM(total_amount) as total_revenue,
+    COUNT(*) as order_count
+FROM orders
+GROUP BY status
+ORDER BY total_revenue DESC;

sidemantic/cli.py

@@ -274,69 +274,130 @@ def info(
 
 @app.command()
 def coverage(
-    directory: Path = typer.Argument(..., help="Directory containing semantic layer files"),
+    directory: Path = typer.Argument(
+        None, help="Directory containing semantic layer files (optional if using --generate-models)"
+    ),
     queries: Path = typer.Option(
         None, "--queries", "-q", help="Path to file or folder containing SQL queries to analyze"
     ),
     verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed analysis for each query"),
+    generate_models: Path = typer.Option(
+        None,
+        "--generate-models",
+        "-g",
+        help="Generate model definitions from queries and write to this directory",
+    ),
 ):
     """
     Analyze SQL queries for semantic layer coverage.
 
     Determines which queries can be rewritten using your semantic layer and
     identifies missing models, dimensions, and metrics.
 
+    Can also bootstrap a semantic layer from raw SQL queries.
+
     Examples:
+      # Analyze coverage
       sidemantic coverage models/ --queries queries/
-      sidemantic coverage models/ --queries query.sql
-      sidemantic coverage models/ --queries queries/ --verbose
+      sidemantic coverage models/ --queries query.sql --verbose
+
+      # Bootstrap semantic layer from raw queries
+      sidemantic coverage --queries raw_queries/ --generate-models output/
     """
     from sidemantic.core.coverage_analyzer import CoverageAnalyzer
 
-    if not directory.exists():
-        typer.echo(f"Error: Directory {directory} does not exist", err=True)
-        raise typer.Exit(1)
-
     if not queries:
         typer.echo("Error: --queries is required", err=True)
-        typer.echo("Usage: sidemantic coverage <models_dir> --queries <path>", err=True)
+        typer.echo("Usage: sidemantic coverage [models_dir] --queries <path>", err=True)
         raise typer.Exit(1)
 
     if not queries.exists():
         typer.echo(f"Error: {queries} does not exist", err=True)
         raise typer.Exit(1)
 
-    try:
-        # Load semantic layer
-        layer = SemanticLayer()
-        load_from_directory(layer, str(directory))
+    # Bootstrap mode - generate models from queries
+    if generate_models:
+        try:
+            # Create empty semantic layer for analysis
+            layer = SemanticLayer(auto_register=False)
+            analyzer = CoverageAnalyzer(layer)
+
+            # Analyze queries
+            if queries.is_file():
+                query_list = queries.read_text().split(";")
+                query_list = [q.strip() for q in query_list if q.strip()]
+                report = analyzer.analyze_queries(query_list)
+            else:
+                report = analyzer.analyze_folder(str(queries))
 
-        if not layer.graph.models:
-            typer.echo("Error: No models found in semantic layer", err=True)
+            # Generate model definitions
+            typer.echo("\nGenerating model definitions...", err=True)
+            models = analyzer.generate_models(report)
+
+            models_dir = generate_models / "models"
+            analyzer.write_model_files(models, str(models_dir))
+
+            # Generate rewritten queries
+            typer.echo("\nGenerating rewritten queries...", err=True)
+            rewritten = analyzer.generate_rewritten_queries(report)
+
+            queries_dir = generate_models / "rewritten_queries"
+            analyzer.write_rewritten_queries(rewritten, str(queries_dir))
+
+            typer.echo(
+                f"\n✓ Generated {len(models)} models and {len(rewritten)} rewritten queries in {generate_models}",
+                err=True,
+            )
+
+        except Exception as e:
+            typer.echo(f"Error: {e}", err=True)
+            import traceback
+
+            traceback.print_exc()
             raise typer.Exit(1)
 
-        # Create analyzer
-        analyzer = CoverageAnalyzer(layer)
+    # Coverage analysis mode - compare queries against existing models
+    else:
+        if not directory:
+            typer.echo("Error: directory is required when not using --generate-models", err=True)
+            typer.echo("Usage: sidemantic coverage <models_dir> --queries <path>", err=True)
+            raise typer.Exit(1)
 
-        # Analyze queries
-        if queries.is_file():
-            # Single file - load queries from it
-            query_list = queries.read_text().split(";")
-            query_list = [q.strip() for q in query_list if q.strip()]
-            report = analyzer.analyze_queries(query_list)
-        else:
-            # Directory - load all .sql files
-            report = analyzer.analyze_folder(str(queries))
+        if not directory.exists():
+            typer.echo(f"Error: Directory {directory} does not exist", err=True)
+            raise typer.Exit(1)
 
-        # Print report
-        analyzer.print_report(report, verbose=verbose)
+        try:
+            # Load semantic layer
+            layer = SemanticLayer()
+            load_from_directory(layer, str(directory))
 
-    except Exception as e:
-        typer.echo(f"Error: {e}", err=True)
-        import traceback
+            if not layer.graph.models:
+                typer.echo("Error: No models found in semantic layer", err=True)
+                raise typer.Exit(1)
 
-        traceback.print_exc()
-        raise typer.Exit(1)
+            # Create analyzer
+            analyzer = CoverageAnalyzer(layer)
+
+            # Analyze queries
+            if queries.is_file():
+                # Single file - load queries from it
+                query_list = queries.read_text().split(";")
+                query_list = [q.strip() for q in query_list if q.strip()]
+                report = analyzer.analyze_queries(query_list)
+            else:
+                # Directory - load all .sql files
+                report = analyzer.analyze_folder(str(queries))
+
+            # Print report
+            analyzer.print_report(report, verbose=verbose)
+
+        except Exception as e:
+            typer.echo(f"Error: {e}", err=True)
+            import traceback
+
+            traceback.print_exc()
+            raise typer.Exit(1)
 
 
 @app.command()