diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 00000000..0a0c9b22 Binary files /dev/null and b/.DS_Store differ diff --git a/Alerts/Alerts.sql b/Alerts/Alerts.sql index daec9eaa..361082ad 100644 --- a/Alerts/Alerts.sql +++ b/Alerts/Alerts.sql @@ -15,7 +15,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..4a926227 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,880 @@ +# Erik Darling's T-SQL Coding Style Guide + +This document outlines the T-SQL coding style preferences for Erik Darling (Darling Data, LLC) and must be strictly followed when writing or modifying SQL code. + +## General Formatting + +- **Keywords**: All SQL keywords in UPPERCASE (SELECT, FROM, WHERE, JOIN, etc.) +- **Functions**: All SQL functions in UPPERCASE (CONVERT, ISNULL, OBJECT_ID, etc.) +- **Data types**: + - Never abbreviate data types (use INTEGER instead of INT) + - All data types must be lowercase (varchar, nvarchar, datetime2, bigint, etc.) + - Length specifications must also be lowercase: nvarchar(max), not nvarchar(MAX) + - Precision and scale specifications must be lowercase: decimal(38,2), not DECIMAL(38,2) +- **Keywords**: Never abbreviate keywords (use EXECUTE instead of EXEC, TRANSACTION instead of TRAN, PROCEDURE instead of PROC) +- **Indentation**: 4 spaces for each level of indentation (NEVER use tabs) +- **Line breaks**: Each statement on a new line +- **Spacing**: Consistent spacing around operators (=, <, >, etc.) +- **Block separation**: Empty line between logical code blocks (maximum of two empty lines between statements) +- **Quotes**: Use single quotes for string literals and N-prefix for Unicode strings (N'string') +- **TOP syntax**: Always include parentheses, as in TOP (100) not TOP 100 +- **Object creation**: Generally use CREATE OR ALTER for objects instead of DROP/CREATE +- **Table aliases**: Tables must always have aliases, even in simple queries +- **Column references**: Always qualify columns with their table alias +- **Commas**: Trailing commas always. + +## Comments + +- Always use block comments with /* ... */ for most comments, never use double dash (--) +- Include parameter descriptions as inline comments after parameter definitions +- Use ASCII art for header blocks to visually distinguish sections +- Include copyright and attribution information in header comments +- Prefix code sections with descriptive comments about what the section does +- Use comments to describe: + - New code blocks + - Complex expressions + - Table purposes + - Complex logic + - The logical flow of code + +## Naming Conventions + +- **Parameters**: Prefixed with @ and use snake_case (@database_name, @debug) +- **Variables**: Same as parameters (@database_id, @sql) +- **Temporary Tables**: Prefixed with # and use descriptive snake_case (#filtered_objects) +- **Aliases**: Short, meaningful lowercase names (ap, o, t) +- **Objects**: Use clear, descriptive names + +## Query Structure + +- **SELECT statements**: + - SELECT keyword on first line + - Column list starts on next line, indented + - Trailing commas for multi-line column lists + - Columns aligned vertically for readability + - FROM clause on new line at same indent level as SELECT + - Column aliases should always use the pattern: column_name = column_expression + - Example: some_date = DATEADD(DAY, 1, GETDATE()) + - Always terminate queries with a semicolon + +- **Table references**: + - Always use schema prefixes for all objects except temporary objects + - Examples: FROM dbo.objects, FROM tempdb.dbo.objects + - Temporary tables don't need schema: FROM #temp_table + +- **Table aliases**: + - Always use the AS keyword with table aliases: table_name AS alias + - Example: FROM dbo.sys_objects AS o + +- **Windowing functions**: + - Format with OVER on same line as function + - PARTITION BY and ORDER BY on separate lines indented + - Parentheses on their own lines + ```sql + SELECT + n = ROW_NUMBER() OVER + ( + PARTITION BY + column_name + ORDER BY + other_column + ) + ``` + +- **JOIN syntax**: + - Use modern ANSI JOIN syntax (JOIN table ON condition) + - JOIN keyword on new line at same indent level as FROM + - ON conditions indented from JOIN + - JOIN conditions with AND should be aligned like this: + ```sql + FROM dbo.table_a AS a0 + JOIN dbo.table_a AS a1 + ON a0.col = a1.col + AND a0.col = a1.col + ``` + - For correlated queries and joins, the table most recently referenced should come first in the ON clause: + ```sql + FROM first_table AS ft0 + JOIN dbo.first_table AS ft1 + ON ft1.col = ft0.col + ``` + +- **Clauses**: + - GROUP BY, ORDER BY, and HAVING clauses should always begin on a new line, indented four spaces from the main statement + - WHERE clauses with AND conditions should be formatted with AND aligned: + ```sql + WHERE a.col = 1 + AND b.col = 2 + ``` + - EXISTS and NOT EXISTS should use this format with 1/0 in the SELECT: + ```sql + WHERE EXISTS + ( + SELECT + 1/0 + FROM other_table AS ot + WHERE ot.col = t.col + ) + ``` + +- **Subqueries**: + - Subqueries should never be one-liners + - Place on new lines with proper indentation + ```sql + SELECT + column_name = + ( + SELECT + column_name + FROM dbo.table_name AS alias + WHERE condition + ) + ``` + +- **APPLY operators**: + - Format CROSS APPLY and OUTER APPLY with the query on new lines + ```sql + FROM dbo.a_table AS y + CROSS APPLY + ( + SELECT + columns + FROM dbo.table_name AS x + WHERE x.col = y.col + ) AS x + ``` + +- **Set operations**: + - UNION, INTERSECT, EXCEPT should have the operator between statements with blank lines + ```sql + SELECT + a.columns + FROM dbo.a_table AS a + + EXCEPT + + SELECT + b.columns + FROM dbo.b_table AS b; + ``` + +- **Table-valued constructors (VALUES)**: + - Format with VALUES on its own line, and value rows indented: + ```sql + FROM + ( + VALUES + (1, 2, 3) + ) AS v (named_columns); + ``` + +- **CTEs**: + - WITH keyword on its own line + - CTE name indented on next line + - Opening parenthesis on same line as CTE name + - Column list indented on subsequent lines + - Closing parenthesis on its own line + - AS keyword on its own line + - Multiple CTEs separated by commas at the end + ```sql + WITH + database_stats + ( + database_name, + recovery_model, + log_size_mb + ) AS + ( + SELECT + database_name = d.name, + recovery_model = d.recovery_model_desc, + log_size_mb = SUM(f.size) * 8 / 1024 + FROM sys.databases AS d + JOIN sys.master_files AS f + ON f.database_id = d.database_id + GROUP BY + d.name, + d.recovery_model_desc + ), + second_cte + ( + column_list + ) AS + ( + query + ) + ``` + +- **Table Creation**: + - CREATE TABLE on first line + - Schema and table name on next line, indented + - Opening parenthesis on its own line + - Each column on a new line, indented + - Always specify NULL or NOT NULL constraint for each column + - DEFAULT constraints can generally follow other column descriptors on the same line + - Closing parenthesis on its own line + ```sql + CREATE TABLE + dbo.table_name + ( + column_name bigint NOT NULL, + another_column varchar(50) NULL DEFAULT 'value', + third_column datetime2(7) NOT NULL DEFAULT SYSDATETIME() + ); + ``` + +- **Index Creation**: + - For multi-column indexes, format with columns on new lines: + ```sql + CREATE INDEX + index_name + ON dbo.table_name + ( + column1, + column2 + ) + INCLUDE + ( + column3, + column4 + ) + WITH + (options); + ``` + - For single-column indexes, a more compact format is acceptable: + ```sql + CREATE INDEX + index_name + ON dbo.table_name + (column1) + INCLUDE + (column3) + WITH + (options); + ``` + +- **INSERT statements**: + - INSERT on first line + - Schema and table name on next line, indented + - Column list in parentheses on new lines, indented + ```sql + INSERT + dbo.table_name + ( + column1, + column2 + ) + VALUES + ( + value1, + value2 + ); + ``` + +- **Temporary table inserts**: + - Use TABLOCK hint with temporary table inserts + ```sql + INSERT + #table_name + WITH + (TABLOCK) + ( + column_list + ) + ``` + +- **UPDATE statements**: + - UPDATE on first line + - Table alias on next line, indented + - SET on its own line with same indentation as alias + - FROM clause on its own line + ```sql + UPDATE + alias + SET + alias.col1 = value1, + alias.col2 = value2 + FROM dbo.table AS alias + WHERE alias.condition; + ``` + +- **DELETE statements**: + - DELETE on first line + - Table alias on next line, indented + - FROM clause on its own line + ```sql + DELETE + alias + FROM dbo.table AS alias + WHERE alias.condition; + ``` + +- **Parentheses**: + - Opening parenthesis on same line as function/procedure name + - Closing parenthesis aligned with starting line or on its own line for long expressions + - Use extra parentheses for clarity in complex expressions + - Function arguments should be indented four spaces and on new lines: + ```sql + CONVERT + ( + data_type, + value + ) + ``` + +- **Multi-parameter functions**: + - For functions with multiple parameters or complex expressions, format the function name on its own line + - Place parameters on subsequent lines with proper indentation + ```sql + SELECT + formatted_date = + DATEFROMPARTS + ( + YEAR(date_column), + MONTH(date_column), + 1 + ) + ``` + +## Code Organization + +- SET statements grouped at procedure start +- Validation checks before main logic +- Help/documentation sections clearly separated from main logic +- Version information tracked explicitly +- Parameter validation at beginning of procedures +- CREATE/ALTER statements separated with GO + +## Code Blocks and Control Structures + +- BEGIN/END contents should be indented four spaces: + ```sql + BEGIN + /*logic*/ + END; + ``` + +- CASE expression contents should be indented, with each condition on a new line: + ```sql + CASE + WHEN thing + AND other_thing + THEN stuff + ELSE result + END + ``` + +- IF/ELSE blocks should be formatted with BEGIN/END on their own lines: + ```sql + IF condition + BEGIN + logic + END; + ELSE + BEGIN + logic + END; + ``` + +- WHILE loops should follow similar formatting: + ```sql + WHILE condition + BEGIN + work + END; + ``` + +- Error handling should follow this template: + ```sql + BEGIN + BEGIN TRY + do stuff + END TRY + BEGIN CATCH + IF @@TRANCOUNT > 0 + BEGIN + ROLLBACK; + END; + + THROW; + END CATCH; + END; + ``` + +- DECLARE blocks should put everything on a new line: + ```sql + DECLARE + @t1 integer, + @t2 integer; + ``` + +- Variables should be declared and initialized together for static values: + ```sql + DECLARE + @t1 integer = 1, + @t2 integer = 2; + ``` + - Take care when initializing to ensure you don't introduce logical flaws with NULL checks + +- Dynamic SQL should follow specific formatting: + - Initial declaration with empty string + - Each string concatenation part on its own line + - Each QUOTENAME or variable reference on its own line + ```sql + DECLARE + @sql nvarchar(max) = N'' + + SET @sql += N' +SELECT + column_name = + value ' + + QUOTENAME(alias.object_name) + N' +FROM + table_name + '; + + EXECUTE sys.sp_executesql + @sql, + N'@parameters', + @input; + ``` + +- Transaction blocks should use consistent indentation: + ```sql + BEGIN TRANSACTION + work + COMMIT TRANSACTION; + ``` + +- XML and JSON output should be formatted with each option on a new line: + ```sql + FOR + XML + PATH + TYPE + ``` + +## SQL Best Practices + +- Always use IS NULL / IS NOT NULL for NULL comparisons, never = NULL or != NULL +- Use ISNULL() function for value replacement +- Include RECOMPILE hints for procedures with variable data distributions +- Use RAISERROR with NOWAIT for immediate message display +- Include thorough error handling with BEGIN TRY/CATCH blocks +- Always validate user inputs before using them +- Use semicolons at the end of statements (but only at the very end, after any query hints) +- Apply query hints consistently (RECOMPILE, MAXDOP, etc.) +- Always use ROWCOUNT_BIG() instead of @@ROWCOUNT +- Always use COUNT_BIG() instead of COUNT() to avoid potential integer overflow + - Example: `COUNT_BIG(i.index_id)` not `COUNT(i.index_id)` + - Even if the result will never be large enough to overflow, use COUNT_BIG() for consistency +- Always use CONVERT over CAST for data type conversions (except when using TRY_CAST, as TRY_CAST isn't dependent on SQL Server version) +- Use XML for string splitting and string building (concatenation), as these methods aren't dependent on SQL Server version or database compatibility level +- Always use cursor variables instead of normal cursors, as they don't require explicit CLOSE/DEALLOCATE statements +- Do not use MERGE statements unless absolutely necessary for functional reasons +- Prefer temporary tables over table variables for performance reasons, especially when the data will be used in joins +- Table variables are acceptable for situations where contents are not used relationally or when insert performance is critical +- Do not drop temporary tables at the end of stored procedures (they're automatically cleaned up when the procedure exits) +- Prefer + operator for string concatenation as it's not version dependent (though CONCAT is acceptable for SQL Server 2012+) +- FORMAT is preferred for adding commas to numbers, but complex CONVERT to money with substring operations is also acceptable +- Date literals should always follow yyyymmdd format (e.g., 20250101), with additional precision as needed for the data type + +## Stored Procedure Structure + +1. SET configuration statements at the top +2. Procedure declaration (CREATE/ALTER) +3. Parameter definitions with inline comments +4. BEGIN block +5. SET NOCOUNT ON and other session settings +6. Variable declarations +7. Validation checks +8. Help section (@help = 1) +9. Main processing logic +10. Error handling +11. Cleanup + +Basic stored procedure outline: +```sql +CREATE OR ALTER PROCEDURE + dbo.procedure_name +( + @parameter_list +) +AS +BEGIN + SET NOCOUNT, XACT_ABORT ON; + + queries... + +END; +``` + +Trigger template: +```sql +CREATE OR ALTER TRIGGER + dbo.a_trigger +ON dbo.a_table +AFTER/INSTEAD OF +AS +BEGIN + IF ROWCOUNT_BIG() = 0 + BEGIN + RETURN + END; + + work + +END; +``` + +View template: +```sql +CREATE OR ALTER VIEW + dbo.a_view +AS +SELECT + column1 = t.column1, + column2 = t.column2 +FROM dbo.table AS t +WHERE t.condition = 1; +``` + +Function template: +```sql +CREATE OR ALTER FUNCTION + dbo.a_function +( + @parameter1 integer, + @parameter2 varchar(50) +) +RETURNS data_type +AS +BEGIN + RETURN value; +END; +``` + +## Examples + +### Complex SELECT with Multiple JOINs, GROUP BY, and HAVING + +```sql +SELECT + database_name = d.name, + index_count = COUNT_BIG(i.index_id), + total_size_mb = SUM(a.total_pages) * 8 / 1024, + read_operations = SUM(ius.user_seeks + ius.user_scans + ius.user_lookups), + write_operations = SUM(ius.user_updates), + avg_fragmentation = AVG(ps.avg_fragmentation_in_percent) +FROM sys.databases AS d +JOIN sys.tables AS t + ON t.database_id = d.database_id +LEFT JOIN sys.indexes AS i + ON i.object_id = t.object_id + AND i.index_id > 0 + AND i.is_disabled = 0 +LEFT JOIN sys.dm_db_index_usage_stats AS ius + ON ius.database_id = d.database_id + AND ius.object_id = i.object_id + AND ius.index_id = i.index_id +LEFT JOIN sys.dm_db_index_physical_stats +( + DB_ID(), + NULL, + NULL, + NULL, + 'LIMITED' +) AS ps + ON ps.object_id = i.object_id + AND ps.index_id = i.index_id +LEFT JOIN sys.allocation_units AS a + ON a.container_id = i.hobt_id +WHERE d.database_id > 4 +AND d.is_read_only = 0 +AND d.state_desc = N'ONLINE' +GROUP BY + d.name, + d.create_date +HAVING + COUNT(i.index_id) > 10 +ORDER BY + total_size_mb DESC, + database_name ASC +OPTION(MAXDOP 1, RECOMPILE); +``` + +### CTE with Multiple Definitions and Nested Queries + +```sql +WITH + database_stats +( + database_name, + recovery_model, + log_size_mb, + log_used_percent +) AS +( + SELECT + database_name = d.name, + recovery_model = d.recovery_model_desc, + log_size_mb = SUM(CASE WHEN f.type_desc = N'LOG' THEN f.size END) * 8 / 1024, + log_used_percent = SUM(CASE WHEN f.type_desc = N'LOG' THEN CONVERT(decimal(19,2), fileproperty(f.name, 'SpaceUsed')) / f.size * 100 END) + FROM sys.databases AS d + JOIN sys.master_files AS f + ON f.database_id = d.database_id + WHERE d.state_desc = N'ONLINE' + GROUP BY + d.name, + d.recovery_model_desc +), + database_backups AS +( + SELECT + database_name = b.database_name, + last_full_backup = MAX(CASE WHEN b.type = 'D' THEN b.backup_finish_date END), + last_log_backup = MAX(CASE WHEN b.type = 'L' THEN b.backup_finish_date END) + FROM msdb.dbo.backupset AS b + WHERE b.backup_finish_date > DATEADD(DAY, -7, GETDATE()) + GROUP BY + b.database_name +) +SELECT + ds.database_name, + ds.recovery_model, + ds.log_size_mb, + ds.log_used_percent, + days_since_full_backup = + CASE + WHEN db.last_full_backup IS NULL + THEN 999 + ELSE DATEDIFF(DAY, db.last_full_backup, GETDATE()) + END, + days_since_log_backup = + CASE + WHEN db.last_log_backup IS NULL + THEN 999 + ELSE DATEDIFF(DAY, db.last_log_backup, GETDATE()) + END +FROM database_stats AS ds +LEFT JOIN database_backups AS db + ON db.database_name = ds.database_name +WHERE ds.log_size_mb > 100 +ORDER BY + log_size_mb DESC; +``` + +### Dynamic SQL Generation and Execution + +```sql +DECLARE + @database_name sysname = N'AdventureWorks', + @table_name sysname = N'SalesOrderHeader', + @column_name sysname = N'OrderDate', + @sql nvarchar(max) = N''; + +/* +Build query dynamically using proper quoting and formatting +*/ +SET @sql = N' +SELECT + order_month = + DATEFROMPARTS + ( + YEAR + (t.' + + QUOTENAME(@column_name) + + N'), + MONTH + (t.' + + QUOTENAME(@column_name) + + N'), + 1 + ), + order_count = COUNT_BIG(*), + total_amount = SUM(t.TotalDue), + avg_amount = AVG(t.TotalDue) +FROM ' + QUOTENAME(@database_name) + N'.dbo.' + QUOTENAME(@table_name) + N' AS t +WHERE ' + QUOTENAME(@column_name) + N' >= DATEADD(YEAR, -1, GETDATE()) +GROUP BY + DATEFROMPARTS + ( + YEAR + (t.' + + QUOTENAME(@column_name) + + N'), + MONTH + (t.' + + QUOTENAME(@column_name) + + N'), + 1 + ) +ORDER BY + order_month; +'; + +/* +Execute the dynamic SQL with proper parameter passing +*/ +EXECUTE sys.sp_executesql + @sql, + N'', + N''; +``` + +### Stored Procedure with Temp Tables and Flow Control + +```sql +SET ANSI_NULLS ON; +SET QUOTED_IDENTIFIER ON; +GO + +IF OBJECT_ID(N'dbo.sp_MyProcedure', N'P') IS NULL +BEGIN + EXECUTE(N'CREATE PROCEDURE dbo.sp_MyProcedure AS RETURN 138;'); +END; +GO + +ALTER PROCEDURE + dbo.sp_MyProcedure +( + @database_name sysname = NULL, /*the database to analyze*/ + @days_back integer = 7, /*how many days of history to analyze*/ + @threshold_percent integer = 20, /*minimum percentage change to report*/ + @debug bit = 0, /*prints additional diagnostic information*/ + @help bit = 0 /*prints help information*/ +) +WITH RECOMPILE +AS +BEGIN + SET NOCOUNT ON; + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + + BEGIN TRY + /* + Variable declarations + */ + DECLARE + @sql nvarchar(max) = N'', + @database_id integer = NULL, + @start_date datetime2(7) = DATEADD(DAY, -@days_back, GETDATE()), + @error_msg nvarchar(2048) = N''; + + /* + Parameter validation + */ + IF @database_name IS NULL + BEGIN + SELECT + @database_name = DB_NAME(); + END; + + IF @threshold_percent <= 0 OR @threshold_percent > 100 + BEGIN + SELECT + @error_msg = N'@threshold_percent must be between 1 and 100.'; + + RAISERROR(@error_msg, 16, 1); + RETURN; + END; + + /* + Help section + */ + IF @help = 1 + BEGIN + SELECT + help = N'This procedure analyzes database performance changes'; + + RETURN; + END; + + /* + Create temp tables for analysis + */ + CREATE TABLE + #baseline_metrics + ( + object_id bigint NOT NULL, + metric_name varchar(50) NOT NULL, + metric_value decimal(38,2) NOT NULL + ); + + CREATE TABLE + #current_metrics + ( + object_id bigint NOT NULL, + metric_name varchar(50) NOT NULL, + metric_value decimal(38,2) NOT NULL + ); + + /* + Populate baseline data + */ + INSERT + #baseline_metrics + WITH + (TABLOCK) + ( + object_id, + metric_name, + metric_value + ) + SELECT + object_id = t.object_id, + metric_name = 'query_cost', + metric_value = AVG(qs.total_elapsed_time / 1000.0) + FROM sys.dm_exec_query_stats AS qs + CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS t + WHERE qs.creation_time < @start_date + AND t.dbid = DB_ID(@database_name) + GROUP BY + t.object_id + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + baseline_rows = COUNT(*) + FROM #baseline_metrics; + END; + + /* + Main processing logic - analyze changes + */ + SELECT + object_name = o.name, + schema_name = s.name, + b.metric_name, + baseline_value = b.metric_value, + current_value = c.metric_value, + percent_change = + CASE + WHEN b.metric_value = 0 + THEN NULL + ELSE (c.metric_value - b.metric_value) / b.metric_value * 100 + END + FROM #baseline_metrics AS b + JOIN #current_metrics AS c + ON c.object_id = b.object_id + AND c.metric_name = b.metric_name + JOIN sys.objects AS o + ON o.object_id = b.object_id + JOIN sys.schemas AS s + ON s.schema_id = o.schema_id + WHERE ABS((c.metric_value - b.metric_value) / NULLIF(b.metric_value, 0) * 100) >= @threshold_percent + ORDER BY + ABS((c.metric_value - b.metric_value) / NULLIF(b.metric_value, 0) * 100) DESC; + END TRY + BEGIN CATCH + IF @@TRANCOUNT > 0 + BEGIN + ROLLBACK; + END; + THROW; + END CATCH; +END; +GO +``` + +This style guide is based on an analysis of Erik Darling's stored procedures from Darling Data, LLC. diff --git a/Clear Token Perm/ClearTokenPerm Agent Job.sql b/Clear Token Perm/ClearTokenPerm Agent Job.sql index 06cedb4a..34b052df 100644 --- a/Clear Token Perm/ClearTokenPerm Agent Job.sql +++ b/Clear Token Perm/ClearTokenPerm Agent Job.sql @@ -20,7 +20,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: @@ -77,7 +77,7 @@ Copyright 2022 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData', +https://code.erikdarling.com', @category_name = N'[Uncategorized (Local)]', @owner_login_name = N'sa', @job_id = @jobId OUTPUT; diff --git a/Clear Token Perm/ClearTokenPerm.sql b/Clear Token Perm/ClearTokenPerm.sql index c70c9e31..9bf7d3db 100644 --- a/Clear Token Perm/ClearTokenPerm.sql +++ b/Clear Token Perm/ClearTokenPerm.sql @@ -20,7 +20,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/Clear Token Perm/Inflate Security Cache Demo And Analysis Script.sql b/Clear Token Perm/Inflate Security Cache Demo And Analysis Script.sql index 02bef443..9b9cd264 100644 --- a/Clear Token Perm/Inflate Security Cache Demo And Analysis Script.sql +++ b/Clear Token Perm/Inflate Security Cache Demo And Analysis Script.sql @@ -17,7 +17,7 @@ Copyright 2024 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com MIT License diff --git a/Clear Token Perm/README.md b/Clear Token Perm/README.md new file mode 100644 index 00000000..50875907 --- /dev/null +++ b/Clear Token Perm/README.md @@ -0,0 +1,75 @@ + + +# Clear Token Perm + +This directory contains scripts for monitoring and managing SQL Server's security token cache. The security token cache (TokenAndPermUserStore) can grow to a significant size in certain scenarios, potentially causing high memory usage and performance issues. + +## Overview + +SQL Server caches security tokens in memory, and in specific environments (particularly with frequent application role usage, or high numbers of users), this cache can grow to consume gigabytes of memory. These scripts provide solutions to monitor the cache size and automatically clear it when it exceeds a defined threshold. + +## Components + +The directory includes three files: + +1. **ClearTokenPerm.sql**: Creates a stored procedure to monitor and clear the security token cache +2. **ClearTokenPerm Agent Job.sql**: Creates a SQL Agent job to run the ClearTokenPerm procedure on a schedule +3. **Inflate Security Cache Demo And Analysis Script.sql**: Demonstrates how to artificially inflate the security cache for testing and provides analysis queries + +## ClearTokenPerm Stored Procedure + +The main stored procedure monitors the size of the TokenAndPermUserStore cache and clears it when it exceeds a specified threshold. + +| Parameter Name | Data Type | Default Value | Description | +|----------------|-----------|---------------|-------------| +| @CacheSizeGB | decimal(38,2) | None (required) | The threshold size in GB that triggers cache clearing | + +The procedure: +- Creates a logging table (ClearTokenPermLogging) if it doesn't exist +- Checks the current size of the TokenAndPermUserStore cache +- Clears the cache using DBCC FREESYSTEMCACHE if the threshold is exceeded +- Logs all checks with timestamp, cache size, and whether clearing was triggered + +## SQL Agent Job + +The Agent job script: +- Creates a job named "Clear Security Cache Every 30 Minutes" +- Runs the ClearTokenPerm procedure with a 1GB threshold +- Schedules execution every 30 minutes +- Includes error handling and transaction support + +## Demo and Analysis Script + +The demo script: +- Creates an application role and executes a loop to inflate the cache +- Provides detailed analysis queries to examine: + - Token distribution in the cache + - Logins and tokens per login + - Users and tokens per user + - Cache invalidations per database +- Includes cleanup steps + +## Usage Examples + +```sql +-- Check and potentially clear the cache if it's over 2GB +EXECUTE dbo.ClearTokenPerm + @CacheSizeGB = 2; + +-- Query the logging table to see history +SELECT + cl.* +FROM dbo.ClearTokenPermLogging AS cl +ORDER BY + cl.log_date DESC; + +-- Clear the logging table +TRUNCATE TABLE dbo.ClearTokenPermLogging; +``` + +## Warning + +The DBCC FREESYSTEMCACHE command used in these scripts will clear the security cache, which may cause a temporary performance impact as the cache is rebuilt. Test thoroughly in non-production environments before deploying to production. + +Copyright 2025 Darling Data, LLC +Released under MIT license diff --git a/Create Long IN List/Longingly.sql b/Create Long IN List/Longingly.sql index 3f022f66..92c190e2 100644 --- a/Create Long IN List/Longingly.sql +++ b/Create Long IN List/Longingly.sql @@ -17,7 +17,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/Helper Views/README.md b/Helper Views/README.md new file mode 100644 index 00000000..2ddec631 --- /dev/null +++ b/Helper Views/README.md @@ -0,0 +1,104 @@ + + +# Helper Views + +This directory contains helper views and functions for diagnosing various SQL Server performance issues. These scripts are particularly useful for presentations and educational purposes, but can also be used in troubleshooting scenarios. + +## Overview + +The collection includes: +- Views for analyzing index sizes +- Functions for examining lock information +- Views for examining memory usage +- Procedures for testing tempdb performance + +## Components + +### WhatsUpIndexes View + +A view that provides detailed information about index sizes in the current database. + +**Functionality**: +- Displays database, schema, table, and index names +- Shows in-row pages size in MB +- Shows LOB (Large Object) pages size in MB +- Reports number of in-row used pages +- Displays row count for each index +- Filters out system objects and table-valued functions + +Usage: +```sql +SELECT + w.* +FROM dbo.WhatsUpIndexes AS w +ORDER BY + w.in_row_mb DESC; +``` + +### WhatsUpLocks Function + +A table-valued function that provides information about locks taken by specific sessions. + +| Parameter Name | Data Type | Default Value | Description | +|----------------|-----------|---------------|-------------| +| @spid | integer | NULL | Session ID to examine. If NULL, returns information for all sessions | + +**Functionality**: +- Displays session ID and blocking session ID information +- Shows lock modes, resource types, and lock status +- Identifies locked objects and associated index names +- Counts different lock types (HOBT, object, page, and row locks) +- Reports total lock count + +Usage: +```sql +-- Check locks for a specific session +SELECT + wul.* +FROM dbo.WhatsUpLocks(51) AS wul; + +-- Check locks for all sessions +SELECT + wul.* +FROM dbo.WhatsUpLocks(NULL) AS wul; +``` + +### WhatsUpMemory View + +A view that examines what's in SQL Server memory. + +**Functionality**: +- Shows database, schema, object, and index information +- Calculates in-row pages in MB (for data types 1 and 3) +- Calculates LOB pages in MB (for data type 2) +- Reports total buffer cache pages + +Usage: +```sql +SELECT + wum.* +FROM dbo.WhatsUpMemory AS wum +ORDER BY + wum.pages DESC; +``` + +### tempdb_tester Procedure + +A stored procedure that generates semi-random tempdb activity, useful for testing and demonstration purposes. + +**Functionality**: +- Creates a temporary table with approximately 10,000 rows +- Performs various DML operations (UPDATE, DELETE, INSERT) +- Uses RECOMPILE hint for optimal execution plans + +Usage: +```sql +EXECUTE dbo.tempdb_tester; +``` + +## Warning + +Some of these scripts (particularly WhatsUpMemory) may cause performance issues if run on busy production servers. Use with caution, especially on servers with large amounts of memory. + +Copyright 2025 Darling Data, LLC +Released under MIT license diff --git a/Helper Views/WhatsUpIndexes.sql b/Helper Views/WhatsUpIndexes.sql index 605b9c71..f714455d 100644 --- a/Helper Views/WhatsUpIndexes.sql +++ b/Helper Views/WhatsUpIndexes.sql @@ -12,7 +12,7 @@ GO /* This is a quick one-off script I use in some presentations to look at index sizes. -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright (c) 2025 Darling Data, LLC https://www.erikdarling.com/ diff --git a/Helper Views/WhatsUpLocks.sql b/Helper Views/WhatsUpLocks.sql index 163c0501..19ab9af5 100644 --- a/Helper Views/WhatsUpLocks.sql +++ b/Helper Views/WhatsUpLocks.sql @@ -13,7 +13,7 @@ GO This is a helper function I use in some of my presentations to look at locks taken. It's definitely not a replacement for sp_WhoIsActive, it just gives me what I care about at the moment. -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright (c) 2025 Darling Data, LLC https://www.erikdarling.com/ diff --git a/Helper Views/WhatsUpMemory.sql b/Helper Views/WhatsUpMemory.sql index ae5c858f..8f5b4402 100644 --- a/Helper Views/WhatsUpMemory.sql +++ b/Helper Views/WhatsUpMemory.sql @@ -15,7 +15,7 @@ I probably wouldn't run this in production, especially on servers with a lot of The dm_os_buffer_descriptors DMV especially can be really slow at times -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright (c) 2025 Darling Data, LLC https://www.erikdarling.com/ diff --git a/Helper Views/tempdb_tester.sql b/Helper Views/tempdb_tester.sql index 8aa6c80f..0eb38d38 100644 --- a/Helper Views/tempdb_tester.sql +++ b/Helper Views/tempdb_tester.sql @@ -17,7 +17,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/Inline Financial Functions/EffectiveAnnualInterestRate.sql b/Inline Financial Functions/EffectiveAnnualInterestRate.sql index 05934eff..616aa5fb 100644 --- a/Inline Financial Functions/EffectiveAnnualInterestRate.sql +++ b/Inline Financial Functions/EffectiveAnnualInterestRate.sql @@ -20,7 +20,7 @@ AS RETURN /* For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ SELECT Rate = diff --git a/Inline Financial Functions/FutureValue.sql b/Inline Financial Functions/FutureValue.sql index 1b37df3c..17a8fd0f 100644 --- a/Inline Financial Functions/FutureValue.sql +++ b/Inline Financial Functions/FutureValue.sql @@ -23,7 +23,7 @@ AS RETURN /* For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ WITH pre AS ( diff --git a/Inline Financial Functions/InterestPayment.sql b/Inline Financial Functions/InterestPayment.sql index d7b7b78e..f3973642 100644 --- a/Inline Financial Functions/InterestPayment.sql +++ b/Inline Financial Functions/InterestPayment.sql @@ -23,7 +23,7 @@ RETURNS TABLE AS /* For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ RETURN WITH pre AS diff --git a/Inline Financial Functions/NumberOfPayments.sql b/Inline Financial Functions/NumberOfPayments.sql index 4c65bbba..66a6a0fb 100644 --- a/Inline Financial Functions/NumberOfPayments.sql +++ b/Inline Financial Functions/NumberOfPayments.sql @@ -25,7 +25,7 @@ AS RETURN /* For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ SELECT NumPayments = diff --git a/Inline Financial Functions/NumberOfYears.sql b/Inline Financial Functions/NumberOfYears.sql index 8b5431c4..65e69375 100644 --- a/Inline Financial Functions/NumberOfYears.sql +++ b/Inline Financial Functions/NumberOfYears.sql @@ -25,7 +25,7 @@ AS RETURN /* For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ SELECT NumberOfYears = diff --git a/Inline Financial Functions/Payment.sql b/Inline Financial Functions/Payment.sql index 7b17318e..790ee7dc 100644 --- a/Inline Financial Functions/Payment.sql +++ b/Inline Financial Functions/Payment.sql @@ -23,7 +23,7 @@ AS RETURN /* For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ WITH pre AS ( diff --git a/Inline Financial Functions/PrincipalPayment.sql b/Inline Financial Functions/PrincipalPayment.sql index 84aa8412..f03c00f5 100644 --- a/Inline Financial Functions/PrincipalPayment.sql +++ b/Inline Financial Functions/PrincipalPayment.sql @@ -24,7 +24,7 @@ AS RETURN /* For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ SELECT PrincipalPayment = diff --git a/Install-All/DarlingData.sql b/Install-All/DarlingData.sql index 0624039d..31974e47 100644 --- a/Install-All/DarlingData.sql +++ b/Install-All/DarlingData.sql @@ -30,7 +30,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ IF OBJECT_ID('dbo.sp_HealthParser') IS NULL @@ -2916,7 +2916,7 @@ EXECUTE sp_HumanEvents @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ @@ -7695,7 +7695,7 @@ EXECUTE sp_HumanEventsBlockViewer @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ IF OBJECT_ID('dbo.sp_HumanEventsBlockViewer') IS NULL @@ -9767,7 +9767,7 @@ SELECT check_id = -1, database_name = N'erikdarling.com', object_name = N'sp_HumanEventsBlockViewer version ' + CONVERT(nvarchar(30), @version) + N'.', - finding_group = N'https://github.com/erikdarlingdata/DarlingData', + finding_group = N'https://code.erikdarling.com', finding = N'blocking for period ' + CONVERT(nvarchar(30), @start_date_original, 126) + N' through ' + CONVERT(nvarchar(30), @end_date_original, 126) + N'.', 1; @@ -10598,7 +10598,7 @@ SELECT check_id = 2147483647, database_name = N'erikdarling.com', object_name = N'sp_HumanEventsBlockViewer version ' + CONVERT(nvarchar(30), @version) + N'.', - finding_group = N'https://github.com/erikdarlingdata/DarlingData', + finding_group = N'https://code.erikdarling.com', finding = N'thanks for using me!', 2147483647; @@ -12267,7 +12267,7 @@ EXECUTE sp_LogHunter @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com EXECUTE sp_LogHunter; @@ -12993,7 +12993,7 @@ EXECUTE sp_PressureDetector @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ @@ -16106,7 +16106,7 @@ EXECUTE sp_QuickieStore @troubleshoot_performance = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ @@ -27237,7 +27237,7 @@ BEGIN all_done = 'https://www.erikdarling.com/', support = - 'https://github.com/erikdarlingdata/DarlingData', + 'https://code.erikdarling.com', help = 'EXECUTE sp_QuickieStore @help = 1;', problems = diff --git a/Install-All/README.md b/Install-All/README.md new file mode 100644 index 00000000..f5cb7ff8 --- /dev/null +++ b/Install-All/README.md @@ -0,0 +1,61 @@ + + +# Install-All + +This directory contains tools to merge all the Darling Data stored procedures into a single installation file. + +## Overview + +Instead of installing each stored procedure individually, you can use the comprehensive DarlingData.sql file to install all procedures at once. This file is automatically generated using the Merge-All.ps1 PowerShell script. + +## Components + +1. **DarlingData.sql**: A merged file containing all stored procedures from the repository +2. **Merge-All.ps1**: PowerShell script that generates the DarlingData.sql file + +## Included Stored Procedures + +The DarlingData.sql file includes all of the following stored procedures: + +| Procedure | Description | Source | +|-----------|-------------|--------| +| [sp_HealthParser](../sp_HealthParser) | Analyzes the system health extended event for performance information | [Link](../sp_HealthParser) | +| [sp_HumanEvents](../sp_HumanEvents) | Makes extended events easy to use for common scenarios | [Link](../sp_HumanEvents) | +| [sp_HumanEventsBlockViewer](../sp_HumanEvents) | Analyzes blocked process reports | [Link](../sp_HumanEvents) | +| [sp_IndexCleanup](../sp_IndexCleanup) | Identifies unused and duplicate indexes | [Link](../sp_IndexCleanup) | +| [sp_LogHunter](../sp_LogHunter) | Searches SQL Server error logs for important messages | [Link](../sp_LogHunter) | +| [sp_PressureDetector](../sp_PressureDetector) | Detects CPU and memory pressure in SQL Server | [Link](../sp_PressureDetector) | +| [sp_QuickieStore](../sp_QuickieStore) | Fast and configurable way to navigate Query Store data | [Link](../sp_QuickieStore) | + +## Usage + +### Using the Pre-Generated File + +Simply run the DarlingData.sql script in SQL Server Management Studio to install all stored procedures at once. + +### Generating a Fresh Copy + +If you want to generate a fresh copy of the DarlingData.sql file: + +1. Make sure you have PowerShell installed +2. Navigate to the Install-All directory in PowerShell +3. Run the script: + +```powershell +.\Merge-All.ps1 +``` + +The script will: +1. Find all sp_* directories +2. Skip sp_WhoIsActive directories +3. Get all sp_* files (skipping agent job files) +4. Combine them into a single file +5. Add a compile date header +6. Save as DarlingData.sql + +## Note + +The WhoIsActive Logging procedures are not included in this file, as they have a different installation process and depend on Adam Machanic's sp_WhoIsActive. + +Copyright 2025 Darling Data, LLC +Released under MIT license \ No newline at end of file diff --git a/Ola Stats Only Job/README.md b/Ola Stats Only Job/README.md new file mode 100644 index 00000000..c510cfa6 --- /dev/null +++ b/Ola Stats Only Job/README.md @@ -0,0 +1,44 @@ + + +# Ola Stats Only Job + +This directory contains a script to create a SQL Server Agent job for nightly statistics updates using Ola Hallengren's maintenance solution. + +## Overview + +Statistics in SQL Server are vital for query optimization but can become stale over time, leading to suboptimal query plans. This script sets up an automated job to update statistics on a regular schedule using Ola Hallengren's popular IndexOptimize stored procedure, focused specifically on statistics updates rather than the full index maintenance. + +## Prerequisites + +- Ola Hallengren's SQL Server Maintenance Solution must be installed + - Download from: [https://ola.hallengren.com/downloads.html](https://ola.hallengren.com/downloads.html) + - This script requires the version from 2018-06-16 or later, which includes the @StatisticsModificationLevel parameter + +## Configuration Details + +The script creates a SQL Agent job with the following default settings: +- Job name: "Nightly Stats Update Job via Ola" +- Database target: All user databases +- Job owner: sa +- Schedule: Every night at midnight +- Statistics modification level: 5% (only updates statistics that have changed by at least 5%) + +## Customization Options + +You may need to modify the script to: +- Change the target database (currently master) +- Change the job owner from sa +- Adjust the schedule from the default midnight run +- Set up failure emails and alerting +- Change the StatisticsModificationLevel from 5% to match your environment needs + +## Usage + +Simply run the script in SQL Server Management Studio after ensuring you have the prerequisites installed. The job will be created and scheduled automatically. + +## Note + +This script focuses exclusively on statistics updates. If you also need index reorganization or rebuilds, you should consider using Ola's full maintenance solution. + +Copyright 2025 Darling Data, LLC +Released under MIT license \ No newline at end of file diff --git a/README.md b/README.md index 7d9a7d95..e403f4d4 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +# Darling Data Logo + # Darling Data: SQL Server Troubleshooting Scripts ![licence badge] @@ -10,6 +12,7 @@ - [sp_QuickieStore](#quickie-store): The fastest and most configurable way to navigate Query Store data - [sp_HealthParser](#health-parser): Pull all the performance-related data from the system health Extended Event - [sp_LogHunter](#log-hunter): Get all of the worst stuff out of your error log + - [sp_IndexCleanup](#index-cleanup): Identify unused and duplicate indexes ## Who are these scripts for? You need to troubleshoot performance problems with SQL Server, and you need to do it now. @@ -60,8 +63,13 @@ Current valid parameter details: | @minimum_disk_latency_ms | smallint | low bound for reporting disk latency | a reasonable number of milliseconds for disk latency | 100 | | @cpu_utilization_threshold | smallint | low bound for reporting high cpu utlization | a reasonable cpu utlization percentage | 50 | | @skip_waits | bit | skips waits when you do not need them on every run | 0 or 1 | 0 | -| @skip_perfmon | bit | skips perfmon counters when you do not need them on every run | 0 or 1 | 0 | -| @sample_seconds | tinyint | take a sample of your server's metrics | a valid tinyint: 0-255 | 0 | +| @skip_perfmon | bit | skips perfmon counters when you do not need them on every run | a valid tinyint: 0-255 | 0 | +| @sample_seconds | tinyint | take a sample of your server's metrics | 0 or 1 | 0 | +| @log_to_table | bit | enable logging to permanent tables | 0 or 1 | 0 | +| @log_database_name | sysname | database to store logging tables | valid database name | NULL | +| @log_schema_name | sysname | schema to store logging tables | valid schema name | NULL | +| @log_table_name_prefix | sysname | prefix for all logging tables | valid table name prefix | 'PressureDetector' | +| @log_retention_days | integer | Number of days to keep logs, 0 = keep indefinitely | integer | 30 | | @help | bit | how you got here | 0 or 1 | 0 | | @debug | bit | prints dynamic sql, displays parameter and variable values, and table contents | 0 or 1 | 0 | | @version | varchar | OUTPUT; for support | none | none; OUTPUT | @@ -98,28 +106,28 @@ Current valid parameter details: | parameter | name | description | valid_inputs | defaults | |------------------------|----------|----------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| | @event_type | sysname | used to pick which session you want to run | "blocking", "query", "waits", "recompiles", "compiles" and certain variations on those words | "query" | -| @query_duration_ms | int | (>=) used to set a minimum query duration to collect data for | an integer | 500 (ms) | +| @query_duration_ms | integer | (>=) used to set a minimum query duration to collect data for | an integer | 500 (ms) | | @query_sort_order | nvarchar | when you use the "query" event, lets you choose which metrics to sort results by | "cpu", "reads", "writes", "duration", "memory", "spills", and you can add "avg" to sort by averages, e.g. "avg cpu" | "cpu" | | @skip_plans | bit | when you use the "query" event, lets you skip collecting actual execution plans |1 or 0 | 0 | -| @blocking_duration_ms | int | (>=) used to set a minimum blocking duration to collect data for | an integer | 500 (ms) | +| @blocking_duration_ms | integer | (>=) used to set a minimum blocking duration to collect data for | an integer | 500 (ms) | | @wait_type | nvarchar | (inclusive) filter to only specific wait types | a single wait type, or a CSV list of wait types | "all", which uses a list of "interesting" waits | -| @wait_duration_ms | int | (>=) used to set a minimum time per wait to collect data for | an integer | 10 (ms) | +| @wait_duration_ms | integer | (>=) used to set a minimum time per wait to collect data for | an integer | 10 (ms) | | @client_app_name | sysname | (inclusive) filter to only specific app names | a stringy thing | intentionally left blank | | @client_hostname | sysname | (inclusive) filter to only specific host names | a stringy thing | intentionally left blank | | @database_name | sysname | (inclusive) filter to only specific databases | a stringy thing | intentionally left blank | | @session_id | nvarchar | (inclusive) filter to only a specific session id, or a sample of session ids | an integer, or "sample" to sample a workload | intentionally left blank | -| @sample_divisor | int | the divisor for session ids when sampling a workload, e.g. SPID % 5 | an integer | 5 | +| @sample_divisor | integer | the divisor for session ids when sampling a workload, e.g. SPID % 5 | an integer | 5 | | @username | sysname | (inclusive) filter to only a specific user | a stringy thing | intentionally left blank | | @object_name | sysname | (inclusive) to only filter to a specific object name | a stringy thing | intentionally left blank | | @object_schema | sysname | (inclusive) the schema of the object you want to filter to; only needed with blocking events | a stringy thing | dbo | -| @requested_memory_mb | int | (>=) the memory grant a query must ask for to have data collected | an integer | 0 | -| @seconds_sample | int | the duration in seconds to run the event session for | an integer | 10 | +| @requested_memory_mb | integer | (>=) the memory grant a query must ask for to have data collected | an integer | 0 | +| @seconds_sample | tinyint | the duration in seconds to run the event session for | an integer | 10 | | @gimme_danger | bit | used to override default minimums for query, wait, and blocking durations. | 1 or 0 | 0 | | @keep_alive | bit | creates a permanent session, either to watch live or log to a table from | 1 or 0 | 0 | | @custom_name | nvarchar | if you want to custom name a permanent session | a stringy thing | intentionally left blank | | @output_database_name | sysname | the database you want to log data to | a valid database name | intentionally left blank | | @output_schema_name | sysname | the schema you want to log data to | a valid schema | dbo | -| @delete_retention_days | int | how many days of logged data you want to keep | a POSITIVE integer | 3 (days) | +| @delete_retention_days | integer | how many days of logged data you want to keep | a POSITIVE integer | 3 (days) | | @cleanup | bit | deletes all sessions, tables, and views. requires output database and schema. | 1 or 0 | 0 | | @max_memory_kb | bigint | set a max ring buffer size to log data to | an integer | 102400 | | @version | varchar | to make sure you have the most recent bits | none, output | none, output | @@ -181,25 +189,35 @@ ON SERVER Once it has data collected, you can analyze it using this command: ``` -EXEC dbo.sp_HumanEventsBlockViewer +EXECUTE dbo.sp_HumanEventsBlockViewer @session_name = N'blocked_process_report'; ``` Current valid parameter details: -| parameter_name | data_type | description | valid_inputs | defaults | -|----------------|-----------|-------------------------------------------------|------------------------------------------------------------------------|------------------------------------| -| @session_name | nvarchar | name of the extended event session to pull from | extended event session name capturing sqlserver.blocked_process_report | keeper_HumanEvents_blocking | -| @target_type | sysname | target of the extended event session | event_file or ring_buffer | NULL | -| @start_date | datetime2 | filter by date | a reasonable date | NULL; will shortcut to last 7 days | -| @end_date | datetime2 | filter by date | a reasonable date | NULL | -| @database_name | sysname | filter by database name | a database that exists on this server | NULL | -| @object_name | sysname | filter by table name | a schema-prefixed table name | NULL | -| @help | bit | how you got here | 0 or 1 | 0 | -| @debug | bit | dumps raw temp table contents | 0 or 1 | 0 | -| @version | varchar | OUTPUT; for support | none; OUTPUT | none; OUTPUT | -| @version_date | datetime | OUTPUT; for support | none; OUTPUT | none; OUTPUT | +| parameter_name | data_type | description | valid_inputs | defaults | +|-----------------------|-----------|-------------------------------------------------|------------------------------------------------------------------------|------------------------------------| +| @session_name | sysname | name of the extended event session to pull from | extended event session name capturing sqlserver.blocked_process_report | keeper_HumanEvents_blocking | +| @target_type | sysname | target of the extended event session | event_file or ring_buffer | NULL | +| @start_date | datetime2 | filter by date | a reasonable date | NULL; will shortcut to last 7 days | +| @end_date | datetime2 | filter by date | a reasonable date | NULL | +| @database_name | sysname | filter by database name | a database that exists on this server | NULL | +| @object_name | sysname | filter by table name | a schema-prefixed table name | NULL | +| @target_database | sysname | database containing the table with BPR data | a valid database name | NULL | +| @target_schema | sysname | schema of the table | a valid schema name | NULL | +| @target_table | sysname | table name | a valid table name | NULL | +| @target_column | sysname | column containing XML data | a valid column name | NULL | +| @timestamp_column | sysname | column containing timestamp (optional) | a valid column name | NULL | +| @log_to_table | bit | enable logging to permanent tables | 0 or 1 | 0 | +| @log_database_name | sysname | database to store logging tables | a valid database name | NULL | +| @log_schema_name | sysname | schema to store logging tables | a valid schema name | NULL | +| @log_table_name_prefix| sysname | prefix for all logging tables | a valid table name prefix | 'HumanEventsBlockViewer' | +| @log_retention_days | integer | Number of days to keep logs, 0 = keep indefinitely | a valid integer | 30 | +| @help | bit | how you got here | 0 or 1 | 0 | +| @debug | bit | dumps raw temp table contents | 0 or 1 | 0 | +| @version | varchar | OUTPUT; for support | none; OUTPUT | none; OUTPUT | +| @version_date | datetime | OUTPUT; for support | none; OUTPUT | none; OUTPUT | [*Back to top*](#navigatory) @@ -273,6 +291,8 @@ Current valid parameter details: | @hide_help_table | bit | hides the "bottom table" that shows help and support information | 0 or 1 | 0 | | @format_output | bit | returns numbers formatted with commas | 0 or 1 | 1 | | @get_all_databases | bit | looks for query store enabled user databases and returns combined results from all of them | 0 or 1 | 0 | +| @include_databases | nvarchar(4000) | comma-separated list of databases to include (only when @get_all_databases = 1) | a string; comma separated database names | NULL | +| @exclude_databases | nvarchar(4000) | comma-separated list of databases to exclude (only when @get_all_databases = 1) | a string; comma separated database names | NULL | | @workdays | bit | use this to filter out weekends and after-hours queries | 0 or 1 | 0 | | @work_start | time | use this to set a specific start of your work days | a time like 8am, 9am or something | 9am | | @work_end | time | use this to set a specific end of your work days | a time like 5pm, 6pm or something | 5pm | @@ -280,6 +300,7 @@ Current valid parameter details: | @regression_baseline_end_date | datetimeoffset | the end date of the baseline that you are checking for regressions against (if any), will be converted to UTC internally | January 1, 1753, through December 31, 9999 | NULL; One week after @regression_baseline_start_date if that is specified | | @regression_comparator | varchar | what difference to use ('relative' or 'absolute') when comparing @sort_order's metric for the normal time period with any regression time period. | relative, absolute | NULL; absolute if @regression_baseline_start_date is specified | | @regression_direction | varchar | when comparing against any regression baseline, what do you want the results sorted by ('magnitude', 'improved', or 'regressed')? | regressed, worse, improved, better, magnitude, absolute, whatever | NULL; regressed if @regression_baseline_start_date is specified | +| @include_query_hash_totals | bit | will add an additional column to final output with total resource usage by query hash | 0 or 1 | 0 | | @help | bit | how you got here | 0 or 1 | 0 | | @debug | bit | prints dynamic sql, statement length, parameter and variable values, and raw temp table contents | 0 or 1 | 0 | | @troubleshoot_performance | bit | set statistics xml on for queries against views | 0 or 1 | 0 | @@ -326,7 +347,12 @@ Current valid parameter details: | @wait_duration_ms | bigint | minimum wait duration | the minimum duration of a wait for queries with interesting waits | 0 | | @wait_round_interval_minutes | bigint | interval to round minutes to for wait stats | interval to round minutes to for top wait stats by count and duration | 60 | | @skip_locks | bit | skip the blocking and deadlocking section | 0 or 1 | 0 | -| @pending_task_threshold | int | minimum number of pending tasks to display | a valid integer | 10 | +| @pending_task_threshold | integer | minimum number of pending tasks to display | a valid integer | 10 | +| @log_to_table | bit | enable logging to permanent tables | 0 or 1 | 0 | +| @log_database_name | sysname | database to store logging tables | valid database name | NULL | +| @log_schema_name | sysname | schema to store logging tables | valid schema name | NULL | +| @log_table_name_prefix | sysname | prefix for all logging tables | valid table name prefix | 'HealthParser' | +| @log_retention_days | integer | Number of days to keep logs, 0 = keep indefinitely | integer | 30 | | @debug | bit | prints dynamic sql, selects from temp tables | 0 or 1 | 0 | | @help | bit | how you got here | 0 or 1 | 0 | | @version | varchar | OUTPUT; for support | none | none; OUTPUT | @@ -353,13 +379,13 @@ Current valid parameter details: | parameter_name | data_type | description | valid_inputs | defaults | |----------------------|-----------|------------------------------------------------|------------------------------------------------------------------------------|--------------| -| @days_back | int | how many days back you want to search the logs | an integer; will be converted to a negative number automatically | -7 | +| @days_back | integer | how many days back you want to search the logs | an integer; will be converted to a negative number automatically | -7 | | @start_date | datetime | if you want to search a specific time frame | a datetime value | NULL | | @end_date | datetime | if you want to search a specific time frame | a datetime value | NULL | | @custom_message | nvarchar | if you want to search for a custom string | something specific you want to search for. no wildcards or substitions. | NULL | | @custom_message_only | bit | only search for the custom string | NULL, 0, 1 | 0 | | @first_log_only | bit | only search through the first error log | NULL, 0, 1 | 0 | -| @language_id | int | to use something other than English | SELECT DISTINCT m.language_id FROM sys.messages AS m ORDER BY m.language_id; | 1033 | +| @language_id | integer | to use something other than English | SELECT DISTINCT m.language_id FROM sys.messages AS m ORDER BY m.language_id; | 1033 | | @help | bit | how you got here | NULL, 0, 1 | 0 | | @debug | bit | dumps raw temp table contents | NULL, 0, 1 | 0 | | @version | varchar | OUTPUT; for support | OUTPUT; for support | none; OUTPUT | @@ -367,4 +393,39 @@ Current valid parameter details: [*Back to top*](#navigatory) +## Index Cleanup + +This stored procedure helps identify unused and duplicate indexes in your SQL Server databases that could be candidates for removal. It analyzes index usage statistics and can generate scripts for removing unnecessary indexes. + +**IMPORTANT: This is currently a BETA VERSION.** It needs extensive testing in real environments with real indexes to address several issues: +* Data collection accuracy +* Deduping logic +* Result correctness +* Edge cases + +Misuse of this procedure can potentially harm your database. If you run this, only use the output to validate result correctness. **Do not run any of the output scripts without thorough review and testing**, as doing so may be harmful to your database performance. + +The procedure requires SQL Server 2012 (11.0) or later due to the use of FORMAT and CONCAT functions. + +Current valid parameter details: + +| Parameter Name | Data Type | Default Value | Description | +|----------------|-----------|---------------|-------------| +| @database_name | sysname | NULL | The name of the database you wish to analyze | +| @schema_name | sysname | NULL | The schema name to filter indexes by | +| @table_name | sysname | NULL | The table name to filter indexes by | +| @min_reads | bigint | 0 | Minimum number of reads for an index to be considered used | +| @min_writes | bigint | 0 | Minimum number of writes for an index to be considered used | +| @min_size_gb | decimal(10,2) | 0 | Minimum size in GB for an index to be analyzed | +| @min_rows | bigint | 0 | Minimum number of rows for a table to be analyzed | +| @get_all_databases | bit | 0 | When set to 1, analyzes all eligible databases on the server | +| @include_databases | nvarchar(max) | NULL | Comma-separated list of databases to include (used with @get_all_databases = 1) | +| @exclude_databases | nvarchar(max) | NULL | Comma-separated list of databases to exclude (used with @get_all_databases = 1) | +| @help | bit | 0 | Displays help information | +| @debug | bit | 0 | Prints debug information during execution | +| @version | varchar(20) | NULL | OUTPUT parameter that returns the version number of the procedure | +| @version_date | datetime | NULL | OUTPUT parameter that returns the date this version was released | + +[*Back to top*](#navigatory) + [licence badge]:https://img.shields.io/badge/license-MIT-blue.svg diff --git a/Stack Column Store Database/MakeBigStackCCS.sql b/Stack Column Store Database/MakeBigStackCCS.sql index 6f582e9e..b269a4fd 100644 --- a/Stack Column Store Database/MakeBigStackCCS.sql +++ b/Stack Column Store Database/MakeBigStackCCS.sql @@ -6,7 +6,7 @@ Copyright (c) 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Permission is hereby granted, free of charge, to any person obtaining a copy diff --git a/Stack Column Store Database/Script StackOverflowCS.sql b/Stack Column Store Database/Script StackOverflowCS.sql index 4acb6084..16ce00fa 100644 --- a/Stack Column Store Database/Script StackOverflowCS.sql +++ b/Stack Column Store Database/Script StackOverflowCS.sql @@ -6,7 +6,7 @@ Copyright (c) 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/String Functions/README.md b/String Functions/README.md new file mode 100644 index 00000000..c016e946 --- /dev/null +++ b/String Functions/README.md @@ -0,0 +1,100 @@ + + +# String Functions + +This directory contains a set of utility functions for string manipulation in SQL Server. These functions provide efficient ways to extract or remove specific characters from strings. + +## Overview + +The functions in this directory help with common string manipulation tasks: +- Extracting only letters from strings +- Extracting only numbers from strings +- Removing specific characters from strings + +Each function is provided in two versions: +1. A version that requires an existing Numbers table +2. A self-contained version with an inline CTE (no external dependencies) + +## Functions + +### get_letters + +Extracts only alphabetic characters (A-Z, a-z) from a string. + +| Parameter Name | Data Type | Description | +|----------------|-----------|-------------| +| @string | nvarchar(4000) | The input string to extract letters from | + +**Return Value**: Table with a single column `letters_only` containing only the letters from the input string. + +Usage: +```sql +SELECT + gl.letters_only +FROM dbo.get_letters(N'abc123!@#') AS gl; +-- Returns: abc + +-- Self-contained version (no dependency on Numbers table) +SELECT + gl.letters_only +FROM dbo.get_letters_cte(N'abc123!@#') AS gl; +-- Returns: abc +``` + +### get_numbers + +Extracts only numeric characters (0-9) from a string. + +| Parameter Name | Data Type | Description | +|----------------|-----------|-------------| +| @string | nvarchar(4000) | The input string to extract numbers from | + +**Return Value**: Table with a single column `numbers_only` containing only the numbers from the input string. + +Usage: +```sql +SELECT + gn.numbers_only +FROM dbo.get_numbers(N'abc123!@#') AS gn; +-- Returns: 123 + +-- Self-contained version (no dependency on Numbers table) +SELECT + gn.numbers_only +FROM dbo.get_numbers_cte(N'abc123!@#') AS gn; +-- Returns: 123 +``` + +### strip_characters + +Removes specified characters from a string. + +| Parameter Name | Data Type | Description | +|----------------|-----------|-------------| +| @string | nvarchar(4000) | The input string to process | +| @match_expression | nvarchar(100) | Characters to remove, specified as a LIKE pattern | + +**Return Value**: Table with a single column `strip_characters` containing the input string with specified characters removed. + +Usage: +```sql +SELECT + sc.strip_characters +FROM dbo.strip_characters(N'abc123!@#', N'[0-9]') AS sc; +-- Returns: abc!@# + +-- Self-contained version (no dependency on Numbers table) +SELECT + sc.strip_characters +FROM dbo.strip_characters_cte(N'abc123!@#', N'[^a-z]') AS sc; +-- Returns: abc +``` + +## Implementation Notes + +- All functions are schema-bound for better performance +- String concatenation is implemented using XML PATH, making the functions independent of SQL Server version +- The _cte variants don't require an external Numbers table, but may be less efficient for very large strings + +Copyright 2025 Darling Data, LLC +Released under MIT license diff --git a/String Functions/get_letters.sql b/String Functions/get_letters.sql index 3c922c08..aed18558 100644 --- a/String Functions/get_letters.sql +++ b/String Functions/get_letters.sql @@ -11,7 +11,7 @@ WITH SCHEMABINDING AS /* For support: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright 2025 Darling Data, LLC https://erikdarling.com @@ -79,7 +79,7 @@ WITH SCHEMABINDING AS /* For support: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright 2025 Darling Data, LLC https://erikdarling.com diff --git a/String Functions/get_numbers.sql b/String Functions/get_numbers.sql index fe91bfbc..0cff51d9 100644 --- a/String Functions/get_numbers.sql +++ b/String Functions/get_numbers.sql @@ -11,7 +11,7 @@ WITH SCHEMABINDING AS /* For support: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright 2025 Darling Data, LLC https://erikdarling.com @@ -80,7 +80,7 @@ WITH SCHEMABINDING AS /* For support: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright 2025 Darling Data, LLC https://erikdarling.com diff --git a/String Functions/strip_characters.sql b/String Functions/strip_characters.sql index a6fb4e0b..93eb8983 100644 --- a/String Functions/strip_characters.sql +++ b/String Functions/strip_characters.sql @@ -12,7 +12,7 @@ WITH SCHEMABINDING AS /* For support: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright 2025 Darling Data, LLC https://erikdarling.com @@ -82,7 +82,7 @@ WITH SCHEMABINDING AS /* For support: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com Copyright 2025 Darling Data, LLC https://erikdarling.com diff --git a/sp_HealthParser/README.md b/sp_HealthParser/README.md new file mode 100644 index 00000000..6b65ee47 --- /dev/null +++ b/sp_HealthParser/README.md @@ -0,0 +1,83 @@ + + +# sp_HealthParser + +The system health extended event has been around for a while, hiding in the shadows, and collecting all sorts of crazy information about your SQL Server. + +The problem is, hardly anyone ever looks at it, and when they do, they realize how awful the Extended Events GUI is. Or that if they want to dig deeper into anything, they're going to have to parse XML. + +This stored procedure takes all that pain away. + +Note that it focuses on performance data, and does not output errors or security details, or any of the other non-performance related data. + +## Results + +Typical result set will show you: +* Queries with significant waits +* Waits by count +* Waits by duration +* Potential I/O issues +* CPU task details +* Memory conditions +* Overall system health +* A limited version of the blocked process report +* XML deadlock report +* Query plans for queries involved in blocking and deadlocks (when available) + +## Parameters + +| parameter_name | data_type | description | valid_inputs | defaults | +|------------------------------|----------------|---------------------------------------------------------------------|-----------------------------------------------------------------------|-----------------| +| @what_to_check | varchar | areas of system health to check | all, waits, disk, cpu, memory, system, locking | all | +| @start_date | datetimeoffset | earliest date to show data for, will be internally converted to UTC | a reasonable date | seven days back | +| @end_date | datetimeoffset | latest date to show data for, will be internally converted to UTC | a reasonable date | current date | +| @warnings_only | bit | only show rows where a warning was reported | NULL, 0, 1 | 0 | +| @database_name | sysname | database name to show blocking events for | the name of a database | NULL | +| @wait_duration_ms | bigint | minimum wait duration | the minimum duration of a wait for queries with interesting waits | 0 | +| @wait_round_interval_minutes | bigint | interval to round minutes to for wait stats | interval to round minutes to for top wait stats by count and duration | 60 | +| @skip_locks | bit | skip the blocking and deadlocking section | 0 or 1 | 0 | +| @pending_task_threshold | integer | minimum number of pending tasks to display | a valid integer | 10 | +| @log_to_table | bit | enable logging to permanent tables | 0 or 1 | 0 | +| @log_database_name | sysname | database to store logging tables | valid database name | NULL | +| @log_schema_name | sysname | schema to store logging tables | valid schema name | NULL | +| @log_table_name_prefix | sysname | prefix for all logging tables | valid table name prefix | 'HealthParser' | +| @log_retention_days | integer | Number of days to keep logs, 0 = keep indefinitely | integer | 30 | +| @debug | bit | prints dynamic sql, selects from temp tables | 0 or 1 | 0 | +| @help | bit | how you got here | 0 or 1 | 0 | +| @version | varchar | OUTPUT; for support | none | none; OUTPUT | +| @version_date | datetime | OUTPUT; for support | none | none; OUTPUT | + +## Examples + +```sql +-- Basic execution for all health checks +EXECUTE dbo.sp_HealthParser; + +-- Check only memory-related issues +EXECUTE dbo.sp_HealthParser + @what_to_check = 'memory'; + +-- Look at health issues for a specific time period +EXECUTE dbo.sp_HealthParser + @start_date = '2025-01-01 00:00:00', + @end_date = '2025-01-02 00:00:00'; + +-- Show only health events with warnings +EXECUTE dbo.sp_HealthParser + @warnings_only = 1; + +-- Focus on blocking issues for a specific database +EXECUTE dbo.sp_HealthParser + @what_to_check = 'locking', + @database_name = 'YourDatabaseName'; + +-- Log results to table instead of returning result sets +EXECUTE dbo.sp_HealthParser + @log_to_table = 1, + @log_database_name = 'DBA', + @log_schema_name = 'dbo', + @log_table_name_prefix = 'HealthParser'; +``` + +## Resources +* [YouTube introduction](https://youtu.be/1kH-aJcCVxs) \ No newline at end of file diff --git a/sp_HealthParser/sp_HealthParser.sql b/sp_HealthParser/sp_HealthParser.sql index 040cdf4c..5cfca6da 100644 --- a/sp_HealthParser/sp_HealthParser.sql +++ b/sp_HealthParser/sp_HealthParser.sql @@ -29,7 +29,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ IF OBJECT_ID(N'dbo.sp_HealthParser', N'P') IS NULL @@ -50,6 +50,11 @@ ALTER PROCEDURE @wait_round_interval_minutes bigint = 60, /*Nearest interval to round wait stats to*/ @skip_locks bit = 0, /*Skip the blocking and deadlocks*/ @pending_task_threshold integer = 10, /*Minimum number of pending tasks to care about*/ + @log_to_table bit = 0, /*enable logging to permanent tables*/ + @log_database_name sysname = NULL, /*database to store logging tables*/ + @log_schema_name sysname = NULL, /*schema to store logging tables*/ + @log_table_name_prefix sysname = 'HealthParser', /*prefix for all logging tables*/ + @log_retention_days integer = 30, /*Number of days to keep logs, 0 = keep indefinitely*/ @debug bit = 0, /*Select from temp tables to get event data in raw xml*/ @help bit = 0, /*Get help*/ @version varchar(30) = NULL OUTPUT, /*Script version*/ @@ -65,8 +70,8 @@ BEGIN SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT - @version = '2.1', - @version_date = '20250101'; + @version = '2.4', + @version_date = '20250401'; IF @help = 1 BEGIN @@ -97,6 +102,11 @@ BEGIN WHEN N'@wait_round_interval_minutes' THEN N'interval to round minutes to for wait stats' WHEN N'@skip_locks' THEN N'skip the blocking and deadlocking section' WHEN N'@pending_task_threshold' THEN N'minimum number of pending tasks to display' + WHEN N'@log_to_table' THEN N'enable logging to permanent tables instead of returning results' + WHEN N'@log_database_name' THEN N'database to store logging tables' + WHEN N'@log_schema_name' THEN N'schema to store logging tables' + WHEN N'@log_table_name_prefix' THEN N'prefix for all logging tables' + WHEN N'@log_retention_days' THEN N'how many days of data to retain' WHEN N'@version' THEN N'OUTPUT; for support' WHEN N'@version_date' THEN N'OUTPUT; for support' WHEN N'@help' THEN N'how you got here' @@ -114,6 +124,11 @@ BEGIN WHEN N'@wait_round_interval_minutes' THEN N'interval to round minutes to for top wait stats by count and duration' WHEN N'@skip_locks' THEN N'0 or 1' WHEN N'@pending_task_threshold' THEN N'a valid integer' + WHEN N'@log_to_table' THEN N'0 or 1' + WHEN N'@log_database_name' THEN N'any valid database name' + WHEN N'@log_schema_name' THEN N'any valid schema name' + WHEN N'@log_table_name_prefix' THEN N'any valid identifier' + WHEN N'@log_retention_days' THEN N'a positive integer' WHEN N'@version' THEN N'none' WHEN N'@version_date' THEN N'none' WHEN N'@help' THEN N'0 or 1' @@ -131,15 +146,20 @@ BEGIN WHEN N'@wait_round_interval_minutes' THEN N'60' WHEN N'@skip_locks' THEN N'0' WHEN N'@pending_task_threshold' THEN N'10' + WHEN N'@log_to_table' THEN N'0' + WHEN N'@log_database_name' THEN N'NULL (current database)' + WHEN N'@log_schema_name' THEN N'NULL (dbo)' + WHEN N'@log_table_name_prefix' THEN N'HealthParser' + WHEN N'@log_retention_days' THEN N'30' WHEN N'@version' THEN N'none; OUTPUT' WHEN N'@version_date' THEN N'none; OUTPUT' WHEN N'@help' THEN N'0' WHEN N'@debug' THEN N'0' END FROM sys.all_parameters AS ap - INNER JOIN sys.all_objects AS o + JOIN sys.all_objects AS o ON ap.object_id = o.object_id - INNER JOIN sys.types AS t + JOIN sys.types AS t ON ap.system_type_id = t.system_type_id AND ap.user_type_id = t.user_type_id WHERE o.name = N'sp_HealthParser' @@ -171,20 +191,20 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -', 0, 1) WITH NOWAIT; +', 0, 0) WITH NOWAIT; RETURN; END; /*End help section*/ IF @debug = 1 BEGIN - RAISERROR('Declaring variables', 0, 1) WITH NOWAIT; + RAISERROR('Declaring variables', 0, 0) WITH NOWAIT; END; DECLARE - @sql nvarchar(MAX) = + @sql nvarchar(max) = N'', - @params nvarchar(MAX) = + @params nvarchar(max) = N'@start_date datetimeoffset(7), @end_date datetimeoffset(7)', @azure bit = @@ -192,7 +212,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. WHEN CONVERT ( - int, + integer, SERVERPROPERTY('EngineEdition') ) = 5 THEN 1 @@ -204,7 +224,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. WHEN CONVERT ( - int, + integer, SERVERPROPERTY('EngineEdition') ) = 8 THEN 1 @@ -212,8 +232,39 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. END, @mi_msg nchar(1), @dbid integer = - DB_ID(@database_name); - + DB_ID(@database_name), + @timestamp_utc_mode tinyint, + @sql_template nvarchar(max) = N'', + @time_filter nvarchar(max) = N'', + @cross_apply nvarchar(max) = N'', + @collection_cursor CURSOR, + @area_name varchar(20), + @object_name sysname, + @temp_table sysname, + @insert_list sysname, + @collection_sql nvarchar(max), + /*Log to table stuff*/ + @log_table_significant_waits sysname, + @log_table_waits_by_count sysname, + @log_table_waits_by_duration sysname, + @log_table_io_issues sysname, + @log_table_cpu_tasks sysname, + @log_table_memory_conditions sysname, + @log_table_memory_broker sysname, + @log_table_memory_node_oom sysname, + @log_table_system_health sysname, + @log_table_scheduler_issues sysname, + @log_table_severe_errors sysname, + @cleanup_date datetime2(7), + @check_sql nvarchar(max) = N'', + @create_sql nvarchar(max) = N'', + @insert_sql nvarchar(max) = N'', + @log_database_schema nvarchar(1024), + @max_event_time datetime2(7), + @dsql nvarchar(max) = N'', + @mdsql_template nvarchar(max) = N'', + @mdsql_execute nvarchar(MAX) = N''; + IF @azure = 1 BEGIN RAISERROR('This won''t work in Azure because it''s horrible', 11, 1) WITH NOWAIT; @@ -222,7 +273,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Fixing variables', 0, 1) WITH NOWAIT; + RAISERROR('Fixing parameters and variables', 0, 0) WITH NOWAIT; END; SELECT @@ -296,35 +347,149 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. @azure_msg = CONVERT(nchar(1), @azure), @mi_msg = - CONVERT(nchar(1), @mi); + CONVERT(nchar(1), @mi), + @timestamp_utc_mode = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM sys.all_columns AS ac + WHERE ac.object_id = OBJECT_ID(N'sys.fn_xe_file_target_read_file') + AND ac.name = N'timestamp_utc' + ) + THEN 1 + + CASE + WHEN + PARSENAME + ( + CONVERT + ( + sysname, + SERVERPROPERTY('PRODUCTVERSION') + ), + 4 + ) > 17 + THEN 1 + ELSE 0 + END + + CASE + WHEN @mi = 1 + THEN 1 + ELSE 0 + END + ELSE 0 + END, + @sql_template += N' + INSERT INTO + {temp_table} + WITH + (TABLOCK) + ( + {insert_list} + ) + SELECT + {object_name} = + ISNULL + ( + xml.{object_name}, + CONVERT(xml, N''event'') + ) + FROM + ( + SELECT + {object_name} = + TRY_CAST(fx.event_data AS xml) + FROM sys.fn_xe_file_target_read_file(N''system_health*.xel'', NULL, NULL, NULL) AS fx + WHERE fx.object_name = N''{object_name}'' {time_filter} + ) AS xml + {cross_apply} + OPTION(RECOMPILE); +', + @mdsql_template = N' + IF OBJECT_ID(''{table_check}'', ''U'') IS NOT NULL + BEGIN + SELECT + @max_event_time = + ISNULL + ( + MAX({date_column}), + DATEADD + ( + MINUTE, + DATEDIFF + ( + MINUTE, + SYSDATETIME(), + GETUTCDATE() + ), + DATEADD + ( + DAY, + -1, + SYSDATETIME() + ) + ) + ) + FROM {table_check}; + END; + '; + IF @timestamp_utc_mode = 0 + BEGIN + /* Pre-2017 handling */ + SET @time_filter = N''; + SET @cross_apply = N'CROSS APPLY xml.{object_name}.nodes(''/event'') AS e(x) +CROSS APPLY (SELECT x.value( ''(@timestamp)[1]'', ''datetimeoffset'' )) ca ([utc_timestamp]) +WHERE ca.utc_timestamp >= @start_date +AND ca.utc_timestamp < @end_date'; + END; + ELSE + BEGIN + /* 2017+ handling */ + SET @cross_apply = N'CROSS APPLY xml.{object_name}.nodes(''/event'') AS e(x)'; + + IF @timestamp_utc_mode = 1 + SET @time_filter = N' + AND CONVERT(datetimeoffset(7), fx.timestamp_utc) BETWEEN @start_date AND @end_date'; + ELSE + SET @time_filter = ' + AND fx.timestamp_utc BETWEEN @start_date AND @end_date'; + END; + + SET @sql_template = + REPLACE + ( + REPLACE + ( + @sql_template, + '{time_filter}', + @time_filter + ), + '{cross_apply}', + @cross_apply + ); + /*If any parameters that expect non-NULL default values get passed in with NULLs, fix them*/ SELECT - @what_to_check = ISNULL(@what_to_check, 'all'), + @what_to_check = LOWER(ISNULL(@what_to_check, 'all')), @warnings_only = ISNULL(@warnings_only, 0), @wait_duration_ms = ISNULL(@wait_duration_ms, 0), @wait_round_interval_minutes = ISNULL(@wait_round_interval_minutes, 60), @skip_locks = ISNULL(@skip_locks, 0), @pending_task_threshold = ISNULL(@pending_task_threshold, 10); - SELECT - @what_to_check = LOWER(@what_to_check); - + /*Validate what to check*/ IF @what_to_check NOT IN - ( - 'all', - 'waits', - 'disk', - 'cpu', - 'memory', - 'system', - 'blocking', - 'blocks', - 'deadlock', - 'deadlocks', - 'locking', - 'locks' - ) + ( + 'all', + 'cpu', + 'disk', + 'locking', + 'memory', + 'system', + 'waits' + ) BEGIN SELECT @what_to_check = @@ -332,21 +497,718 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. WHEN @what_to_check = 'wait' THEN 'waits' WHEN @what_to_check IN - ('blocking', 'blocks', 'deadlock', 'deadlocks', 'lock', 'locks') + ( + 'blocking', 'blocks', + 'deadlock', 'deadlocks', + 'lock', 'locks' + ) THEN 'locking' ELSE 'all' END; END; + /* Validate logging parameters */ + IF @log_to_table = 1 + BEGIN + SELECT + /* Default database name to current database if not specified */ + @log_database_name = ISNULL(@log_database_name, DB_NAME()), + /* Default schema name to dbo if not specified */ + @log_schema_name = ISNULL(@log_schema_name, N'dbo'), + @log_retention_days = + CASE + WHEN @log_retention_days < 0 + THEN ABS(@log_retention_days) + ELSE @log_retention_days + END; + + /* Validate database exists */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM sys.databases AS d + WHERE d.name = @log_database_name + ) + BEGIN + RAISERROR('The specified logging database %s does not exist. Logging will be disabled.', 11, 1, @log_database_name) WITH NOWAIT; + RETURN; + END; + + SET + @log_database_schema = + QUOTENAME(@log_database_name) + + N'.' + + QUOTENAME(@log_schema_name) + + N'.'; + + /* Generate fully qualified table names */ + SELECT + @log_table_significant_waits = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_SignificantWaits'), + @log_table_waits_by_count = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_WaitsByCount'), + @log_table_waits_by_duration = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_WaitsByDuration'), + @log_table_io_issues = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_IOIssues'), + @log_table_cpu_tasks = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_CPUTasks'), + @log_table_memory_conditions = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_MemoryConditions'), + @log_table_memory_broker = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_MemoryBroker'), + @log_table_memory_node_oom = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_MemoryNodeOOM'), + @log_table_system_health = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_SystemHealth'), + @log_table_scheduler_issues = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_SchedulerIssues'), + @log_table_severe_errors = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_SevereErrors'); + + /* Check if schema exists and create it if needed */ + SET @check_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + WHERE s.name = @schema_name + ) + BEGIN + DECLARE + @create_schema_sql nvarchar(max) = N''CREATE SCHEMA '' + QUOTENAME(@schema_name); + + EXECUTE ' + QUOTENAME(@log_database_name) + N'.sys.sp_executesql @create_schema_sql; + IF @debug = 1 BEGIN RAISERROR(''Created schema %s in database %s for logging.'', 0, 1, @schema_name, @db_name) WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @check_sql, + N'@schema_name sysname, + @db_name sysname, + @debug bit', + @log_schema_name, + @log_database_name, + @debug; + + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_SignificantWaits'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_significant_waits + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + wait_type nvarchar(60) NULL, + duration_ms nvarchar(30) NULL, + signal_duration_ms nvarchar(30) NULL, + wait_resource nvarchar(256) NULL, + query_text xml NULL, + session_id integer NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for significant waits logging.'', 0, 1, ''' + @log_table_significant_waits + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create WaitsByCount table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_WaitsByCount'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_waits_by_count + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time_rounded datetime2(7) NULL, + wait_type nvarchar(60) NULL, + waits nvarchar(30) NULL, + average_wait_time_ms nvarchar(30) NULL, + max_wait_time_ms nvarchar(30) NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for waits by count logging.'', 0, 1, ''' + @log_table_waits_by_count + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create WaitsByDuration table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_WaitsByDuration'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_waits_by_duration + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time_rounded datetime2(7) NULL, + wait_type nvarchar(60) NULL, + average_wait_time_ms nvarchar(30) NULL, + max_wait_time_ms nvarchar(30) NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for waits by duration logging.'', 0, 1, ''' + @log_table_waits_by_duration + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create IOIssues table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_IOIssues'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_io_issues + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + state nvarchar(256) NULL, + ioLatchTimeouts bigint NULL, + intervalLongIos bigint NULL, + totalLongIos bigint NULL, + longestPendingRequests_duration_ms nvarchar(30) NULL, + longestPendingRequests_filePath nvarchar(500) NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for IO issues logging.'', 0, 1, ''' + @log_table_io_issues + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create CPUTasks table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_CPUTasks'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_cpu_tasks + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + state nvarchar(256) NULL, + maxWorkers bigint NULL, + workersCreated bigint NULL, + workersIdle bigint NULL, + tasksCompletedWithinInterval bigint NULL, + pendingTasks bigint NULL, + oldestPendingTaskWaitingTime bigint NULL, + hasUnresolvableDeadlockOccurred bit NULL, + hasDeadlockedSchedulersOccurred bit NULL, + didBlockingOccur bit NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for CPU tasks logging.'', 0, 1, ''' + @log_table_cpu_tasks + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create MemoryConditions table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_MemoryConditions'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_memory_conditions + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + lastNotification nvarchar(128) NULL, + outOfMemoryExceptions bigint NULL, + isAnyPoolOutOfMemory bit NULL, + processOutOfMemoryPeriod bigint NULL, + name nvarchar(128) NULL, + available_physical_memory_gb bigint NULL, + available_virtual_memory_gb bigint NULL, + available_paging_file_gb bigint NULL, + working_set_gb bigint NULL, + percent_of_committed_memory_in_ws bigint NULL, + page_faults bigint NULL, + system_physical_memory_high bigint NULL, + system_physical_memory_low bigint NULL, + process_physical_memory_low bigint NULL, + process_virtual_memory_low bigint NULL, + vm_reserved_gb bigint NULL, + vm_committed_gb bigint NULL, + locked_pages_allocated bigint NULL, + large_pages_allocated bigint NULL, + emergency_memory_gb bigint NULL, + emergency_memory_in_use_gb bigint NULL, + target_committed_gb bigint NULL, + current_committed_gb bigint NULL, + pages_allocated bigint NULL, + pages_reserved bigint NULL, + pages_free bigint NULL, + pages_in_use bigint NULL, + page_alloc_potential bigint NULL, + numa_growth_phase bigint NULL, + last_oom_factor bigint NULL, + last_os_error bigint NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for memory conditions logging.'', 0, 1, ''' + @log_table_memory_conditions + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create MemoryBroker table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_MemoryBroker'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_memory_broker + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + node_id int NULL, + memory_available_gb nvarchar(30) NULL, + memory_requested_gb nvarchar(30) NULL, + memory_allocator nvarchar(256) NULL, + memory_allocation_type nvarchar(256) NULL, + memory_clerk_name nvarchar(256) NULL, + os_error int NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for memory broker logging.'', 0, 1, ''' + @log_table_memory_broker + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create MemoryNodeOOM table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_MemoryNodeOOM'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_memory_node_oom + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + node_id int NULL, + memory_available_kb nvarchar(30) NULL, + memory_requested_kb nvarchar(30) NULL, + memory_available_mb nvarchar(30) NULL, + memory_requested_mb nvarchar(30) NULL, + memory_allocator nvarchar(256) NULL, + memory_allocation_type nvarchar(256) NULL, + memory_clerk_name nvarchar(256) NULL, + os_error int NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for memory node OOM logging.'', 0, 1, ''' + @log_table_memory_node_oom + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create SystemHealth table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_SystemHealth'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_system_health + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + state nvarchar(256) NULL, + spinlockBackoffs bigint NULL, + sickSpinlockType nvarchar(256) NULL, + sickSpinlockTypeAfterAv nvarchar(256) NULL, + latchWarnings bigint NULL, + isAccessViolationOccurred bigint NULL, + writeAccessViolationCount bigint NULL, + totalDumpRequests bigint NULL, + intervalDumpRequests bigint NULL, + nonYieldingTasksReported bigint NULL, + pageFaults bigint NULL, + systemCpuUtilization bigint NULL, + sqlCpuUtilization bigint NULL, + BadPagesDetected bigint NULL, + BadPagesFixed bigint NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for system health logging.'', 0, 1, ''' + @log_table_system_health + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create SchedulerIssues table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_SchedulerIssues'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_scheduler_issues + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + scheduler_id int NULL, + cpu_id int NULL, + status nvarchar(256) NULL, + is_online bit NULL, + is_runnable bit NULL, + is_running bit NULL, + non_yielding_time_ms nvarchar(30) NULL, + thread_quantum_ms nvarchar(30) NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for scheduler issues logging.'', 0, 1, ''' + @log_table_scheduler_issues + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Create SevereErrors table if it doesn't exist */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_SevereErrors'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_severe_errors + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + event_time datetime2(7) NULL, + error_number int NULL, + severity int NULL, + state int NULL, + message nvarchar(max) NULL, + database_name sysname NULL, + database_id int NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for severe errors logging.'', 0, 1, ''' + @log_table_severe_errors + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Handle log retention if specified */ + IF @log_to_table = 1 AND @log_retention_days > 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Cleaning up log tables older than %i days', 0, 1, @log_retention_days) WITH NOWAIT; + END; + + SET @cleanup_date = + DATEADD + ( + DAY, + -@log_retention_days, + SYSDATETIME() + ); + + /* Clean up each log table */ + SET @dsql = N' + DELETE FROM ' + @log_table_significant_waits + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_waits_by_count + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_waits_by_duration + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_io_issues + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_cpu_tasks + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_memory_conditions + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_memory_broker + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_memory_node_oom + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_system_health + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_scheduler_issues + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_severe_errors + ' + WHERE collection_time < @cleanup_date; + '; + + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql, + N'@cleanup_date datetime2(7)', + @cleanup_date; + + IF @debug = 1 + BEGIN + RAISERROR('Log cleanup complete', 0, 0) WITH NOWAIT; + END; + END; + END; + + IF @debug = 1 + BEGIN + RAISERROR('Creating temp tables', 0, 0) WITH NOWAIT; + END; + + DECLARE + @collection_areas table + ( + id tinyint IDENTITY PRIMARY KEY CLUSTERED, + area_name varchar(20) NOT NULL, + object_name sysname NOT NULL, + temp_table sysname NOT NULL, + insert_list sysname NOT NULL, + should_collect bit NOT NULL DEFAULT 0, + is_processed bit NOT NULL DEFAULT 0 + ); + + INSERT INTO + @collection_areas + ( + area_name, + object_name, + temp_table, + insert_list, + should_collect + ) + SELECT + v.area_name, + v.object_name, + v.temp_table, + v.insert_list, + should_collect = + CASE + WHEN @what_to_check = 'all' + THEN + CASE + WHEN v.area_name = 'locking' + AND @skip_locks = 1 + THEN 0 + ELSE 1 + END + WHEN @what_to_check = v.area_name + THEN 1 + ELSE 0 + END + FROM + ( + VALUES + ('cpu', 'scheduler_monitor_system_health', '#scheduler_monitor', 'scheduler_monitor'), + ('disk', 'sp_server_diagnostics_component_result', '#sp_server_diagnostics_component_result', 'sp_server_diagnostics_component_result'), + ('locking', 'xml_deadlock_report', '#xml_deadlock_report', 'xml_deadlock_report'), + ('locking', 'sp_server_diagnostics_component_result', '#sp_server_diagnostics_component_result', 'sp_server_diagnostics_component_result'), + ('waits', 'wait_info', '#wait_info', 'wait_info'), + ('system', 'sp_server_diagnostics_component_result', '#sp_server_diagnostics_component_result', 'sp_server_diagnostics_component_result'), + ('system', 'error_reported', '#error_reported', 'error_reported'), + ('memory', 'memory_broker_ring_buffer_recorded', '#memory_broker', 'memory_broker'), + ('memory', 'memory_node_oom_ring_buffer_recorded', '#memory_node_oom', 'memory_node_oom') + ) AS v(area_name, object_name, temp_table, insert_list); + IF @debug = 1 BEGIN - RAISERROR('Creating temp tables', 0, 1) WITH NOWAIT; + SELECT + table_name = '@collection_areas', + ca.* + FROM @collection_areas AS ca + ORDER BY + ca.id + OPTION(RECOMPILE); END; CREATE TABLE - #ignore + #ignore_waits + ( + wait_type nvarchar(60) NOT NULL + ); + + CREATE TABLE + #ignore_errors ( - wait_type nvarchar(60) + error_number integer NOT NULL ); CREATE TABLE @@ -367,6 +1229,13 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. xml_deadlock_report xml NOT NULL ); + CREATE TABLE + #blocking_xml + ( + event_time datetime2 NOT NULL, + human_events_xml xml NOT NULL + ); + CREATE TABLE #x ( @@ -379,16 +1248,42 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ring_buffer xml NOT NULL ); + CREATE TABLE + #scheduler_monitor + ( + scheduler_monitor xml NOT NULL + ); + + CREATE TABLE + #error_reported + ( + error_reported xml NOT NULL + ); + + CREATE TABLE + #memory_broker + ( + memory_broker xml NOT NULL + ); + + CREATE TABLE + #memory_node_oom + ( + memory_node_oom xml NOT NULL + ); + /*The more you ignore waits, the worser they get*/ IF @what_to_check IN ('all', 'waits') BEGIN IF @debug = 1 BEGIN - RAISERROR('Inserting ignorable waits to #ignore', 0, 1) WITH NOWAIT; + RAISERROR('Inserting ignorable waits to #ignore_waits', 0, 0) WITH NOWAIT; END; INSERT - #ignore WITH(TABLOCKX) + #ignore_waits + WITH + (TABLOCKX) ( wait_type ) @@ -416,8 +1311,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. N'SQLTRACE_BUFFER_FLUSH', N'SQLTRACE_INCREMENTAL_FLUSH_SLEEP', N'SQLTRACE_WAIT_ENTRIES', N'UCS_SESSION_REGISTRATION', N'VDI_CLIENT_OTHER', N'WAIT_FOR_RESULTS', N'WAIT_XTP_CKPT_CLOSE', N'WAIT_XTP_HOST_WAIT', N'WAIT_XTP_OFFLINE_CKPT_NEW_LOG', N'WAIT_XTP_RECOVERY', N'WAITFOR', N'WAITFOR_TASKSHUTDOWN', - N'XE_DISPATCHER_JOIN', N'XE_DISPATCHER_WAIT', N'XE_FILE_TARGET_TVF', N'XE_LIVE_TARGET_TVF', - N'XE_TIMER_EVENT' + N'XE_DISPATCHER_JOIN', N'XE_DISPATCHER_WAIT', N'XE_FILE_TARGET_TVF', N'XE_LIVE_TARGET_TVF', N'XE_TIMER_EVENT' ) OPTION(RECOMPILE); END; /*End waits ignore*/ @@ -425,298 +1319,225 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN SELECT - table_name = '#ignore', + table_name = '#ignore_waits', i.* - FROM #ignore AS i ORDER BY i.wait_type + FROM #ignore_waits AS i ORDER BY i.wait_type OPTION(RECOMPILE); END; - /* - The column timestamp_utc is 2017+ only, but terribly broken: - https://dba.stackexchange.com/q/323147/32281 - https://feedback.azure.com/d365community/idea/5f8e52d6-f3d2-ec11-a81b-6045bd7ac9f9 - */ - IF EXISTS - ( - SELECT - 1/0 - FROM sys.all_columns AS ac - WHERE ac.object_id = OBJECT_ID(N'sys.fn_xe_file_target_read_file') - AND ac.name = N'timestamp_utc' - ) - AND @mi = 0 + /* First, ensure we're working with the correct collection areas */ + IF @debug = 1 BEGIN - /*Grab data from the wait info component*/ - IF @what_to_check IN ('all', 'waits') - BEGIN - IF @debug = 1 - BEGIN - RAISERROR('Checking waits for not Managed Instance, 2017+', 0, 1) WITH NOWAIT; - END; - - SELECT - @sql = N' - SELECT - wait_info = - ISNULL + RAISERROR('Beginning collection loop for system_health data', 0, 0) WITH NOWAIT; + END; + + /* Declare a cursor to process each collection area */ + SET @collection_cursor = + CURSOR + LOCAL + SCROLL + DYNAMIC + READ_ONLY + FOR + SELECT + ca.area_name, + ca.object_name, + ca.temp_table, + ca.insert_list + FROM @collection_areas AS ca + WHERE ca.should_collect = 1 + AND ca.is_processed = 0 + ORDER BY + ca.id; + + OPEN @collection_cursor; + + FETCH NEXT + FROM @collection_cursor + INTO + @area_name, + @object_name, + @temp_table, + @insert_list; + + WHILE @@FETCH_STATUS = 0 + BEGIN + /* Build the SQL statement for this collection area */ + SET + @collection_sql = + REPLACE + ( + REPLACE ( - xml.wait_info, - CONVERT(xml, N''event'') - ) - FROM - ( - SELECT - wait_info = - TRY_CAST(fx.event_data AS xml) - FROM sys.fn_xe_file_target_read_file(N''system_health*.xel'', NULL, NULL, NULL) AS fx - WHERE fx.object_name = N''wait_info'' - AND CONVERT(datetimeoffset(7), fx.timestamp_utc) BETWEEN @start_date AND @end_date - ) AS xml - CROSS APPLY xml.wait_info.nodes(''/event'') AS e(x) - OPTION(RECOMPILE);'; - - IF @debug = 1 - BEGIN - PRINT @sql; - RAISERROR('Inserting #wait_info', 0, 1) WITH NOWAIT; - SET STATISTICS XML ON; - END; + REPLACE + ( + @sql_template, + '{object_name}', + @object_name + ), + '{temp_table}', + @temp_table + ), + '{insert_list}', + @insert_list + ); + + IF @debug = 1 + BEGIN + RAISERROR('Collecting data for area: %s, object: %s, target table: %s', 0, 1, @area_name, @object_name, @temp_table) WITH NOWAIT; + PRINT @collection_sql; + END; + + IF @debug = 1 + BEGIN + RAISERROR('Executing collection SQL', 0, 0) WITH NOWAIT; + SET STATISTICS XML ON; + END; + + EXECUTE sys.sp_executesql + @collection_sql, + @params, + @start_date, + @end_date; + + IF @debug = 1 + BEGIN + SET STATISTICS XML OFF; + END; - INSERT INTO - #wait_info WITH (TABLOCKX) - ( - wait_info - ) - EXECUTE sys.sp_executesql - @sql, - @params, - @start_date, - @end_date; + UPDATE + @collection_areas + SET + is_processed = 1 + WHERE temp_table = @temp_table + AND should_collect = 1; + + FETCH NEXT + FROM @collection_cursor + INTO + @area_name, + @object_name, + @temp_table, + @insert_list; + END; + + IF @debug = 1 + BEGIN + RAISERROR('Data collection complete', 0, 0) WITH NOWAIT; + END; - IF @debug = 1 - BEGIN - SET STATISTICS XML OFF; - END; + IF @mi = 1 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Starting Managed Instance analysis', 0, 0) WITH NOWAIT; + RAISERROR('Inserting #x', 0, 0) WITH NOWAIT; END; - /*Grab data from the sp_server_diagnostics_component_result component*/ - SELECT - @sql = N' + INSERT + #x + WITH + (TABLOCKX) + ( + x + ) SELECT - sp_server_diagnostics_component_result = + x = ISNULL ( - xml.sp_server_diagnostics_component_result, - CONVERT(xml, N''event'') + TRY_CAST(t.target_data AS xml), + CONVERT(xml, N'event') ) - FROM - ( - SELECT - sp_server_diagnostics_component_result = - TRY_CAST(fx.event_data AS xml) - FROM sys.fn_xe_file_target_read_file(N''system_health*.xel'', NULL, NULL, NULL) AS fx - WHERE fx.object_name = N''sp_server_diagnostics_component_result'' - AND CONVERT(datetimeoffset(7), fx.timestamp_utc) BETWEEN @start_date AND @end_date - ) AS xml - CROSS APPLY xml.sp_server_diagnostics_component_result.nodes(''/event'') AS e(x) - OPTION(RECOMPILE);'; + FROM sys.dm_xe_session_targets AS t + JOIN sys.dm_xe_sessions AS s + ON s.address = t.event_session_address + WHERE s.name = N'system_health' + AND t.target_name = N'ring_buffer' + OPTION(RECOMPILE); IF @debug = 1 BEGIN - PRINT @sql; - RAISERROR('Inserting #sp_server_diagnostics_component_result', 0, 1) WITH NOWAIT; - SET STATISTICS XML ON; + SELECT TOP (100) + table_name = '#x, top 100 rows', + x.* + FROM #x AS x; + END; + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #ring_buffer', 0, 0) WITH NOWAIT; END; - INSERT INTO - #sp_server_diagnostics_component_result WITH(TABLOCKX) + INSERT + #ring_buffer + WITH + (TABLOCKX) ( - sp_server_diagnostics_component_result + ring_buffer ) - EXECUTE sys.sp_executesql - @sql, - @params, - @start_date, - @end_date; + SELECT + x = e.x.query('.') + FROM #x AS x + CROSS APPLY x.x.nodes('//event') AS e(x) + WHERE 1 = 1 + AND e.x.exist('@timestamp[.>= sql:variable("@start_date") and .< sql:variable("@end_date")]') = 1 + AND e.x.exist('@name[.= "security_error_ring_buffer_recorded"]') = 0 + AND e.x.exist('@name[.= "connectivity_ring_buffer_recorded"]') = 0 + OPTION(RECOMPILE); IF @debug = 1 BEGIN - SET STATISTICS XML OFF; + SELECT TOP (100) + table_name = '#ring_buffer, top 100 rows', + x.* + FROM #ring_buffer AS x; END; - /*Grab data from the xml_deadlock_report component*/ - IF - ( - @what_to_check IN ('all', 'locking') - AND @skip_locks = 0 - ) + IF @what_to_check IN ('all', 'waits') BEGIN IF @debug = 1 BEGIN - RAISERROR('Checking locking for not Managed Instance, 2017+', 0, 1) WITH NOWAIT; + RAISERROR('Checking Managed Instance waits', 0, 0) WITH NOWAIT; + RAISERROR('Inserting #wait_info', 0, 0) WITH NOWAIT; END; - SELECT - @sql = N' - SELECT - xml_deadlock_report = - ISNULL - ( - xml.xml_deadlock_report, - CONVERT(xml, N''event'') - ) - FROM + INSERT + #wait_info + WITH + (TABLOCKX) ( - SELECT - xml_deadlock_report = - TRY_CAST(fx.event_data AS xml) - FROM sys.fn_xe_file_target_read_file(N''system_health*.xel'', NULL, NULL, NULL) AS fx - WHERE fx.object_name = N''xml_deadlock_report'' - AND CONVERT(datetimeoffset(7), fx.timestamp_utc) BETWEEN @start_date AND @end_date - ) AS xml - CROSS APPLY xml.xml_deadlock_report.nodes(''/event'') AS e(x) - OPTION(RECOMPILE);'; - - IF @debug = 1 - BEGIN - PRINT @sql; - RAISERROR('Inserting #xml_deadlock_report', 0, 1) WITH NOWAIT; - SET STATISTICS XML ON; - END; - - INSERT INTO - #xml_deadlock_report WITH(TABLOCKX) - ( - xml_deadlock_report + wait_info ) - EXECUTE sys.sp_executesql - @sql, - @params, - @start_date, - @end_date; - - IF @debug = 1 - BEGIN - SET STATISTICS XML OFF; - END; - END; - END; /*End 2016+ data collection*/ - - IF NOT EXISTS - ( - SELECT - 1/0 - FROM sys.all_columns AS ac - WHERE ac.object_id = OBJECT_ID(N'sys.fn_xe_file_target_read_file') - AND ac.name = N'timestamp_utc' - ) - AND @mi = 0 - BEGIN - IF @debug = 1 - BEGIN - RAISERROR('Checking waits for not Managed Instance, up to 2016', 0, 1) WITH NOWAIT; + SELECT + e.x.query('.') + FROM #ring_buffer AS rb + CROSS APPLY rb.ring_buffer.nodes('/event') AS e(x) + WHERE e.x.exist('@name[.= "wait_info"]') = 1 + OPTION(RECOMPILE); END; - - /*Grab data from the wait info component*/ - IF @what_to_check IN ('all', 'waits') + IF @what_to_check IN ('all', 'disk', 'locking', 'system', 'memory') BEGIN - SELECT - @sql = N' - SELECT - wait_info = - ISNULL - ( - xml.wait_info, - CONVERT(xml, N''event'') - ) - FROM - ( - SELECT - wait_info = - TRY_CAST(fx.event_data AS xml) - FROM sys.fn_xe_file_target_read_file(N''system_health*.xel'', NULL, NULL, NULL) AS fx - WHERE fx.object_name = N''wait_info'' - ) AS xml - CROSS APPLY xml.wait_info.nodes(''/event'') AS e(x) - CROSS APPLY (SELECT x.value( ''(@timestamp)[1]'', ''datetimeoffset'' )) ca ([utc_timestamp]) - WHERE ca.utc_timestamp >= @start_date - AND ca.utc_timestamp < @end_date - OPTION(RECOMPILE);'; - IF @debug = 1 BEGIN - PRINT @sql; - RAISERROR('Inserting #wait_info', 0, 1) WITH NOWAIT; - SET STATISTICS XML ON; + RAISERROR('Checking Managed Instance sp_server_diagnostics_component_result', 0, 0) WITH NOWAIT; + RAISERROR('Inserting #sp_server_diagnostics_component_result', 0, 0) WITH NOWAIT; END; - - INSERT INTO - #wait_info WITH (TABLOCKX) - ( - wait_info - ) - EXECUTE sys.sp_executesql - @sql, - @params, - @start_date, - @end_date; - - IF @debug = 1 BEGIN SET STATISTICS XML OFF; END; - END; - - /*Grab data from the sp_server_diagnostics_component_result component*/ - IF @debug = 1 - BEGIN - RAISERROR('Checking sp_server_diagnostics_component_result for not Managed Instance, 2017+', 0, 1) WITH NOWAIT; - END; - - SELECT - @sql = N' - SELECT - sp_server_diagnostics_component_result = - ISNULL - ( - xml.sp_server_diagnostics_component_result, - CONVERT(xml, N''event'') - ) - FROM - ( + + INSERT + #sp_server_diagnostics_component_result + WITH + (TABLOCKX) + ( + sp_server_diagnostics_component_result + ) SELECT - sp_server_diagnostics_component_result = - TRY_CAST(fx.event_data AS xml) - FROM sys.fn_xe_file_target_read_file(N''system_health*.xel'', NULL, NULL, NULL) AS fx - WHERE fx.object_name = N''sp_server_diagnostics_component_result'' - ) AS xml - CROSS APPLY xml.sp_server_diagnostics_component_result.nodes(''/event'') AS e(x) - CROSS APPLY (SELECT x.value( ''(@timestamp)[1]'', ''datetimeoffset'' )) ca ([utc_timestamp]) - WHERE ca.utc_timestamp >= @start_date - AND ca.utc_timestamp < @end_date - OPTION(RECOMPILE);'; - - IF @debug = 1 - BEGIN - RAISERROR('Inserting #sp_server_diagnostics_component_result', 0, 1) WITH NOWAIT; - PRINT @sql; - SET STATISTICS XML ON; - END; - - INSERT INTO - #sp_server_diagnostics_component_result WITH(TABLOCKX) - ( - sp_server_diagnostics_component_result - ) - EXECUTE sys.sp_executesql - @sql, - @params, - @start_date, - @end_date; - - IF @debug = 1 - BEGIN - SET STATISTICS XML OFF; + e.x.query('.') + FROM #ring_buffer AS rb + CROSS APPLY rb.ring_buffer.nodes('/event') AS e(x) + WHERE e.x.exist('@name[.= "sp_server_diagnostics_component_result"]') = 1 + OPTION(RECOMPILE); END; - /*Grab data from the xml_deadlock_report component*/ IF ( @what_to_check IN ('all', 'locking') @@ -725,191 +1546,97 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. BEGIN IF @debug = 1 BEGIN - RAISERROR('Checking locking for not Managed Instance', 0, 1) WITH NOWAIT; - END; - - SELECT - @sql = N' - SELECT - xml_deadlock_report = - ISNULL - ( - xml.xml_deadlock_report, - CONVERT(xml, N''event'') - ) - FROM - ( - SELECT - xml_deadlock_report = - TRY_CAST(fx.event_data AS xml) - FROM sys.fn_xe_file_target_read_file(N''system_health*.xel'', NULL, NULL, NULL) AS fx - WHERE fx.object_name = N''xml_deadlock_report'' - ) AS xml - CROSS APPLY xml.xml_deadlock_report.nodes(''/event'') AS e(x) - CROSS APPLY (SELECT x.value( ''(@timestamp)[1]'', ''datetimeoffset'' )) ca ([utc_timestamp]) - WHERE ca.utc_timestamp >= @start_date - AND ca.utc_timestamp < @end_date - OPTION(RECOMPILE);'; - - IF @debug = 1 - BEGIN - PRINT @sql; - RAISERROR('Inserting #xml_deadlock_report', 0, 1) WITH NOWAIT; - SET STATISTICS XML ON; + RAISERROR('Checking Managed Instance deadlocks', 0, 0) WITH NOWAIT; + RAISERROR('Inserting #xml_deadlock_report', 0, 0) WITH NOWAIT; END; - - INSERT INTO - #xml_deadlock_report WITH(TABLOCKX) + + INSERT + #xml_deadlock_report + WITH + (TABLOCKX) ( xml_deadlock_report ) - EXECUTE sys.sp_executesql - @sql, - @params, - @start_date, - @end_date; + SELECT + e.x.query('.') + FROM #ring_buffer AS rb + CROSS APPLY rb.ring_buffer.nodes('/event') AS e(x) + WHERE e.x.exist('@name[.= "xml_deadlock_report"]') = 1 + OPTION(RECOMPILE); + END; + /* Add scheduler_monitor collection for MI */ + IF @what_to_check IN ('all', 'system', 'cpu') + BEGIN IF @debug = 1 BEGIN - SET STATISTICS XML OFF; + RAISERROR('Checking Managed Instance scheduler monitor', 0, 0) WITH NOWAIT; + RAISERROR('Inserting #scheduler_monitor', 0, 0) WITH NOWAIT; END; - END; - END; /*End < 2017 collection*/ - - IF @mi = 1 - BEGIN - IF @debug = 1 - BEGIN - RAISERROR('Starting Managed Instance analysis', 0, 1) WITH NOWAIT; - RAISERROR('Inserting #x', 0, 1) WITH NOWAIT; - END; - - INSERT - #x WITH(TABLOCKX) - ( - x - ) - SELECT - x = - ISNULL - ( - TRY_CAST(t.target_data AS xml), - CONVERT(xml, N'event') - ) - FROM sys.dm_xe_session_targets AS t - JOIN sys.dm_xe_sessions AS s - ON s.address = t.event_session_address - WHERE s.name = N'system_health' - AND t.target_name = N'ring_buffer' - OPTION(RECOMPILE); - - IF @debug = 1 - BEGIN - SELECT TOP (100) - table_name = '#x, top 100 rows', - x.* - FROM #x AS x; - END; - - IF @debug = 1 - BEGIN - RAISERROR('Inserting #ring_buffer', 0, 1) WITH NOWAIT; - END; - - INSERT - #ring_buffer WITH(TABLOCKX) - ( - ring_buffer - ) - SELECT - x = e.x.query('.') - FROM - ( + + INSERT + #scheduler_monitor + WITH + (TABLOCKX) + ( + scheduler_monitor + ) SELECT - x - FROM #x - ) AS x - CROSS APPLY x.x.nodes('//event') AS e(x) - WHERE 1 = 1 - AND e.x.exist('@timestamp[.>= sql:variable("@start_date") and .< sql:variable("@end_date")]') = 1 - AND e.x.exist('@name[.= "security_error_ring_buffer_recorded"]') = 0 - AND e.x.exist('@name[.= "error_reported"]') = 0 - AND e.x.exist('@name[.= "memory_broker_ring_buffer_recorded"]') = 0 - AND e.x.exist('@name[.= "connectivity_ring_buffer_recorded"]') = 0 - AND e.x.exist('@name[.= "scheduler_monitor_system_health_ring_buffer_recorded"]') = 0 - OPTION(RECOMPILE); - - IF @debug = 1 - BEGIN - SELECT TOP (100) - table_name = '#ring_buffer, top 100 rows', - x.* - FROM #ring_buffer AS x; + e.x.query('.') + FROM #ring_buffer AS rb + CROSS APPLY rb.ring_buffer.nodes('/event') AS e(x) + WHERE e.x.exist('@name[.= "scheduler_monitor_system_health"]') = 1 + OPTION(RECOMPILE); END; - IF @what_to_check IN ('all', 'waits') + /* Add error_reported collection for MI */ + IF @what_to_check IN ('all', 'system') BEGIN IF @debug = 1 BEGIN - RAISERROR('Checking Managed Instance waits', 0, 1) WITH NOWAIT; - RAISERROR('Inserting #wait_info', 0, 1) WITH NOWAIT; + RAISERROR('Checking Managed Instance error reported events', 0, 0) WITH NOWAIT; + RAISERROR('Inserting #error_reported', 0, 0) WITH NOWAIT; END; - + INSERT - #wait_info WITH(TABLOCKX) + #error_reported + WITH + (TABLOCKX) ( - wait_info + error_reported ) SELECT e.x.query('.') FROM #ring_buffer AS rb CROSS APPLY rb.ring_buffer.nodes('/event') AS e(x) - WHERE e.x.exist('@name[.= "wait_info"]') = 1 + WHERE e.x.exist('@name[.= "error_reported"]') = 1 OPTION(RECOMPILE); END; - IF @debug = 1 - BEGIN - RAISERROR('Checking Managed Instance sp_server_diagnostics_component_result', 0, 1) WITH NOWAIT; - RAISERROR('Inserting #sp_server_diagnostics_component_result', 0, 1) WITH NOWAIT; - END; - - INSERT - #sp_server_diagnostics_component_result WITH(TABLOCKX) - ( - sp_server_diagnostics_component_result - ) - SELECT - e.x.query('.') - FROM #ring_buffer AS rb - CROSS APPLY rb.ring_buffer.nodes('/event') AS e(x) - WHERE e.x.exist('@name[.= "sp_server_diagnostics_component_result"]') = 1 - OPTION(RECOMPILE); - - IF - ( - @what_to_check IN ('all', 'locking') - AND @skip_locks = 0 - ) + /* Add memory_broker collection for MI */ + IF @what_to_check IN ('all', 'memory') BEGIN - IF @debug = 1 + IF @debug = 1 BEGIN - RAISERROR('Checking Managed Instance deadlocks', 0, 1) WITH NOWAIT; - RAISERROR('Inserting #xml_deadlock_report', 0, 1) WITH NOWAIT; + RAISERROR('Checking Managed Instance memory broker events', 0, 0) WITH NOWAIT; + RAISERROR('Inserting #memory_broker', 0, 0) WITH NOWAIT; END; - + INSERT - #xml_deadlock_report WITH(TABLOCKX) + #memory_broker + WITH + (TABLOCKX) ( - xml_deadlock_report + memory_broker ) SELECT e.x.query('.') FROM #ring_buffer AS rb CROSS APPLY rb.ring_buffer.nodes('/event') AS e(x) - WHERE e.x.exist('@name[.= "xml_deadlock_report"]') = 1 + WHERE e.x.exist('@name[.= "memory_broker_ring_buffer_recorded"]') = 1 OPTION(RECOMPILE); END; + END; /*End Managed Instance collection*/ IF @debug = 1 @@ -928,6 +1655,26 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. table_name = '#xml_deadlock_report, top 100 rows', x.* FROM #xml_deadlock_report AS x; + + SELECT TOP (100) + table_name = '#scheduler_monitor, top 100 rows', + x.* + FROM #scheduler_monitor AS x; + + SELECT TOP (100) + table_name = '#error_reported, top 100 rows', + x.* + FROM #error_reported AS x; + + SELECT TOP (100) + table_name = '#memory_broker, top 100 rows', + x.* + FROM #memory_broker AS x; + + SELECT TOP (100) + table_name = '#memory_node_oom, top 100 rows', + x.* + FROM #memory_node_oom AS x; END; /*Parse out the wait_info data*/ @@ -935,7 +1682,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. BEGIN IF @debug = 1 BEGIN - RAISERROR('Parsing queries with significant waits', 0, 1) WITH NOWAIT; + RAISERROR('Parsing queries with significant waits', 0, 0) WITH NOWAIT; END; SELECT @@ -963,20 +1710,20 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. CROSS APPLY wi.wait_info.nodes('//event') AS w(x) WHERE w.x.exist('(action[@name="session_id"]/value/text())[.= 0]') = 0 AND w.x.exist('(action[@name="sql_text"]/value/text())') = 1 - AND w.x.exist('(action[@name="sql_text"]/value/text()[contains(., "BACKUP")] )') = 0 + AND w.x.exist('(action[@name="sql_text"]/value/text()[contains(upper-case(.), "BACKUP")] )') = 0 AND w.x.exist('(data[@name="duration"]/value/text())[.>= sql:variable("@wait_duration_ms")]') = 1 AND NOT EXISTS ( SELECT 1/0 - FROM #ignore AS i + FROM #ignore_waits AS i WHERE w.x.exist('(data[@name="wait_type"]/text/text())[1][.= sql:column("i.wait_type")]') = 1 ) OPTION(RECOMPILE); IF @debug = 1 BEGIN - RAISERROR('Adding query_text to #waits_queries', 0, 1) WITH NOWAIT; + RAISERROR('Adding query_text to #waits_queries', 0, 0) WITH NOWAIT; END; ALTER TABLE #waits_queries @@ -1000,6 +1747,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. x.event_time DESC; END; + /* First logging section, queries with significant waits*/ IF NOT EXISTS ( SELECT @@ -1007,27 +1755,40 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. FROM #waits_queries AS wq ) BEGIN - SELECT - finding = - CASE - WHEN @what_to_check NOT IN ('all', 'waits') - THEN 'waits skipped, @what_to_check set to ' + - @what_to_check - WHEN @what_to_check IN ('all', 'waits') - THEN 'no queries with significant waits found between ' + - RTRIM(CONVERT(date, @start_date)) + - ' and ' + - RTRIM(CONVERT(date, @end_date)) + - ' with a minimum duration of ' + - RTRIM(@wait_duration_ms) + - '.' - ELSE 'no queries with significant waits found!' - END; + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'waits') + THEN 'waits skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'waits') + THEN 'no queries with significant waits found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with a minimum duration of ' + + RTRIM(@wait_duration_ms) + + '.' + ELSE 'no queries with significant waits found!' + END; + + RAISERROR('No queries with significant waits found', 0, 0) WITH NOWAIT; + END; END; ELSE BEGIN + /* Build the query */ + SET @dsql = N' SELECT - finding = 'queries with significant waits', + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''queries with significant waits'',' + END + + N' wq.event_time, wq.wait_type, duration_ms = @@ -1043,8 +1804,8 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), signal_duration_ms = REPLACE @@ -1059,8 +1820,8 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), wq.wait_resource, query_text = @@ -1069,20 +1830,94 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. [processing-instruction(query)] = wq.query_text FOR XML - PATH(N''), + PATH(N''''), TYPE ), wq.session_id - FROM #waits_queries AS wq + FROM #waits_queries AS wq'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_significant_waits + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + WHERE wq.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' ORDER BY wq.duration_ms DESC OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_significant_waits + N' + ( + event_time, + wait_type, + duration_ms, + signal_duration_ms, + wait_resource, + query_text, + session_id + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; END; /*Waits by count*/ IF @debug = 1 BEGIN - RAISERROR('Parsing #waits_by_count', 0, 1) WITH NOWAIT; + RAISERROR('Parsing #waits_by_count', 0, 0) WITH NOWAIT; END; SELECT @@ -1113,7 +1948,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ( SELECT 1/0 - FROM #ignore AS i + FROM #ignore_waits AS i WHERE w2.x2.exist('@waitType[.= sql:column("i.wait_type")]') = 1 ) OPTION(RECOMPILE); @@ -1165,6 +2000,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ) OPTION(RECOMPILE); + /* Waits by count logging section */ IF NOT EXISTS ( SELECT @@ -1172,25 +2008,38 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. FROM #tc AS t ) BEGIN - SELECT - finding = - CASE - WHEN @what_to_check NOT IN ('all', 'waits') - THEN 'waits skipped, @what_to_check set to ' + - @what_to_check - WHEN @what_to_check IN ('all', 'waits') - THEN 'no significant waits found between ' + - RTRIM(CONVERT(date, @start_date)) + - ' and ' + - RTRIM(CONVERT(date, @end_date)) + - '.' - ELSE 'no significant waits found!' - END; + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'waits') + THEN 'waits skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'waits') + THEN 'no significant waits found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + '.' + ELSE 'no significant waits found!' + END + + RAISERROR('No waits by count found', 0, 0) WITH NOWAIT; + END; END; ELSE BEGIN + /* Build the query */ + SET @dsql = N' SELECT - t.finding, + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''waits by count'',' + END + + N' t.event_time_rounded, t.wait_type, waits = @@ -1206,8 +2055,8 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), average_wait_time_ms = REPLACE @@ -1222,8 +2071,8 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), max_wait_time_ms = REPLACE @@ -1238,20 +2087,92 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ) - FROM #tc AS t + FROM #tc AS t'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_waits_by_count + ), + '{date_column}', + 'event_time_rounded' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + WHERE t.event_time_rounded > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' ORDER BY t.event_time_rounded DESC, t.waits DESC OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_waits_by_count + N' + ( + event_time_rounded, + wait_type, + waits, + average_wait_time_ms, + max_wait_time_ms + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; END; /*Grab waits by duration*/ IF @debug = 1 BEGIN - RAISERROR('Parsing waits by duration', 0, 1) WITH NOWAIT; + RAISERROR('Parsing waits by duration', 0, 0) WITH NOWAIT; END; SELECT @@ -1283,7 +2204,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ( SELECT 1/0 - FROM #ignore AS i + FROM #ignore_waits AS i WHERE w2.x2.exist('@waitType[.= sql:column("i.wait_type")]') = 1 ) OPTION(RECOMPILE); @@ -1338,6 +2259,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. td.max_wait_time_ms OPTION(RECOMPILE); + /* Waits by duration logging section */ IF NOT EXISTS ( SELECT @@ -1345,27 +2267,40 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. FROM #td AS t ) BEGIN - SELECT - finding = - CASE - WHEN @what_to_check NOT IN ('all', 'waits') - THEN 'waits skipped, @what_to_check set to ' + - @what_to_check - WHEN @what_to_check IN ('all', 'waits') - THEN 'no significant waits found between ' + - RTRIM(CONVERT(date, @start_date)) + - ' and ' + - RTRIM(CONVERT(date, @end_date)) + - ' with a minimum average duration of ' + - RTRIM(@wait_duration_ms) + - '.' - ELSE 'no significant waits found!' - END; + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'waits') + THEN 'waits skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'waits') + THEN 'no significant waits found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with a minimum average duration of ' + + RTRIM(@wait_duration_ms) + + '.' + ELSE 'no significant waits found!' + END + + RAISERROR('No waits by duration', 0, 0) WITH NOWAIT; + END; END; ELSE BEGIN + /* Build the query */ + SET @dsql = N' SELECT - x.finding, + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''waits by duration'',' + END + + N' x.event_time_rounded, x.wait_type, x.average_wait_time_ms, @@ -1389,8 +2324,8 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), average_wait_time_ms = REPLACE @@ -1405,8 +2340,8 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), max_wait_time_ms = REPLACE @@ -1421,8 +2356,8 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), s = ROW_NUMBER() OVER @@ -1444,19 +2379,90 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ) FROM #td AS t ) AS x - WHERE x.n = 1 - ORDER BY - x.s - OPTION(RECOMPILE); - END; - END; /*End wait stats*/ - - /*Grab IO stuff*/ + WHERE x.n = 1'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_waits_by_duration + ), + '{date_column}', + 'event_time_rounded' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + AND x.event_time_rounded > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' + ORDER BY + x.s + OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_waits_by_duration + N' + ( + event_time_rounded, + wait_type, + average_wait_time_ms, + max_wait_time_ms + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; + END; + END; /*End wait stats*/ + + /*Grab IO stuff*/ IF @what_to_check IN ('all', 'disk') BEGIN IF @debug = 1 BEGIN - RAISERROR('Parsing disk stuff', 0, 1) WITH NOWAIT; + RAISERROR('Parsing disk stuff', 0, 0) WITH NOWAIT; END; SELECT @@ -1505,11 +2511,12 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. i.intervalLongIos, i.totalLongIos, longestPendingRequests_duration_ms = - ISNULL(SUM(i.longestPendingRequests_duration_ms), 0), + SUM(i.longestPendingRequests_duration_ms), longestPendingRequests_filePath = ISNULL(i.longestPendingRequests_filePath, 'N/A') INTO #i FROM #io AS i + WHERE i.longestPendingRequests_duration_ms IS NOT NULL GROUP BY i.event_time, i.state, @@ -1519,6 +2526,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ISNULL(i.longestPendingRequests_filePath, 'N/A') OPTION(RECOMPILE); + /* Potential IO issues logging section */ IF NOT EXISTS ( SELECT @@ -1526,27 +2534,39 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. FROM #i AS i ) BEGIN - SELECT - finding = - CASE - WHEN @what_to_check NOT IN ('all', 'disk') - THEN 'disk skipped, @what_to_check set to ' + - @what_to_check - WHEN @what_to_check IN ('all', 'disk') - THEN 'no io issues found between ' + - RTRIM(CONVERT(date, @start_date)) + - ' and ' + - RTRIM(CONVERT(date, @end_date)) + - ' with @warnings_only set to ' + - RTRIM(@warnings_only) + - '.' - ELSE 'no io issues found!' - END; + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'disk') + THEN 'disk skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'disk') + THEN 'no io issues found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with @warnings_only set to ' + + RTRIM(@warnings_only) + + '.' + ELSE 'no io issues found!' + END + RAISERROR('No io data found', 0, 0) WITH NOWAIT; + END; END; ELSE BEGIN + /* Build the query */ + SET @dsql = N' SELECT - i.finding, + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''potential io issues'',' + END + + N' i.event_time, i.state, i.ioLatchTimeouts, @@ -1565,14 +2585,89 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), 1 ), - N'.00', - N'' + N''.00'', + N'''' ), i.longestPendingRequests_filePath - FROM #i AS i + FROM #i AS i'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for IO issues */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_io_issues + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + WHERE i.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' ORDER BY i.event_time DESC OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_io_issues + N' + ( + event_time, + state, + ioLatchTimeouts, + intervalLongIos, + totalLongIos, + longestPendingRequests_duration_ms, + longestPendingRequests_filePath + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; END; END; /*End disk*/ @@ -1581,7 +2676,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. BEGIN IF @debug = 1 BEGIN - RAISERROR('Parsing CPU stuff', 0, 1) WITH NOWAIT; + RAISERROR('Parsing CPU stuff', 0, 0) WITH NOWAIT; END; SELECT @@ -1628,13 +2723,19 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. x.event_time DESC; END; - IF NOT EXISTS - ( - SELECT - 1/0 - FROM #scheduler_details AS sd - ) - BEGIN +END; + + /* CPU task details logging section */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM #scheduler_details AS sd + ) + BEGIN + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ SELECT finding = CASE @@ -1650,12 +2751,22 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. RTRIM(@warnings_only) + '.' ELSE 'no cpu issues found!' - END; + END + + RAISERROR('No scheduler data found', 0, 0) WITH NOWAIT; END; - ELSE - BEGIN + END; + ELSE + BEGIN + /* Build the query */ + SET @dsql = N' SELECT - finding = 'cpu task details', + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''cpu task details'',' + END + + N' sd.event_time, sd.state, sd.maxWorkers, @@ -1667,10 +2778,88 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. sd.hasUnresolvableDeadlockOccurred, sd.hasDeadlockedSchedulersOccurred, sd.didBlockingOccur - FROM #scheduler_details AS sd - ORDER BY - sd.event_time DESC - OPTION(RECOMPILE); + FROM #scheduler_details AS sd'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for CPU task details */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_cpu_tasks + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + WHERE sd.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' + ORDER BY + sd.event_time DESC + OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_cpu_tasks + N' + ( + event_time, + state, + maxWorkers, + workersCreated, + workersIdle, + tasksCompletedWithinInterval, + pendingTasks, + oldestPendingTaskWaitingTime, + hasUnresolvableDeadlockOccurred, + hasDeadlockedSchedulersOccurred, + didBlockingOccur + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; END; END; /*End CPU*/ @@ -1679,7 +2868,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. BEGIN IF @debug = 1 BEGIN - RAISERROR('Parsing memory stuff', 0, 1) WITH NOWAIT; + RAISERROR('Parsing memory stuff', 0, 0) WITH NOWAIT; END; SELECT @@ -1743,6 +2932,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. x.event_time DESC; END; + /* Memory conditions logging section */ IF NOT EXISTS ( SELECT @@ -1750,28 +2940,40 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. FROM #memory AS m ) BEGIN - SELECT - finding = - CASE - WHEN @what_to_check NOT IN ('all', 'memory') - THEN 'memory skipped, @what_to_check set to ' + - @what_to_check - WHEN @what_to_check IN ('all', 'memory') - THEN 'no memory issues found between ' + - RTRIM(CONVERT(date, @start_date)) + - ' and ' + - RTRIM(CONVERT(date, @end_date)) + - ' with @warnings_only set to ' + - RTRIM(@warnings_only) + - '.' - ELSE 'no memory issues found!' - END; + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'memory') + THEN 'memory skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'memory') + THEN 'no memory issues found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with @warnings_only set to ' + + RTRIM(@warnings_only) + + '.' + ELSE 'no memory issues found!' + END + + RAISERROR('No memory condition data found', 0, 0) WITH NOWAIT; + END; END; ELSE BEGIN + /* Build the query */ + SET @dsql = N' SELECT - finding = 'memory conditions', - m.event_time, + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''memory conditions'',' + END + + N'm.event_time, m.lastNotification, m.outOfMemoryExceptions, m.isAnyPoolOutOfMemory, @@ -1803,21 +3005,1028 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. m.numa_growth_phase, m.last_oom_factor, m.last_os_error - FROM #memory AS m + FROM #memory AS m'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for memory conditions */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_memory_conditions + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + WHERE m.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' + ORDER BY + m.event_time DESC + OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_memory_conditions + N' + ( + event_time, + lastNotification, + outOfMemoryExceptions, + isAnyPoolOutOfMemory, + processOutOfMemoryPeriod, + name, + available_physical_memory_gb, + available_virtual_memory_gb, + available_paging_file_gb, + working_set_gb, + percent_of_committed_memory_in_ws, + page_faults, + system_physical_memory_high, + system_physical_memory_low, + process_physical_memory_low, + process_virtual_memory_low, + vm_reserved_gb, + vm_committed_gb, + locked_pages_allocated, + large_pages_allocated, + emergency_memory_gb, + emergency_memory_in_use_gb, + target_committed_gb, + current_committed_gb, + pages_allocated, + pages_reserved, + pages_free, + pages_in_use, + page_alloc_potential, + numa_growth_phase, + last_oom_factor, + last_os_error + )' + + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; + END; + END; /*End memory*/ + + /*Parse memory broker data*/ + IF @what_to_check IN ('all', 'memory') + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parsing memory broker data', 0, 0) WITH NOWAIT; + END; + + SELECT + event_time = + DATEADD + ( + MINUTE, + DATEDIFF + ( + MINUTE, + GETUTCDATE(), + SYSDATETIME() + ), + w.x.value('@timestamp', 'datetime2') + ), + notification_type = w.x.value('(data[@name="notification_type"]/text)[1]', 'nvarchar(256)'), + reclaim_target_kb = w.x.value('(data[@name="reclaim_target_kb"]/value)[1]', 'bigint'), + reclaimed_kb = w.x.value('(data[@name="reclaimed_kb"]/value)[1]', 'bigint'), + pressure = w.x.value('(data[@name="pressure"]/value)[1]', 'bigint'), + currently_available_kb = w.x.value('(data[@name="currently_available_kb"]/value)[1]', 'bigint'), + reserved_kb = w.x.value('(data[@name="reserved_kb"]/value)[1]', 'bigint'), + committed_kb = w.x.value('(data[@name="committed_kb"]/value)[1]', 'bigint'), + worker_count = w.x.value('(data[@name="worker_count"]/value)[1]', 'integer'), + xml = w.x.query('.') + INTO #memory_broker_info + FROM #memory_broker AS mb + CROSS APPLY mb.memory_broker.nodes('//event') AS w(x) + WHERE (w.x.exist('(data[@name="notification_type"]/text[.= "RESOURCE_MEMPHYSICAL_LOW"])') = @warnings_only OR @warnings_only = 0) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT TOP (100) + table_name = '#memory_broker_info, top 100 rows', + x.* + FROM #memory_broker_info AS x + ORDER BY + x.event_time DESC; + END; + + /* Memory broker notifications logging section */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM #memory_broker_info AS mbi + ) + BEGIN + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'memory') + THEN 'memory broker skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'memory') + THEN 'no memory pressure events found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with @warnings_only set to ' + + RTRIM(@warnings_only) + + '.' + ELSE 'no memory pressure events found!' + END + + RAISERROR('No memory broker data found', 0, 0) WITH NOWAIT; + END; + END; + ELSE + BEGIN + /* Build the query for memory node OOM events */ + SET @dsql = N' + SELECT + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''memory node OOM events'',' + END + + N' + mnoi.event_time, + mnoi.node_id, + memory_available_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mnoi.memory_available_kb / 1024.0 / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + memory_requested_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mnoi.memory_requested_kb / 1024.0 / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + mnoi.memory_allocator, + mnoi.memory_allocation_type, + mnoi.memory_clerk_name, + mnoi.os_error + FROM #memory_node_oom_info AS mnoi'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for memory broker */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_memory_broker + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + WHERE mbi.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' + ORDER BY + mbi.event_time DESC + OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO ' + + @log_table_memory_broker + N' + ( + event_time, + node_id, + memory_available_gb, + memory_requested_gb, + memory_allocator, + memory_allocation_type, + memory_clerk_name, + os_error + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; + END; + END; /*End memory broker analysis*/ + + /*Parse memory node OOM data*/ + IF @what_to_check IN ('all', 'memory') + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parsing memory node OOM data', 0, 0) WITH NOWAIT; + END; + + SELECT + event_time = + DATEADD + ( + MINUTE, + DATEDIFF + ( + MINUTE, + GETUTCDATE(), + SYSDATETIME() + ), + w.x.value('@timestamp', 'datetime2') + ), + node_id = w.x.value('(data[@name="id"]/value)[1]', 'int'), + memory_available_kb = w.x.value('(data[@name="availableMemory"]/value)[1]', 'bigint'), + memory_requested_kb = w.x.value('(data[@name="requestedMemory"]/value)[1]', 'bigint'), + memory_allocator = w.x.value('(data[@name="allocator"]/text)[1]', 'nvarchar(256)'), + memory_allocation_type = w.x.value('(data[@name="allocationType"]/text)[1]', 'nvarchar(256)'), + memory_clerk_name = w.x.value('(data[@name="memoryClerk"]/text)[1]', 'nvarchar(256)'), + os_error = w.x.value('(data[@name="oserror"]/value)[1]', 'integer'), + xml = w.x.query('.') + INTO #memory_node_oom_info + FROM #memory_node_oom AS mno + CROSS APPLY mno.memory_node_oom.nodes('//event') AS w(x) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT TOP (100) + table_name = '#memory_node_oom_info, top 100 rows', + x.* + FROM #memory_node_oom_info AS x + ORDER BY + x.event_time DESC; + END; + + /* Memory node OOM events logging section */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM #memory_node_oom_info AS mnoi + ) + BEGIN + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'memory') + THEN 'memory node OOM skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'memory') + THEN 'no memory node OOM events found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + '.' + ELSE 'no memory node OOM events found!' + END + + RAISERROR('No memory oom data found', 0, 0) WITH NOWAIT; + END; + END; + ELSE + BEGIN + /* Build the query for memory broker notifications */ + SET @dsql = N' + SELECT + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''memory broker notifications'',' + END + + N' + mbi.event_time, + mbi.notification_type, + reclaim_target_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mbi.reclaim_target_kb / 1024.0 / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + reclaimed_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mbi.reclaimed_kb / 1024.0 / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + reclaim_success_percent = + CASE + WHEN mbi.reclaim_target_kb > 0 + THEN CONVERT(DECIMAL(5,2), 100.0 * mbi.reclaimed_kb / mbi.reclaim_target_kb) + ELSE 0 + END, + pressure_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mbi.pressure_mb / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + currently_available_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mbi.currently_available_kb / 1024.0 / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + reserved_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mbi.reserved_kb / 1024.0 / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + committed_gb = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + mbi.committed_kb / 1024.0 / 1024.0 + ), + 1 + ), + N''.00'', + N'''' + ), + mbi.worker_count + FROM #memory_broker_info AS mbi'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for memory node OOM */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_memory_node_oom + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql += N' + WHERE mnoi.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql += N' + ORDER BY + mnoi.event_time DESC + OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_memory_node_oom + N' + ( + event_time, + notification_type, + reclaim_target_gb, + reclaimed_gb, + reclaim_success_percent, + pressure_gb, + currently_available_gb, + reserved_gb, + committed_gb, + worker_count + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; + END; + END; /*End memory node OOM analysis*/ + + /*Grab health stuff*/ + IF @what_to_check IN ('all', 'system') + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parsing system stuff', 0, 0) WITH NOWAIT; + END; + + SELECT + event_time = + DATEADD + ( + MINUTE, + DATEDIFF + ( + MINUTE, + GETUTCDATE(), + SYSDATETIME() + ), + w.x.value('@timestamp', 'datetime2') + ), + state = w.x.value('(data[@name="state"]/text/text())[1]', 'nvarchar(256)'), + spinlockBackoffs = w.x.value('(/event/data[@name="data"]/value/system/@spinlockBackoffs)[1]', 'bigint'), + sickSpinlockType = w.x.value('(/event/data[@name="data"]/value/system/@sickSpinlockType)[1]', 'nvarchar(256)'), + sickSpinlockTypeAfterAv = w.x.value('(/event/data[@name="data"]/value/system/@sickSpinlockTypeAfterAv)[1]', 'nvarchar(256)'), + latchWarnings = w.x.value('(/event/data[@name="data"]/value/system/@latchWarnings)[1]', 'bigint'), + isAccessViolationOccurred = w.x.value('(/event/data[@name="data"]/value/system/@isAccessViolationOccurred)[1]', 'bigint'), + writeAccessViolationCount = w.x.value('(/event/data[@name="data"]/value/system/@writeAccessViolationCount)[1]', 'bigint'), + totalDumpRequests = w.x.value('(/event/data[@name="data"]/value/system/@totalDumpRequests)[1]', 'bigint'), + intervalDumpRequests = w.x.value('(/event/data[@name="data"]/value/system/@intervalDumpRequests)[1]', 'bigint'), + nonYieldingTasksReported = w.x.value('(/event/data[@name="data"]/value/system/@nonYieldingTasksReported)[1]', 'bigint'), + pageFaults = w.x.value('(/event/data[@name="data"]/value/system/@pageFaults)[1]', 'bigint'), + systemCpuUtilization = w.x.value('(/event/data[@name="data"]/value/system/@systemCpuUtilization)[1]', 'bigint'), + sqlCpuUtilization = w.x.value('(/event/data[@name="data"]/value/system/@sqlCpuUtilization)[1]', 'bigint'), + BadPagesDetected = w.x.value('(/event/data[@name="data"]/value/system/@BadPagesDetected)[1]', 'bigint'), + BadPagesFixed = w.x.value('(/event/data[@name="data"]/value/system/@BadPagesFixed)[1]', 'bigint'), + xml = w.x.query('.') + INTO #health + FROM #sp_server_diagnostics_component_result AS wi + CROSS APPLY wi.sp_server_diagnostics_component_result.nodes('//event') AS w(x) + WHERE w.x.exist('(data[@name="component"]/text[.= "SYSTEM"])') = 1 + AND (w.x.exist('(data[@name="state"]/text[.= "WARNING"])') = @warnings_only OR @warnings_only = 0) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT TOP (100) + table_name = '#health, top 100 rows', + x.* + FROM #health AS x + ORDER BY + x.event_time DESC; + END; + + /* Overall system health logging section */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM #health AS h + ) + BEGIN + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'system') + THEN 'system health skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'system') + THEN 'no system health issues found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with @warnings_only set to ' + + RTRIM(@warnings_only) + + '.' + ELSE 'no system health issues found!' + END + + RAISERROR('No system health data found', 0, 0) WITH NOWAIT; + END; + END; + ELSE + BEGIN + /* Build the query */ + SET @dsql = N' + SELECT + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''overall system health'',' + END + + N' + h.event_time, + h.state, + h.spinlockBackoffs, + h.sickSpinlockType, + h.sickSpinlockTypeAfterAv, + h.latchWarnings, + h.isAccessViolationOccurred, + h.writeAccessViolationCount, + h.totalDumpRequests, + h.intervalDumpRequests, + h.nonYieldingTasksReported, + h.pageFaults, + h.systemCpuUtilization, + h.sqlCpuUtilization, + h.BadPagesDetected, + h.BadPagesFixed + FROM #health AS h'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for system health */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_system_health + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql = @dsql + N' + WHERE h.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql = @dsql + N' + ORDER BY + h.event_time DESC + OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_system_health + N' + ( + event_time, + state, + spinlockBackoffs, + sickSpinlockType, + sickSpinlockTypeAfterAv, + latchWarnings, + isAccessViolationOccurred, + writeAccessViolationCount, + totalDumpRequests, + intervalDumpRequests, + nonYieldingTasksReported, + pageFaults, + systemCpuUtilization, + sqlCpuUtilization, + BadPagesDetected, + BadPagesFixed + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; + END; + END; /*End system*/ + + /*Parse scheduler monitor data*/ + IF @what_to_check IN ('all', 'system', 'cpu') + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parsing scheduler monitor data', 0, 0) WITH NOWAIT; + END; + + SELECT + event_time = + DATEADD + ( + MINUTE, + DATEDIFF + ( + MINUTE, + GETUTCDATE(), + SYSDATETIME() + ), + w.x.value('@timestamp', 'datetime2') + ), + scheduler_id = w.x.value('(data[@name="scheduler_id"]/value)[1]', 'integer'), + cpu_id = w.x.value('(data[@name="cpu_id"]/value)[1]', 'integer'), + status = w.x.value('(data[@name="status"]/text)[1]', 'nvarchar(256)'), + is_online = w.x.value('(data[@name="is_online"]/value)[1]', 'bit'), + is_runnable = w.x.value('(data[@name="is_runnable"]/value)[1]', 'bit'), + is_running = w.x.value('(data[@name="is_running"]/value)[1]', 'bit'), + non_yielding_time_ms = w.x.value('(data[@name="non_yielding_time"]/value)[1]', 'bigint'), + thread_quantum_ms = w.x.value('(data[@name="thread_quantum"]/value)[1]', 'bigint'), + xml = w.x.query('.') + INTO #scheduler_issues + FROM #scheduler_monitor AS sm + CROSS APPLY sm.scheduler_monitor.nodes('//event') AS w(x) + WHERE (w.x.exist('(data[@name="status"]/text[.= "WARNING"])') = @warnings_only OR @warnings_only = 0) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT TOP (100) + table_name = '#scheduler_issues, top 100 rows', + x.* + FROM #scheduler_issues AS x + ORDER BY + x.event_time DESC; + END; + + /* Scheduler monitor issues logging section */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM #scheduler_issues AS si + ) + BEGIN + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'system', 'cpu') + THEN 'scheduler monitoring skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'system', 'cpu') + THEN 'no scheduler issues found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with @warnings_only set to ' + + RTRIM(@warnings_only) + + '.' + ELSE 'no scheduler issues found!' + END + + RAISERROR('No scheduler issues data found', 0, 0) WITH NOWAIT; + END; + END; + ELSE + BEGIN + /* Build the query */ + SET @dsql = N' + SELECT + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''scheduler monitor issues'',' + END + + N' + si.event_time, + si.scheduler_id, + si.cpu_id, + si.status, + si.is_online, + si.is_runnable, + si.is_running, + non_yielding_time_ms = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + si.non_yielding_time_ms + ), + 1 + ), + N''.00'', + N'''' + ), + thread_quantum_ms = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + si.thread_quantum_ms + ), + 1 + ), + N''.00'', + N'''' + ) + FROM #scheduler_issues AS si'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for scheduler issues */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_scheduler_issues + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql = @dsql + N' + WHERE si.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql = @dsql + N' ORDER BY - m.event_time DESC + si.event_time DESC OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_scheduler_issues + N' + ( + event_time, + scheduler_id, + cpu_id, + status, + is_online, + is_runnable, + is_running, + non_yielding_time_ms, + thread_quantum_ms + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; END; - END; /*End memory*/ - /*Grab health stuff*/ + END; /*End scheduler monitor analysis*/ + + /*Parse error_reported data*/ IF @what_to_check IN ('all', 'system') BEGIN IF @debug = 1 BEGIN - RAISERROR('Parsing system stuff', 0, 1) WITH NOWAIT; + RAISERROR('Parsing error_reported data', 0, 0) WITH NOWAIT; END; + INSERT + #ignore_errors + ( + error_number + ) + VALUES + (17830); + SELECT event_time = DATEADD @@ -1831,89 +4040,178 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ), w.x.value('@timestamp', 'datetime2') ), - state = w.x.value('(data[@name="state"]/text/text())[1]', 'nvarchar(256)'), - spinlockBackoffs = w.x.value('(/event/data[@name="data"]/value/system/@spinlockBackoffs)[1]', 'bigint'), - sickSpinlockType = w.x.value('(/event/data[@name="data"]/value/system/@sickSpinlockType)[1]', 'nvarchar(256)'), - sickSpinlockTypeAfterAv = w.x.value('(/event/data[@name="data"]/value/system/@sickSpinlockTypeAfterAv)[1]', 'nvarchar(256)'), - latchWarnings = w.x.value('(/event/data[@name="data"]/value/system/@latchWarnings)[1]', 'bigint'), - isAccessViolationOccurred = w.x.value('(/event/data[@name="data"]/value/system/@isAccessViolationOccurred)[1]', 'bigint'), - writeAccessViolationCount = w.x.value('(/event/data[@name="data"]/value/system/@writeAccessViolationCount)[1]', 'bigint'), - totalDumpRequests = w.x.value('(/event/data[@name="data"]/value/system/@totalDumpRequests)[1]', 'bigint'), - intervalDumpRequests = w.x.value('(/event/data[@name="data"]/value/system/@intervalDumpRequests)[1]', 'bigint'), - nonYieldingTasksReported = w.x.value('(/event/data[@name="data"]/value/system/@nonYieldingTasksReported)[1]', 'bigint'), - pageFaults = w.x.value('(/event/data[@name="data"]/value/system/@pageFaults)[1]', 'bigint'), - systemCpuUtilization = w.x.value('(/event/data[@name="data"]/value/system/@systemCpuUtilization)[1]', 'bigint'), - sqlCpuUtilization = w.x.value('(/event/data[@name="data"]/value/system/@sqlCpuUtilization)[1]', 'bigint'), - BadPagesDetected = w.x.value('(/event/data[@name="data"]/value/system/@BadPagesDetected)[1]', 'bigint'), - BadPagesFixed = w.x.value('(/event/data[@name="data"]/value/system/@BadPagesFixed)[1]', 'bigint'), + error_number = w.x.value('(data[@name="error_number"]/value)[1]', 'integer'), + severity = w.x.value('(data[@name="severity"]/value)[1]', 'integer'), + state = w.x.value('(data[@name="state"]/value)[1]', 'integer'), + message = w.x.value('(data[@name="message"]/value)[1]', 'nvarchar(max)'), + database_name = DB_NAME(w.x.value('(data[@name="database_id"]/value)[1]', 'integer')), + database_id = w.x.value('(data[@name="database_id"]/value)[1]', 'integer'), xml = w.x.query('.') - INTO #health - FROM #sp_server_diagnostics_component_result AS wi - CROSS APPLY wi.sp_server_diagnostics_component_result.nodes('//event') AS w(x) - WHERE w.x.exist('(data[@name="component"]/text[.= "SYSTEM"])') = 1 - AND (w.x.exist('(data[@name="state"]/text[.= "WARNING"])') = @warnings_only OR @warnings_only = 0) + INTO #error_info + FROM #error_reported AS er + CROSS APPLY er.error_reported.nodes('//event') AS w(x) + WHERE w.x.exist('(data[@name="severity"]/value)[. >= 16]') = 1 + AND (@warnings_only = 0 OR w.x.exist('(data[@name="severity"]/value)[. >= 19]') = 1) + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #ignore_errors AS ie + WHERE w.x.value('(data[@name="error_number"]/value)[1]', 'integer') = ie.error_number + ) OPTION(RECOMPILE); - + IF @debug = 1 BEGIN SELECT TOP (100) - table_name = '#health, top 100 rows', + table_name = '#error_info, top 100 rows', x.* - FROM #health AS x + FROM #error_info AS x ORDER BY x.event_time DESC; END; - + + /* Severe errors reported logging section */ IF NOT EXISTS ( SELECT 1/0 - FROM #health AS h + FROM #error_info AS ei ) BEGIN - SELECT - finding = - CASE - WHEN @what_to_check NOT IN ('all', 'system') - THEN 'system health skipped, @what_to_check set to ' + - @what_to_check - WHEN @what_to_check IN ('all', 'system') - THEN 'no system health issues found between ' + - RTRIM(CONVERT(date, @start_date)) + - ' and ' + - RTRIM(CONVERT(date, @end_date)) + - ' with @warnings_only set to ' + - RTRIM(@warnings_only) + - '.' - ELSE 'no system health issues found!' - END; + IF @log_to_table = 0 + BEGIN + /* No results logic, only return if not logging */ + SELECT + finding = + CASE + WHEN @what_to_check NOT IN ('all', 'system') + THEN 'error reporting skipped, @what_to_check set to ' + + @what_to_check + WHEN @what_to_check IN ('all', 'system') + THEN 'no severe errors found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with @warnings_only set to ' + + RTRIM(@warnings_only) + + '.' + ELSE 'no severe errors found!' + END + + RAISERROR('No error data found', 0, 0) WITH NOWAIT; + END; END; ELSE BEGIN + /* Build the query */ + SET @dsql = N' SELECT - finding = 'overall system health', - h.event_time, - h.state, - h.spinlockBackoffs, - h.sickSpinlockType, - h.sickSpinlockTypeAfterAv, - h.latchWarnings, - h.isAccessViolationOccurred, - h.writeAccessViolationCount, - h.totalDumpRequests, - h.intervalDumpRequests, - h.nonYieldingTasksReported, - h.pageFaults, - h.systemCpuUtilization, - h.sqlCpuUtilization, - h.BadPagesDetected, - h.BadPagesFixed - FROM #health AS h + ' + CASE + WHEN @log_to_table = 1 + THEN N'' + ELSE N'finding = ''severe errors reported'',' + END + + N' + ei.event_time, + ei.error_number, + ei.severity, + ei.state, + ei.message, + ei.database_name, + ei.database_id + FROM #error_info AS ei'; + + /* Add the WHERE clause only for table logging */ + IF @log_to_table = 1 + BEGIN + /* Get max event_time for severe errors */ + SET @mdsql_execute = + REPLACE + ( + REPLACE + ( + @mdsql_template, + '{table_check}', + @log_table_severe_errors + ), + '{date_column}', + 'event_time' + ); + + IF @debug = 1 + BEGIN + PRINT @mdsql_execute; + END; + + EXECUTE sys.sp_executesql + @mdsql_execute, + N'@max_event_time datetime2(7) OUTPUT', + @max_event_time OUTPUT; + + SET @dsql = @dsql + N' + WHERE ei.event_time > @max_event_time'; + END; + + /* Add the ORDER BY clause */ + SET @dsql = @dsql + N' ORDER BY - h.event_time DESC + ei.event_time DESC, + ei.severity DESC OPTION(RECOMPILE); + '; + + /* Handle table logging */ + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + INSERT INTO + ' + @log_table_severe_errors + N' + ( + event_time, + error_number, + severity, + state, + message, + database_name, + database_id + )' + + @dsql; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7)', + @max_event_time; + END; + + /* Execute the query for client results */ + IF @log_to_table = 0 + BEGIN + IF @debug = 1 + BEGIN + PRINT @dsql; + END; + + EXECUTE sys.sp_executesql + @dsql; + END; + + /* For ignored errors, only display to client */ + IF @log_to_table = 0 + BEGIN + SELECT + error_numbers_ignored = + N'Error Number Ignored: ' + + CONVERT(nvarchar(100), ie.error_number) + FROM #ignore_errors AS ie; + END; END; - END; /*End system*/ + END; /*End error_reported analysis*/ /*Grab useless stuff*/ @@ -1994,13 +4292,22 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ( @what_to_check IN ('all', 'locking') AND @skip_locks = 0 + AND @log_to_table = 0 ) BEGIN IF @debug = 1 BEGIN - RAISERROR('Parsing locking stuff', 0, 1) WITH NOWAIT; + RAISERROR('Parsing locking stuff', 0, 0) WITH NOWAIT; END; + INSERT + #blocking_xml + WITH + (TABLOCK) + ( + event_time, + human_events_xml + ) SELECT event_time = DATEADD @@ -2015,7 +4322,6 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. w.x.value('(//@timestamp)[1]', 'datetime2') ), human_events_xml = w.x.query('//data[@name="data"]/value/queryProcessing/blockingTasks/blocked-process-report') - INTO #blocking_xml FROM #sp_server_diagnostics_component_result AS wi CROSS APPLY wi.sp_server_diagnostics_component_result.nodes('//event') AS w(x) WHERE w.x.exist('(data[@name="component"]/text[.= "QUERY_PROCESSING"])') = 1 @@ -2036,22 +4342,22 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. /*Blocked queries*/ IF @debug = 1 BEGIN - RAISERROR('Parsing blocked queries', 0, 1) WITH NOWAIT; + RAISERROR('Parsing blocked queries', 0, 0) WITH NOWAIT; END; SELECT bx.event_time, currentdbname = bd.value('(process/@currentdbname)[1]', 'nvarchar(128)'), - spid = bd.value('(process/@spid)[1]', 'int'), - ecid = bd.value('(process/@ecid)[1]', 'int'), - query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + spid = bd.value('(process/@spid)[1]', 'integer'), + ecid = bd.value('(process/@ecid)[1]', 'integer'), + query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bd.value('(process/@waittime)[1]', 'bigint'), lastbatchstarted = bd.value('(process/@lastbatchstarted)[1]', 'datetime2'), lastbatchcompleted = bd.value('(process/@lastbatchcompleted)[1]', 'datetime2'), wait_resource = bd.value('(process/@waitresource)[1]', 'nvarchar(100)'), status = bd.value('(process/@status)[1]', 'nvarchar(10)'), - priority = bd.value('(process/@priority)[1]', 'int'), - transaction_count = bd.value('(process/@trancount)[1]', 'int'), + priority = bd.value('(process/@priority)[1]', 'integer'), + transaction_count = bd.value('(process/@trancount)[1]', 'integer'), client_app = bd.value('(process/@clientapp)[1]', 'nvarchar(256)'), host_name = bd.value('(process/@hostname)[1]', 'nvarchar(256)'), login_name = bd.value('(process/@loginname)[1]', 'nvarchar(256)'), @@ -2070,7 +4376,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Adding query_text to #blocked', 0, 1) WITH NOWAIT; + RAISERROR('Adding query_text to #blocked', 0, 0) WITH NOWAIT; END; ALTER TABLE #blocked @@ -2097,22 +4403,22 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. /*Blocking queries*/ IF @debug = 1 BEGIN - RAISERROR('Parsing blocking queries', 0, 1) WITH NOWAIT; + RAISERROR('Parsing blocking queries', 0, 0) WITH NOWAIT; END; SELECT bx.event_time, currentdbname = bg.value('(process/@currentdbname)[1]', 'nvarchar(128)'), - spid = bg.value('(process/@spid)[1]', 'int'), - ecid = bg.value('(process/@ecid)[1]', 'int'), - query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + spid = bg.value('(process/@spid)[1]', 'integer'), + ecid = bg.value('(process/@ecid)[1]', 'integer'), + query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bg.value('(process/@waittime)[1]', 'bigint'), last_transaction_started = bg.value('(process/@lastbatchstarted)[1]', 'datetime2'), last_transaction_completed = bg.value('(process/@lastbatchcompleted)[1]', 'datetime2'), wait_resource = bg.value('(process/@waitresource)[1]', 'nvarchar(100)'), status = bg.value('(process/@status)[1]', 'nvarchar(10)'), - priority = bg.value('(process/@priority)[1]', 'int'), - transaction_count = bg.value('(process/@trancount)[1]', 'int'), + priority = bg.value('(process/@priority)[1]', 'integer'), + transaction_count = bg.value('(process/@trancount)[1]', 'integer'), client_app = bg.value('(process/@clientapp)[1]', 'nvarchar(256)'), host_name = bg.value('(process/@hostname)[1]', 'nvarchar(256)'), login_name = bg.value('(process/@loginname)[1]', 'nvarchar(256)'), @@ -2131,7 +4437,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Adding query_text to #blocking', 0, 1) WITH NOWAIT; + RAISERROR('Adding query_text to #blocking', 0, 0) WITH NOWAIT; END; ALTER TABLE #blocking @@ -2158,7 +4464,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. /*Put it together*/ IF @debug = 1 BEGIN - RAISERROR('Inserting to #blocks', 0, 1) WITH NOWAIT; + RAISERROR('Inserting to #blocks', 0, 0) WITH NOWAIT; END; SELECT @@ -2307,44 +4613,70 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ORDER BY x.event_time DESC; END; - - SELECT - finding = 'blocked process report', - b.event_time, - b.currentdbname, - b.activity, - b.spid, - b.ecid, - b.query_text, - b.wait_time_ms, - b.status, - b.isolation_level, - b.transaction_count, - b.last_transaction_started, - b.last_transaction_completed, - b.client_option_1, - b.client_option_2, - b.wait_resource, - b.priority, - b.log_used, - b.client_app, - b.host_name, - b.login_name, - b.blocked_process_report - FROM #blocks AS b - ORDER BY - b.event_time DESC, - CASE - WHEN b.activity = 'blocking' - THEN -1 - ELSE +1 - END - OPTION(RECOMPILE); + + IF EXISTS + ( + SELECT + 1/0 + FROM #blocks AS b + ) + BEGIN + SELECT + finding = 'blocked process report', + b.event_time, + b.currentdbname, + b.activity, + b.spid, + b.ecid, + b.query_text, + b.wait_time_ms, + b.status, + b.isolation_level, + b.transaction_count, + b.last_transaction_started, + b.last_transaction_completed, + b.client_option_1, + b.client_option_2, + b.wait_resource, + b.priority, + b.log_used, + b.client_app, + b.host_name, + b.login_name, + b.blocked_process_report + FROM #blocks AS b + ORDER BY + b.event_time DESC, + CASE + WHEN b.activity = 'blocking' + THEN -1 + ELSE +1 + END + OPTION(RECOMPILE); + END; + ELSE + BEGIN + SELECT + finding = CASE + WHEN @what_to_check NOT IN ('all', 'locking') + THEN 'blocking skipped, @what_to_check set to ' + @what_to_check + WHEN @skip_locks = 1 + THEN 'blocking skipped, @skip_locks set to 1' + WHEN @what_to_check IN ('all', 'locking') + THEN 'no blocking found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + + ' with @warnings_only set to ' + + RTRIM(@warnings_only) + ELSE 'no blocking found!' + END; + END; /*Grab available plans from the cache*/ IF @debug = 1 BEGIN - RAISERROR('Inserting to #available_plans (blocking)', 0, 1) WITH NOWAIT; + RAISERROR('Inserting to #available_plans (blocking)', 0, 0) WITH NOWAIT; END; SELECT DISTINCT @@ -2357,13 +4689,13 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 'available plans for blocking', b.currentdbname, query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), + TRY_CAST(b.query_text AS nvarchar(max)), sql_handle = CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), stmtstart = - ISNULL(n.c.value('@stmtstart', 'int'), 0), + ISNULL(n.c.value('@stmtstart', 'integer'), 0), stmtend = - ISNULL(n.c.value('@stmtend', 'int'), -1) + ISNULL(n.c.value('@stmtend', 'integer'), -1) FROM #blocks AS b CROSS APPLY b.blocked_process_report.nodes('/blocked-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) WHERE (b.currentdbname = @database_name @@ -2376,13 +4708,13 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. CONVERT(varchar(30), 'available plans for blocking'), b.currentdbname, query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), + TRY_CAST(b.query_text AS nvarchar(max)), sql_handle = CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), stmtstart = - ISNULL(n.c.value('@stmtstart', 'int'), 0), + ISNULL(n.c.value('@stmtstart', 'integer'), 0), stmtend = - ISNULL(n.c.value('@stmtend', 'int'), -1) + ISNULL(n.c.value('@stmtend', 'integer'), -1) FROM #blocks AS b CROSS APPLY b.blocked_process_report.nodes('/blocking-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) WHERE (b.currentdbname = @database_name @@ -2392,7 +4724,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Inserting to #deadlocks', 0, 1) WITH NOWAIT; + RAISERROR('Inserting to #deadlocks', 0, 0) WITH NOWAIT; END; SELECT @@ -2414,7 +4746,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Inserting to #deadlocks_parsed', 0, 1) WITH NOWAIT; + RAISERROR('Inserting to #deadlocks_parsed', 0, 0) WITH NOWAIT; END; SELECT @@ -2540,7 +4872,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Adding query_text to #deadlocks_parsed', 0, 1) WITH NOWAIT; + RAISERROR('Adding query_text to #deadlocks_parsed', 0, 0) WITH NOWAIT; END; ALTER TABLE #deadlocks_parsed @@ -2564,102 +4896,126 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Returning deadlocks', 0, 1) WITH NOWAIT; + RAISERROR('Returning deadlocks', 0, 0) WITH NOWAIT; END; - SELECT - finding = 'xml deadlock report', - dp.event_date, - is_victim = - CASE - WHEN dp.id = dp.victim_id - THEN 1 - ELSE 0 - END, - dp.database_name, - dp.current_database_name, - query_text = - CASE - WHEN dp.query_text - LIKE CONVERT(nvarchar(1), 0x0a00, 0) + N'Proc |[Database Id = %' ESCAPE N'|' - THEN - ( - SELECT - [processing-instruction(query)] = - OBJECT_SCHEMA_NAME - ( - SUBSTRING - ( - dp.query_text, - CHARINDEX(N'Object Id = ', dp.query_text) + 12, - LEN(dp.query_text) - (CHARINDEX(N'Object Id = ', dp.query_text) + 12) - ) - , - SUBSTRING - ( - dp.query_text, - CHARINDEX(N'Database Id = ', dp.query_text) + 14, - CHARINDEX(N'Object Id', dp.query_text) - (CHARINDEX(N'Database Id = ', dp.query_text) + 14) - ) - ) + - N'.' + - OBJECT_NAME - ( - SUBSTRING - ( - dp.query_text, - CHARINDEX(N'Object Id = ', dp.query_text) + 12, - LEN(dp.query_text) - (CHARINDEX(N'Object Id = ', dp.query_text) + 12) - ) - , - SUBSTRING - ( - dp.query_text, - CHARINDEX(N'Database Id = ', dp.query_text) + 14, - CHARINDEX(N'Object Id', dp.query_text) - (CHARINDEX(N'Database Id = ', dp.query_text) + 14) - ) - ) - FOR XML - PATH(N''), - TYPE - ) - ELSE - ( - SELECT - [processing-instruction(query)] = - dp.query_text - FOR XML - PATH(N''), - TYPE - ) - END, - dp.deadlock_resources, - dp.isolation_level, - dp.lock_mode, - dp.status, - dp.wait_time, - dp.log_used, - dp.transaction_name, - dp.transaction_count, - dp.client_option_1, - dp.client_option_2, - dp.last_tran_started, - dp.last_batch_started, - dp.last_batch_completed, - dp.client_app, - dp.host_name, - dp.login_name, - dp.priority, - dp.deadlock_graph - FROM #deadlocks_parsed AS dp - ORDER BY - dp.event_date, - is_victim - OPTION(RECOMPILE); + IF EXISTS + ( + SELECT + 1/0 + FROM #deadlocks_parsed AS dp + ) + BEGIN + SELECT + finding = 'xml deadlock report', + dp.event_date, + is_victim = + CASE + WHEN dp.id = dp.victim_id + THEN 1 + ELSE 0 + END, + dp.database_name, + dp.current_database_name, + query_text = + CASE + WHEN dp.query_text + LIKE CONVERT(nvarchar(1), 0x0a00, 0) + N'Proc |[Database Id = %' ESCAPE N'|' + THEN + ( + SELECT + [processing-instruction(query)] = + OBJECT_SCHEMA_NAME + ( + SUBSTRING + ( + dp.query_text, + CHARINDEX(N'Object Id = ', dp.query_text) + 12, + LEN(dp.query_text) - (CHARINDEX(N'Object Id = ', dp.query_text) + 12) + ) + , + SUBSTRING + ( + dp.query_text, + CHARINDEX(N'Database Id = ', dp.query_text) + 14, + CHARINDEX(N'Object Id', dp.query_text) - (CHARINDEX(N'Database Id = ', dp.query_text) + 14) + ) + ) + + N'.' + + OBJECT_NAME + ( + SUBSTRING + ( + dp.query_text, + CHARINDEX(N'Object Id = ', dp.query_text) + 12, + LEN(dp.query_text) - (CHARINDEX(N'Object Id = ', dp.query_text) + 12) + ) + , + SUBSTRING + ( + dp.query_text, + CHARINDEX(N'Database Id = ', dp.query_text) + 14, + CHARINDEX(N'Object Id', dp.query_text) - (CHARINDEX(N'Database Id = ', dp.query_text) + 14) + ) + ) + FOR XML + PATH(N''), + TYPE + ) + ELSE + ( + SELECT + [processing-instruction(query)] = + dp.query_text + FOR XML + PATH(N''), + TYPE + ) + END, + dp.deadlock_resources, + dp.isolation_level, + dp.lock_mode, + dp.status, + dp.wait_time, + dp.log_used, + dp.transaction_name, + dp.transaction_count, + dp.client_option_1, + dp.client_option_2, + dp.last_tran_started, + dp.last_batch_started, + dp.last_batch_completed, + dp.client_app, + dp.host_name, + dp.login_name, + dp.priority, + dp.deadlock_graph + FROM #deadlocks_parsed AS dp + ORDER BY + dp.event_date, + is_victim + OPTION(RECOMPILE); + END; + ELSE + BEGIN + SELECT + finding = CASE + WHEN @what_to_check NOT IN ('all', 'locking') + THEN 'deadlocks skipped, @what_to_check set to ' + @what_to_check + WHEN @skip_locks = 1 + THEN 'deadlocks skipped, @skip_locks set to 1' + WHEN @what_to_check IN ('all', 'locking') + THEN 'no deadlocks found between ' + + RTRIM(CONVERT(date, @start_date)) + + ' and ' + + RTRIM(CONVERT(date, @end_date)) + ELSE 'no deadlocks found!' + END; + END; IF @debug = 1 BEGIN - RAISERROR('Inserting #available_plans (deadlocks)', 0, 1) WITH NOWAIT; + RAISERROR('Inserting #available_plans (deadlocks)', 0, 0) WITH NOWAIT; END; INSERT @@ -2697,7 +5053,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Inserting #dm_exec_query_stats_sh', 0, 1) WITH NOWAIT; + RAISERROR('Inserting #dm_exec_query_stats_sh', 0, 0) WITH NOWAIT; END; SELECT @@ -2764,7 +5120,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Indexing #dm_exec_query_stats_sh', 0, 1) WITH NOWAIT; + RAISERROR('Indexing #dm_exec_query_stats_sh', 0, 0) WITH NOWAIT; END; CREATE CLUSTERED INDEX @@ -2777,7 +5133,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. IF @debug = 1 BEGIN - RAISERROR('Inserting #all_available_plans (deadlocks)', 0, 1) WITH NOWAIT; + RAISERROR('Inserting #all_available_plans (deadlocks)', 0, 0) WITH NOWAIT; END; SELECT @@ -2859,21 +5215,67 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ap.avg_worker_time_ms DESC OPTION(RECOMPILE); - SELECT - aap.* - FROM #all_avalable_plans AS aap - WHERE aap.finding = 'available plans for blocking' - ORDER BY - aap.avg_worker_time_ms DESC - OPTION(RECOMPILE); - - SELECT - aap.* - FROM #all_avalable_plans AS aap - WHERE aap.finding = 'available plans for deadlocks' - ORDER BY - aap.avg_worker_time_ms DESC - OPTION(RECOMPILE); + IF EXISTS + ( + SELECT + 1/0 + FROM #all_avalable_plans AS ap + WHERE ap.finding = 'available plans for blocking' + ) + BEGIN + SELECT + aap.* + FROM #all_avalable_plans AS aap + WHERE aap.finding = 'available plans for blocking' + ORDER BY + aap.avg_worker_time_ms DESC + OPTION(RECOMPILE); + END; + ELSE + BEGIN + /* Only show this message if we found blocking but no plans */ + IF EXISTS + ( + SELECT + 1/0 + FROM #blocks AS b + ) + BEGIN + SELECT + finding = 'no cached plans found for blocking queries'; + END; + END; + + IF EXISTS + ( + SELECT + 1/0 + FROM #all_avalable_plans AS ap + WHERE ap.finding = 'available plans for deadlocks' + ) + BEGIN + SELECT + aap.* + FROM #all_avalable_plans AS aap + WHERE aap.finding = 'available plans for deadlocks' + ORDER BY + aap.avg_worker_time_ms DESC + OPTION(RECOMPILE); + END; + ELSE + BEGIN + /* Only show this message if we found deadlocks but no plans */ + IF EXISTS + ( + SELECT + 1/0 + FROM #deadlocks_parsed AS dp + ) + BEGIN + SELECT + finding = 'no cached plans found for deadlock queries'; + END; + END; END; /*End locks*/ END; /*Final End*/ GO diff --git a/sp_HumanEvents/Examples.sql b/sp_HumanEvents/Examples.sql deleted file mode 100644 index 8e184637..00000000 --- a/sp_HumanEvents/Examples.sql +++ /dev/null @@ -1,165 +0,0 @@ -/* -Copyright 2025 Darling Data, LLC -https://www.erikdarling.com/ - -For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData - -MIT License - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), -to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, -sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the -following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE -FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -*/ - --- Here are some example calls to get you started. - --- To capture all types of “completed” queries that have run for at least one second, for 20 seconds, from a specific database - -EXECUTE dbo.sp_HumanEvents - @event_type = 'query', - @query_duration_ms = 1000, - @seconds_sample = 20, - @database_name = 'YourMom'; - --- Maybe you want to filter out queries that have asked for a bit of memory: - -EXECUTE dbo.sp_HumanEvents - @event_type = 'query', - @query_duration_ms = 1000, - @seconds_sample = 20, - @requested_memory_mb = 1024; - --- Or maybe you want to find unparameterized queries from a poorly written app that constructs strings in ugly ways, but it generates a lot of queries so you only want data on about a third of them. - -EXECUTE dbo.sp_HumanEvents - @event_type = 'compilations', - @client_app_name = N'GL00SNIFЯ', - @session_id = 'sample', - @sample_divisor = 3; - --- Perhaps you think queries recompiling are the cause of your problems! Heck, they might be. Have you tried removing recompile hints? 😁 - -EXECUTE dbo.sp_HumanEvents - @event_type = 'recompilations', - @seconds_sample = 30; - --- Look, blocking is annoying. Just turn on RCSI, you goblin. Unless you’re not allowed to. - -EXECUTE dbo.sp_HumanEvents - @event_type = 'blocking', - @seconds_sample = 60, - @blocking_duration_ms = 5000; - --- If you want to track wait stats, this’ll work pretty well. Keep in mind “all” is a focused list of “interesting” waits to queries, not every wait stat. - -EXECUTE dbo.sp_HumanEvents - @event_type = 'waits', - @wait_duration_ms = 10, - @seconds_sample = 100, - @wait_type = N'all'; - --- Note that THREADPOOL is SOS_WORKER in xe-land. why? I dunno. - -EXECUTE dbo.sp_HumanEvents - @event_type = 'waits', - @wait_duration_ms = 100, - @seconds_sample = 10, - @wait_type = N'SOS_WORKER,RESOURCE_SEMAPHORE'; - --- For some event types that allow you to set a minimum duration, I’ve set a default minimum to try to avoid you introducing a lot of observer overhead to the server. If you understand the potential danger here, or you’re just trying to test things, you need to use the @gimme_danger parameter. You would also use this if you wanted to set an impermanent session to run for longer than 10 minutes. - --- For example, if you run this command: - -EXECUTE sp_HumanEvents - @event_type = N'query', - @query_duration_ms = 1; - --- You’ll see this message in the output: - --- Checking query duration filter --- You chose a really dangerous value for @query_duration --- If you really want that, please set @gimme_danger = 1, and re-run --- Setting @query_duration to 500 - - --- You need to use this command instead: - -EXECUTE sp_HumanEvents - @event_type = N'query', - @query_duration_ms = 1, - @gimme_danger = 1; - --- Logging Data To Tables - --- First, you need to set up permanent sessions to collect data. You can use commands like these to do that, but I urge you to add some filters like above to cut down on the data collected. On busy servers, over-collection can cause performance issues. - - -EXECUTE sp_HumanEvents - @event_type = N'compiles', - @keep_alive = 1; - -EXECUTE sp_HumanEvents - @event_type = N'recompiles', - @keep_alive = 1; - - -EXECUTE sp_HumanEvents - @event_type = N'query', - @keep_alive = 1; - -EXECUTE sp_HumanEvents - @event_type = N'waits', - @keep_alive = 1; - -EXECUTE sp_HumanEvents - @event_type = N'blocking', - @keep_alive = 1; - - --- Once your sessions are set up, this is the command to tell sp_HumanEvents which database and schema to log data to. --- Table names are created internally, so don’t worry about those. - -EXECUTE sp_HumanEvents - @output_database_name = N'YourDatabase', - @output_schema_name = N'dbo'; - --- Ideally, you’ll stick this in an Agent Job, so you don’t need to rely on an SSMS window being open all the time. --- The job creation code linked is set to check in every 10 seconds, in case of errors. --- Internally, this will run in its own loop with a WAITFOR of 5 seconds to flush data out. - - --- Part of what gets installed when you log data to tables are some views in the same database. - --- You can check in on them like this: - -/*Queries*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_Queries; -/*Waits*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_WaitsByQueryAndDatabase; -SELECT TOP 1000 * FROM dbo.HumanEvents_WaitsByDatabase; -SELECT TOP 1000 * FROM dbo.HumanEvents_WaitsTotal; -/*Blocking*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_Blocking; -/*Compiles, only on newer versions of SQL Server*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_CompilesByDatabaseAndObject; -SELECT TOP 1000 * FROM dbo.HumanEvents_CompilesByQuery; -SELECT TOP 1000 * FROM dbo.HumanEvents_CompilesByDuration; -/*Otherwise*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_Compiles_Legacy; -/*Parameterization data, if available (comes along with compiles)*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_Parameterization; -/*Recompiles, only on newer versions of SQL Server*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_RecompilesByDatabaseAndObject; -SELECT TOP 1000 * FROM dbo.HumanEvents_RecompilesByQuery; -SELECT TOP 1000 * FROM dbo.HumanEvents_RecompilesByDuration; -/*Otherwise*/ -SELECT TOP 1000 * FROM dbo.HumanEvents_Recompiles_Legacy; diff --git a/sp_HumanEvents/README.md b/sp_HumanEvents/README.md new file mode 100644 index 00000000..27591218 --- /dev/null +++ b/sp_HumanEvents/README.md @@ -0,0 +1,222 @@ + + +# Human Events Toolkit + +This directory contains two stored procedures for managing and analyzing Extended Events in SQL Server: + +- **[sp_HumanEvents](#sp_humanevents)**: Makes extended events easy to use for common scenarios +- **[sp_HumanEventsBlockViewer](#sp_humaneventsblockviewer)**: Analyzes blocked process reports + +## Table of Contents + +- [sp_HumanEvents](#sp_humanevents) + - [Overview](#overview) + - [Warning](#warning) + - [Parameters](#parameters) + - [Usage Examples](#usage-examples) + - [Resources](#resources) +- [sp_HumanEventsBlockViewer](#sp_humaneventsblockviewer) + - [Overview](#overview-1) + - [Setup](#setup) + - [Parameters](#parameters-1) + - [Usage Examples](#usage-examples-1) + +--- + +# sp_HumanEvents + +## Overview + +Extended Events are hard. You don't know which ones to use, when to use them, or how to get useful information out of them. + +This procedure is designed to make them easier for you, by creating event sessions to help you troubleshoot common scenarios: +* Blocking: blocked process report +* Query performance: query execution metrics an actual execution plans +* Compiles: catch query compilations +* Recompiles: catch query recompilations +* Wait Stats: server wait stats, broken down by query and database + +The default behavior is to run a session for a set period of time to capture information, but you can also set sessions up to data to permanent tables. + +## Warning + +Misuse of this procedure can harm performance. Be very careful about introducing observer overhead, especially when gathering query plans. Be even more careful when setting up permanent sessions! + +## Parameters + +| parameter | data_type | description | valid_inputs | defaults | +|------------------------|----------------|----------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| +| @event_type | sysname | used to pick which session you want to run | "blocking", "query", "waits", "recompiles", "compiles" and certain variations on those words | "query" | +| @query_duration_ms | integer | (>=) used to set a minimum query duration to collect data for | an integer | 500 (ms) | +| @query_sort_order | nvarchar | when you use the "query" event, lets you choose which metrics to sort results by | "cpu", "reads", "writes", "duration", "memory", "spills", and you can add "avg" to sort by averages, e.g. "avg cpu" | "cpu" | +| @skip_plans | bit | when you use the "query" event, lets you skip collecting actual execution plans |1 or 0 | 0 | +| @blocking_duration_ms | integer | (>=) used to set a minimum blocking duration to collect data for | an integer | 500 (ms) | +| @wait_type | nvarchar | (inclusive) filter to only specific wait types | a single wait type, or a CSV list of wait types | "all", which uses a list of "interesting" waits | +| @wait_duration_ms | integer | (>=) used to set a minimum time per wait to collect data for | an integer | 10 (ms) | +| @client_app_name | sysname | (inclusive) filter to only specific app names | a stringy thing | intentionally left blank | +| @client_hostname | sysname | (inclusive) filter to only specific host names | a stringy thing | intentionally left blank | +| @database_name | sysname | (inclusive) filter to only specific databases | a stringy thing | intentionally left blank | +| @session_id | nvarchar | (inclusive) filter to only a specific session id, or a sample of session ids | an integer, or "sample" to sample a workload | intentionally left blank | +| @sample_divisor | integer | the divisor for session ids when sampling a workload, e.g. SPID % 5 | an integer | 5 | +| @username | sysname | (inclusive) filter to only a specific user | a stringy thing | intentionally left blank | +| @object_name | sysname | (inclusive) to only filter to a specific object name | a stringy thing | intentionally left blank | +| @object_schema | sysname | (inclusive) the schema of the object you want to filter to; only needed with blocking events | a stringy thing | dbo | +| @requested_memory_mb | integer | (>=) the memory grant a query must ask for to have data collected | an integer | 0 | +| @seconds_sample | tinyint | the duration in seconds to run the event session for | an integer | 10 | +| @gimme_danger | bit | used to override default minimums for query, wait, and blocking durations. | 1 or 0 | 0 | +| @keep_alive | bit | creates a permanent session, either to watch live or log to a table from | 1 or 0 | 0 | +| @custom_name | sysname | if you want to custom name a permanent session | a stringy thing | intentionally left blank | +| @output_database_name | sysname | the database you want to log data to | a valid database name | intentionally left blank | +| @output_schema_name | sysname | the schema you want to log data to | a valid schema | dbo | +| @delete_retention_days | integer | how many days of logged data you want to keep | a POSITIVE integer | 3 (days) | +| @cleanup | bit | deletes all sessions, tables, and views. requires output database and schema. | 1 or 0 | 0 | +| @max_memory_kb | bigint | set a max ring buffer size to log data to | an integer | 102400 | +| @version | varchar | to make sure you have the most recent bits | none, output | none, output | +| @version_date | datetime | to make sure you have the most recent bits | none, output | none, output | +| @debug | bit | use to print out dynamic SQL | 1 or 0 | 0 | +| @help | bit | well you're here so you figured this one out | 1 or 0 | 0 | + +## Usage Examples + +If you set up sessions to capture long term data, you'll need an agent job set up to poll them. You can find an example of that here: [sp_Human Events Agent Job Example.sql](sp_Human%20Events%20Agent%20Job%20Example.sql) + +Here are some basic usage examples: + +```sql +-- Basic execution to capture queries +EXECUTE dbo.sp_HumanEvents; + +-- Capture blocking events for at least 1 second +EXECUTE dbo.sp_HumanEvents + @event_type = 'blocking', + @blocking_duration_ms = 1000; + +-- Capture waits in a specific database +EXECUTE dbo.sp_HumanEvents + @event_type = 'waits', + @database_name = 'YourDatabase'; + +-- Set up a permanent session for logging +EXECUTE dbo.sp_HumanEvents + @event_type = 'query', + @keep_alive = 1, + @output_database_name = 'DBA', + @output_schema_name = 'dbo'; + +-- Clean up all sessions and tables +EXECUTE dbo.sp_HumanEvents + @cleanup = 1, + @output_database_name = 'DBA', + @output_schema_name = 'dbo'; +``` + +## Resources +* [YouTube playlist](https://www.youtube.com/playlist?list=PLt4QZ-7lfQifgpvqsa21WLt-u2tZlyoC_) +* [Blog post](https://www.erikdarlingdata.com/sp_humanevents/) + +--- + +# sp_HumanEventsBlockViewer + +## Overview + +This was originally a companion script to analyze the blocked process report Extended Event created by sp_HumanEvents, but has since turned into its own monster. + +It will work on any Extended Event that captures the blocked process report. If you need to set that up, run these two pieces of code. + +## Setup + +Enable the blocked process report: +```sql +EXECUTE sys.sp_configure + N'show advanced options', + 1; +RECONFIGURE; +GO + +EXECUTE sys.sp_configure + N'blocked process threshold', + 5; --Seconds +RECONFIGURE; +GO +``` + +Set up the Extended Event: +```sql +CREATE EVENT SESSION + blocked_process_report +ON SERVER + ADD EVENT + sqlserver.blocked_process_report + ADD TARGET + package0.event_file + ( + SET filename = N'bpr' + ) +WITH +( + MAX_MEMORY = 4096KB, + EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS, + MAX_DISPATCH_LATENCY = 5 SECONDS, + MAX_EVENT_SIZE = 0KB, + MEMORY_PARTITION_MODE = NONE, + TRACK_CAUSALITY = OFF, + STARTUP_STATE = ON +); + +ALTER EVENT SESSION + blocked_process_report +ON SERVER + STATE = START; +``` + +## Parameters + +| parameter_name | data_type | description | valid_inputs | defaults | +|-----------------------|-----------|-------------------------------------------------|------------------------------------------------------------------------|------------------------------------| +| @session_name | sysname | name of the extended event session to pull from | extended event session name capturing sqlserver.blocked_process_report | keeper_HumanEvents_blocking | +| @target_type | sysname | target of the extended event session | event_file or ring_buffer | NULL | +| @start_date | datetime2 | filter by date | a reasonable date | NULL; will shortcut to last 7 days | +| @end_date | datetime2 | filter by date | a reasonable date | NULL | +| @database_name | sysname | filter by database name | a database that exists on this server | NULL | +| @object_name | sysname | filter by table name | a schema-prefixed table name | NULL | +| @target_database | sysname | database containing the table with BPR data | a valid database name | NULL | +| @target_schema | sysname | schema of the table | a valid schema name | NULL | +| @target_table | sysname | table name | a valid table name | NULL | +| @target_column | sysname | column containing XML data | a valid column name | NULL | +| @timestamp_column | sysname | column containing timestamp (optional) | a valid column name | NULL | +| @log_to_table | bit | enable logging to permanent tables | 0 or 1 | 0 | +| @log_database_name | sysname | database to store logging tables | a valid database name | NULL | +| @log_schema_name | sysname | schema to store logging tables | a valid schema name | NULL | +| @log_table_name_prefix| sysname | prefix for all logging tables | a valid table name prefix | 'HumanEventsBlockViewer' | +| @log_retention_days | integer | Number of days to keep logs, 0 = keep indefinitely | a valid integer | 30 | +| @help | bit | how you got here | 0 or 1 | 0 | +| @debug | bit | dumps raw temp table contents | 0 or 1 | 0 | +| @version | varchar | OUTPUT; for support | none; OUTPUT | none; OUTPUT | +| @version_date | datetime | OUTPUT; for support | none; OUTPUT | none; OUTPUT | + +## Usage Examples + +```sql +-- Basic usage with default session name +EXECUTE dbo.sp_HumanEventsBlockViewer; + +-- Use with a custom extended event session name +EXECUTE dbo.sp_HumanEventsBlockViewer + @session_name = N'blocked_process_report'; + +-- Filter by a specific database +EXECUTE dbo.sp_HumanEventsBlockViewer + @database_name = 'YourDatabase'; + +-- Analyze blocking events for a specific time period +EXECUTE dbo.sp_HumanEventsBlockViewer + @start_date = '2025-01-01 08:00', + @end_date = '2025-01-01 17:00'; + +-- Log results to permanent tables +EXECUTE dbo.sp_HumanEventsBlockViewer + @log_to_table = 1, + @log_database_name = 'DBA', + @log_schema_name = 'dbo'; +``` \ No newline at end of file diff --git a/sp_HumanEvents/sp_Human Events Agent Job Example.sql b/sp_HumanEvents/sp_Human Events Agent Job Example.sql index d3ea4070..a811dc50 100644 --- a/sp_HumanEvents/sp_Human Events Agent Job Example.sql +++ b/sp_HumanEvents/sp_Human Events Agent Job Example.sql @@ -11,7 +11,7 @@ Copyright 2025 Darling Data, LLC https://www.erikdarling.com/ For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com MIT License diff --git a/sp_HumanEvents/sp_HumanEvents.sql b/sp_HumanEvents/sp_HumanEvents.sql index c793cd5c..2eeddbeb 100644 --- a/sp_HumanEvents/sp_HumanEvents.sql +++ b/sp_HumanEvents/sp_HumanEvents.sql @@ -36,7 +36,7 @@ EXECUTE sp_HumanEvents @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ @@ -87,8 +87,8 @@ SET XACT_ABORT ON; SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT - @version = '6.1', - @version_date = '20250101'; + @version = '6.4', + @version_date = '20250401'; IF @help = 1 BEGIN @@ -404,14 +404,14 @@ CREATE TABLE ( id integer PRIMARY KEY IDENTITY, view_name sysname NOT NULL, - view_definition varbinary(MAX) NOT NULL, + view_definition varbinary(max) NOT NULL, output_database sysname NOT NULL DEFAULT N'', output_schema sysname NOT NULL DEFAULT N'', output_table sysname NOT NULL DEFAULT N'', view_converted AS CONVERT ( - nvarchar(MAX), + nvarchar(max), view_definition ), view_converted_length AS @@ -419,7 +419,7 @@ CREATE TABLE ( CONVERT ( - nvarchar(MAX), + nvarchar(max), view_definition ) ) @@ -454,7 +454,7 @@ DECLARE CASE WHEN CONVERT ( - int, + integer, SERVERPROPERTY('EngineEdition') ) = 5 THEN 1 @@ -463,57 +463,57 @@ DECLARE @drop_old_sql nvarchar(1000) = N'', @waitfor nvarchar(20) = N'', @session_name nvarchar(512) = N'', - @session_with nvarchar(MAX) = N'', - @session_sql nvarchar(MAX) = N'', - @start_sql nvarchar(MAX) = N'', - @stop_sql nvarchar(MAX) = N'', - @drop_sql nvarchar(MAX) = N'', - @session_filter nvarchar(MAX) = N'', - @session_filter_limited nvarchar(MAX) = N'', - @session_filter_query_plans nvarchar(MAX) = N'', - @session_filter_waits nvarchar(MAX) = N'', - @session_filter_recompile nvarchar(MAX)= N'', - @session_filter_statement_completed nvarchar(MAX) = N'', - @session_filter_blocking nvarchar(MAX) = N'', - @session_filter_parameterization nvarchar(MAX) = N'', - @query_duration_filter nvarchar(MAX) = N'', - @blocking_duration_ms_filter nvarchar(MAX) = N'', - @wait_type_filter nvarchar(MAX) = N'', - @wait_duration_filter nvarchar(MAX) = N'', - @client_app_name_filter nvarchar(MAX) = N'', - @client_hostname_filter nvarchar(MAX) = N'', - @database_name_filter nvarchar(MAX) = N'', - @session_id_filter nvarchar(MAX) = N'', - @username_filter nvarchar(MAX) = N'', - @object_name_filter nvarchar(MAX) = N'', - @requested_memory_mb_filter nvarchar(MAX) = N'', + @session_with nvarchar(max) = N'', + @session_sql nvarchar(max) = N'', + @start_sql nvarchar(max) = N'', + @stop_sql nvarchar(max) = N'', + @drop_sql nvarchar(max) = N'', + @session_filter nvarchar(max) = N'', + @session_filter_limited nvarchar(max) = N'', + @session_filter_query_plans nvarchar(max) = N'', + @session_filter_waits nvarchar(max) = N'', + @session_filter_recompile nvarchar(max)= N'', + @session_filter_statement_completed nvarchar(max) = N'', + @session_filter_blocking nvarchar(max) = N'', + @session_filter_parameterization nvarchar(max) = N'', + @query_duration_filter nvarchar(max) = N'', + @blocking_duration_ms_filter nvarchar(max) = N'', + @wait_type_filter nvarchar(max) = N'', + @wait_duration_filter nvarchar(max) = N'', + @client_app_name_filter nvarchar(max) = N'', + @client_hostname_filter nvarchar(max) = N'', + @database_name_filter nvarchar(max) = N'', + @session_id_filter nvarchar(max) = N'', + @username_filter nvarchar(max) = N'', + @object_name_filter nvarchar(max) = N'', + @requested_memory_mb_filter nvarchar(max) = N'', @compile_events bit = 0, @parameterization_events bit = 0, @fully_formed_babby nvarchar(1000) = N'', - @s_out int, - @s_sql nvarchar(MAX) = N'', - @s_params nvarchar(MAX) = N'', + @s_out integer, + @s_sql nvarchar(max) = N'', + @s_params nvarchar(max) = N'', @object_id sysname = N'', @requested_memory_kb nvarchar(11) = N'', - @the_sleeper_must_awaken nvarchar(MAX) = N'', - @min_id int, - @max_id int, + @the_sleeper_must_awaken nvarchar(max) = N'', + @min_id integer, + @max_id integer, @event_type_check sysname, @object_name_check nvarchar(1000) = N'', - @table_sql nvarchar(MAX) = N'', + @table_sql nvarchar(max) = N'', @view_tracker bit, - @spe nvarchar(MAX) = N'.sys.sp_executesql ', - @view_sql nvarchar(MAX) = N'', + @spe nvarchar(max) = N'.sys.sp_executesql ', + @view_sql nvarchar(max) = N'', @view_database sysname = N'', @date_filter datetime, @Time time, - @delete_tracker int, - @the_deleter_must_awaken nvarchar(MAX) = N'', - @executer nvarchar(MAX), - @cleanup_sessions nvarchar(MAX) = N'', - @cleanup_tables nvarchar(MAX) = N'', - @drop_holder nvarchar(MAX) = N'', - @cleanup_views nvarchar(MAX) = N'', + @delete_tracker integer, + @the_deleter_must_awaken nvarchar(max) = N'', + @executer nvarchar(max), + @cleanup_sessions nvarchar(max) = N'', + @cleanup_tables nvarchar(max) = N'', + @drop_holder nvarchar(max) = N'', + @cleanup_views nvarchar(max) = N'', @nc10 nvarchar(2) = NCHAR(10), @inputbuf_bom nvarchar(1) = CONVERT(nvarchar(1), 0x0a00, 0); @@ -1026,7 +1026,7 @@ AND EXISTS 1/0 FROM sys.configurations AS c WHERE c.name = N'blocked process threshold (s)' - AND CONVERT(int, c.value_in_use) = 0 + AND CONVERT(integer, c.value_in_use) = 0 ) BEGIN RAISERROR(N'You need to set up the blocked process report in order to use this: @@ -1803,8 +1803,8 @@ BEGIN event_type = oa.c.value('@name', 'sysname'), database_name = oa.c.value('(action[@name="database_name"]/value/text())[1]', 'sysname'), object_name = oa.c.value('(data[@name="object_name"]/value/text())[1]', 'sysname'), - sql_text = oa.c.value('(action[@name="sql_text"]/value/text())[1]', 'nvarchar(MAX)'), - statement = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(MAX)'), + sql_text = oa.c.value('(action[@name="sql_text"]/value/text())[1]', 'nvarchar(max)'), + statement = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(max)'), showplan_xml = CASE WHEN @skip_plans = 0 THEN oa.c.query('(data[@name="showplan_xml"]/value/*)[1]') ELSE N'Skipped Plans' END, cpu_ms = oa.c.value('(data[@name="cpu_time"]/value/text())[1]', 'bigint') / 1000., logical_reads = (oa.c.value('(data[@name="logical_reads"]/value/text())[1]', 'bigint') * 8) / 1024., @@ -2155,7 +2155,7 @@ BEGIN event_type = oa.c.value('@name', 'sysname'), database_name = oa.c.value('(action[@name="database_name"]/value/text())[1]', 'sysname'), object_name = oa.c.value('(data[@name="object_name"]/value/text())[1]', 'sysname'), - statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(MAX)'), + statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(max)'), compile_cpu_ms = oa.c.value('(data[@name="cpu_time"]/value/text())[1]', 'bigint'), compile_duration_ms = oa.c.value('(data[@name="duration"]/value/text())[1]', 'bigint') INTO #compiles_1 @@ -2244,7 +2244,7 @@ BEGIN event_type = oa.c.value('@name', 'sysname'), database_name = oa.c.value('(action[@name="database_name"]/value/text())[1]', 'sysname'), object_name = oa.c.value('(data[@name="object_name"]/value/text())[1]', 'sysname'), - statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(MAX)') + statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(max)') INTO #compiles_0 FROM #human_events_xml AS xet OUTER APPLY xet.human_events_xml.nodes('//event') AS oa(c) @@ -2296,7 +2296,7 @@ BEGIN ), event_type = oa.c.value('@name', 'sysname'), database_name = oa.c.value('(action[@name="database_name"]/value/text())[1]', 'sysname'), - sql_text = oa.c.value('(action[@name="sql_text"]/value/text())[1]', 'nvarchar(MAX)'), + sql_text = oa.c.value('(action[@name="sql_text"]/value/text())[1]', 'nvarchar(max)'), compile_cpu_time_ms = oa.c.value('(data[@name="compile_cpu_time"]/value/text())[1]', 'bigint') / 1000., compile_duration_ms = oa.c.value('(data[@name="compile_duration"]/value/text())[1]', 'bigint') / 1000., query_param_type = oa.c.value('(data[@name="query_param_type"]/value/text())[1]', 'integer'), @@ -2408,7 +2408,7 @@ IF @compile_events = 1 database_name = oa.c.value('(action[@name="database_name"]/value/text())[1]', 'sysname'), object_name = oa.c.value('(data[@name="object_name"]/value/text())[1]', 'sysname'), recompile_cause = oa.c.value('(data[@name="recompile_cause"]/text)[1]', 'sysname'), - statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(MAX)'), + statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(max)'), recompile_cpu_ms = oa.c.value('(data[@name="cpu_time"]/value/text())[1]', 'bigint'), recompile_duration_ms = oa.c.value('(data[@name="duration"]/value/text())[1]', 'bigint') INTO #recompiles_1 @@ -2509,7 +2509,7 @@ IF @compile_events = 1 database_name = oa.c.value('(action[@name="database_name"]/value/text())[1]', 'sysname'), object_name = oa.c.value('(data[@name="object_name"]/value/text())[1]', 'sysname'), recompile_cause = oa.c.value('(data[@name="recompile_cause"]/text)[1]', 'sysname'), - statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(MAX)') + statement_text = oa.c.value('(data[@name="statement"]/value/text())[1]', 'nvarchar(max)') INTO #recompiles_0 FROM #human_events_xml AS xet OUTER APPLY xet.human_events_xml.nodes('//event') AS oa(c) @@ -2723,7 +2723,7 @@ BEGIN blocking_ecid = bg.value('(process/@ecid)[1]', 'integer'), blocked_spid = bd.value('(process/@spid)[1]', 'integer'), blocked_ecid = bd.value('(process/@ecid)[1]', 'integer'), - query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bd.value('(process/@waittime)[1]', 'bigint'), transaction_name = bd.value('(process/@transactionname)[1]', 'sysname'), last_transaction_started = bd.value('(process/@lasttranstarted)[1]', 'datetime2'), @@ -2818,7 +2818,7 @@ BEGIN blocking_ecid = bg.value('(process/@ecid)[1]', 'integer'), blocked_spid = bd.value('(process/@spid)[1]', 'integer'), blocked_ecid = bd.value('(process/@ecid)[1]', 'integer'), - query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bg.value('(process/@waittime)[1]', 'bigint'), transaction_name = bg.value('(process/@transactionname)[1]', 'sysname'), last_transaction_started = bg.value('(process/@lastbatchstarted)[1]', 'datetime2'), @@ -3217,7 +3217,7 @@ BEGIN b.currentdbid, b.contentious_object, query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), + TRY_CAST(b.query_text AS nvarchar(max)), sql_handle = CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), stmtstart = @@ -3242,7 +3242,7 @@ BEGIN b.currentdbid, b.contentious_object, query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), + TRY_CAST(b.query_text AS nvarchar(max)), sql_handle = CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), stmtstart = @@ -3694,14 +3694,14 @@ BEGIN THEN N'CREATE TABLE ' + @object_name_check + @nc10 + N'( id bigint PRIMARY KEY IDENTITY, server_name sysname NULL, event_time datetime2 NULL, ' + @nc10 + N' activity nvarchar(20) NULL, database_name sysname NULL, database_id integer NULL, object_id bigint NULL, contentious_object AS OBJECT_NAME(object_id, database_id), ' + @nc10 + - N' transaction_id bigint NULL, resource_owner_type sysname NULL, monitor_loop integer NULL, spid integer NULL, ecid integer NULL, query_text nvarchar(MAX) NULL, ' + + N' transaction_id bigint NULL, resource_owner_type sysname NULL, monitor_loop integer NULL, spid integer NULL, ecid integer NULL, query_text nvarchar(max) NULL, ' + N' wait_time bigint NULL, transaction_name sysname NULL, last_transaction_started nvarchar(30) NULL, wait_resource nvarchar(100) NULL, ' + @nc10 + N' lock_mode nvarchar(10) NULL, status nvarchar(10) NULL, priority integer NULL, transaction_count integer NULL, ' + @nc10 + N' client_app sysname NULL, host_name sysname NULL, login_name sysname NULL, isolation_level nvarchar(30) NULL, sql_handle varbinary(64) NULL, blocked_process_report XML NULL );' WHEN @event_type_check LIKE N'%quer%' THEN N'CREATE TABLE ' + @object_name_check + @nc10 + N'( id bigint PRIMARY KEY IDENTITY, server_name sysname NULL, event_time datetime2 NULL, event_type sysname NULL, ' + @nc10 + - N' database_name sysname NULL, object_name nvarchar(512) NULL, sql_text nvarchar(MAX) NULL, statement nvarchar(MAX) NULL, ' + @nc10 + + N' database_name sysname NULL, object_name nvarchar(512) NULL, sql_text nvarchar(max) NULL, statement nvarchar(max) NULL, ' + @nc10 + N' showplan_xml XML NULL, cpu_ms decimal(18,2) NULL, logical_reads decimal(18,2) NULL, ' + @nc10 + N' physical_reads decimal(18,2) NULL, duration_ms decimal(18,2) NULL, writes_mb decimal(18,2) NULL,' + @nc10 + N' spills_mb decimal(18,2) NULL, row_count decimal(18,2) NULL, estimated_rows decimal(18,2) NULL, dop integer NULL, ' + @nc10 + @@ -3710,19 +3710,19 @@ BEGIN WHEN @event_type_check LIKE N'%recomp%' THEN N'CREATE TABLE ' + @object_name_check + @nc10 + N'( id bigint PRIMARY KEY IDENTITY, server_name sysname NULL, event_time datetime2 NULL, event_type sysname NULL, ' + @nc10 + - N' database_name sysname NULL, object_name nvarchar(512) NULL, recompile_cause sysname NULL, statement_text nvarchar(MAX) NULL, statement_text_checksum AS CHECKSUM(database_name + statement_text) PERSISTED ' + N' database_name sysname NULL, object_name nvarchar(512) NULL, recompile_cause sysname NULL, statement_text nvarchar(max) NULL, statement_text_checksum AS CHECKSUM(database_name + statement_text) PERSISTED ' + CASE WHEN @compile_events = 1 THEN N', compile_cpu_ms bigint NULL, compile_duration_ms bigint NULL );' ELSE N' );' END WHEN @event_type_check LIKE N'%comp%' AND @event_type_check NOT LIKE N'%re%' THEN N'CREATE TABLE ' + @object_name_check + @nc10 + N'( id bigint PRIMARY KEY IDENTITY, server_name sysname NULL, event_time datetime2 NULL, event_type sysname NULL, ' + @nc10 + - N' database_name sysname NULL, object_name nvarchar(512) NULL, statement_text nvarchar(MAX) NULL, statement_text_checksum AS CHECKSUM(database_name + statement_text) PERSISTED ' + N' database_name sysname NULL, object_name nvarchar(512) NULL, statement_text nvarchar(max) NULL, statement_text_checksum AS CHECKSUM(database_name + statement_text) PERSISTED ' + CASE WHEN @compile_events = 1 THEN N', compile_cpu_ms bigint NULL, compile_duration_ms bigint NULL );' ELSE N' );' END + CASE WHEN @parameterization_events = 1 THEN @nc10 + N'CREATE TABLE ' + @object_name_check + N'_parameterization' + @nc10 + N'( id bigint PRIMARY KEY IDENTITY, server_name sysname NULL, event_time datetime2 NULL, event_type sysname NULL, ' + @nc10 + - N' database_name sysname NULL, sql_text nvarchar(MAX) NULL, compile_cpu_time_ms bigint NULL, compile_duration_ms bigint NULL, query_param_type integer NULL, ' + @nc10 + + N' database_name sysname NULL, sql_text nvarchar(max) NULL, compile_cpu_time_ms bigint NULL, compile_duration_ms bigint NULL, query_param_type integer NULL, ' + @nc10 + N' is_cached bit NULL, is_recompiled bit NULL, compile_code sysname NULL, has_literals bit NULL, is_parameterizable bit NULL, parameterized_values_count bigint NULL, ' + @nc10 + N' query_plan_hash binary(8) NULL, query_hash binary(8) NULL, plan_handle varbinary(64) NULL, statement_sql_hash varbinary(64) NULL );' ELSE N'' @@ -4048,12 +4048,12 @@ END; SELECT @table_sql = CONVERT ( - nvarchar(MAX), + nvarchar(max), CASE WHEN @event_type_check LIKE N'%wait%' /*Wait stats!*/ THEN CONVERT ( - nvarchar(MAX), + nvarchar(max), N'INSERT INTO ' + @object_name_check + N' WITH(TABLOCK) ' + @nc10 + N'( server_name, event_time, event_type, database_name, wait_type, duration_ms, ' + @nc10 + N' signal_duration_ms, wait_resource, query_plan_hash_signed, query_hash_signed, plan_handle )' + @nc10 + @@ -4078,13 +4078,13 @@ END; signal_duration_ms = c.value(''(data[@name="signal_duration"]/value/text())[1]'', ''bigint''),' + @nc10 + CONVERT ( - nvarchar(MAX), + nvarchar(max), CASE WHEN @v = 11 /*We can't get the wait resource on older versions of SQL Server*/ THEN N' ''Not Available < 2014'', ' + @nc10 ELSE N' wait_resource = c.value(''(data[@name="wait_resource"]/value/text())[1]'', ''sysname''), ' + @nc10 END -) + CONVERT(nvarchar(MAX), N' query_plan_hash_signed = +) + CONVERT(nvarchar(max), N' query_plan_hash_signed = CONVERT ( binary(8), @@ -4107,13 +4107,13 @@ AND c.exist(''@timestamp[. > sql:variable("@date_filter")]'') = 1;') /*Any existing blocking scenarios will update the blocking duration*/ THEN CONVERT ( - nvarchar(MAX), + nvarchar(max), N'INSERT INTO ' + @object_name_check + N' WITH(TABLOCK) ' + @nc10 + N'( server_name, event_time, activity, database_name, database_id, object_id, ' + @nc10 + N' transaction_id, resource_owner_type, monitor_loop, spid, ecid, query_text, wait_time, ' + @nc10 + N' transaction_name, last_transaction_started, wait_resource, lock_mode, status, priority, ' + @nc10 + N' transaction_count, client_app, host_name, login_name, isolation_level, sql_handle, blocked_process_report )' + @nc10 + -CONVERT(nvarchar(MAX), N' +CONVERT(nvarchar(max), N' SELECT server_name, event_time, activity, database_name, database_id, object_id, transaction_id, resource_owner_type, monitor_loop, spid, ecid, text, waittime, transactionname, lasttranstarted, wait_resource, lockmode, status, priority, @@ -4161,7 +4161,7 @@ FROM monitor_loop = oa.c.value(''(//@monitorLoop)[1]'', ''integer''), spid = bd.value(''(process/@spid)[1]'', ''integer''), ecid = bd.value(''(process/@ecid)[1]'', ''integer''), - text = bd.value(''(process/inputbuf/text())[1]'', ''nvarchar(MAX)''), + text = bd.value(''(process/inputbuf/text())[1]'', ''nvarchar(max)''), waittime = bd.value(''(process/@waittime)[1]'', ''bigint''), transactionname = bd.value(''(process/@transactionname)[1]'', ''sysname''), lasttranstarted = bd.value(''(process/@lasttranstarted)[1]'', ''datetime2''), @@ -4211,7 +4211,7 @@ FROM monitor_loop = oa.c.value(''(//@monitorLoop)[1]'', ''integer''), spid = bg.value(''(process/@spid)[1]'', ''integer''), ecid = bg.value(''(process/@ecid)[1]'', ''integer''), - text = bg.value(''(process/inputbuf/text())[1]'', ''nvarchar(MAX)''), + text = bg.value(''(process/inputbuf/text())[1]'', ''nvarchar(max)''), waittime = NULL, transactionname = NULL, lasttranstarted = NULL, @@ -4285,14 +4285,14 @@ JOIN THEN CONVERT ( - nvarchar(MAX), + nvarchar(max), N'INSERT INTO ' + @object_name_check + N' WITH(TABLOCK) ' + @nc10 + N'( server_name, event_time, event_type, database_name, object_name, sql_text, statement, ' + @nc10 + N' showplan_xml, cpu_ms, logical_reads, physical_reads, duration_ms, writes_mb, ' + @nc10 + N' spills_mb, row_count, estimated_rows, dop, serial_ideal_memory_mb, ' + @nc10 + N' requested_memory_mb, used_memory_mb, ideal_memory_mb, granted_memory_mb, ' + @nc10 + N' query_plan_hash_signed, query_hash_signed, plan_handle )' + @nc10 + - CONVERT(nvarchar(MAX), N'SELECT + CONVERT(nvarchar(max), N'SELECT server_name = @@SERVERNAME, event_time = DATEADD @@ -4309,8 +4309,8 @@ JOIN event_type = oa.c.value(''@name'', ''sysname''), database_name = oa.c.value(''(action[@name="database_name"]/value/text())[1]'', ''sysname''), [object_name] = oa.c.value(''(data[@name="object_name"]/value/text())[1]'', ''sysname''), - sql_text = oa.c.value(''(action[@name="sql_text"]/value/text())[1]'', ''nvarchar(MAX)''), - statement = oa.c.value(''(data[@name="statement"]/value/text())[1]'', ''nvarchar(MAX)''), + sql_text = oa.c.value(''(action[@name="sql_text"]/value/text())[1]'', ''nvarchar(max)''), + statement = oa.c.value(''(data[@name="statement"]/value/text())[1]'', ''nvarchar(max)''), [showplan_xml] = oa.c.query(''(data[@name="showplan_xml"]/value/*)[1]''), cpu_ms = oa.c.value(''(data[@name="cpu_time"]/value/text())[1]'', ''bigint'') / 1000., logical_reads = (oa.c.value(''(data[@name="logical_reads"]/value/text())[1]'', ''bigint'') * 8) / 1024., @@ -4348,12 +4348,12 @@ AND oa.c.exist(''(action[@name="query_hash_signed"]/value[. != 0])'') = 1; ' THEN CONVERT ( - nvarchar(MAX), + nvarchar(max), N'INSERT INTO ' + @object_name_check + N' WITH(TABLOCK) ' + @nc10 + N'( server_name, event_time, event_type, ' + @nc10 + N' database_name, object_name, recompile_cause, statement_text ' - + CONVERT(nvarchar(MAX), CASE WHEN @compile_events = 1 THEN N', compile_cpu_ms, compile_duration_ms )' ELSE N' )' END) + @nc10 + - CONVERT(nvarchar(MAX), N'SELECT + + CONVERT(nvarchar(max), CASE WHEN @compile_events = 1 THEN N', compile_cpu_ms, compile_duration_ms )' ELSE N' )' END) + @nc10 + + CONVERT(nvarchar(max), N'SELECT server = @@SERVERNAME, event_time = DATEADD @@ -4370,8 +4370,8 @@ AND oa.c.exist(''(action[@name="query_hash_signed"]/value[. != 0])'') = 1; ' database_name = oa.c.value(''(action[@name="database_name"]/value/text())[1]'', ''sysname''), [object_name] = oa.c.value(''(data[@name="object_name"]/value/text())[1]'', ''sysname''), recompile_cause = oa.c.value(''(data[@name="recompile_cause"]/text)[1]'', ''sysname''), - statement_text = oa.c.value(''(data[@name="statement"]/value/text())[1]'', ''nvarchar(MAX)'')' - + CONVERT(nvarchar(MAX), CASE WHEN @compile_events = 1 /*Only get these columns if we're using the newer XE: sql_statement_post_compile*/ + statement_text = oa.c.value(''(data[@name="statement"]/value/text())[1]'', ''nvarchar(max)'')' + + CONVERT(nvarchar(max), CASE WHEN @compile_events = 1 /*Only get these columns if we're using the newer XE: sql_statement_post_compile*/ THEN N' , compile_cpu_ms = oa.c.value(''(data[@name="cpu_time"]/value/text())[1]'', ''bigint''), @@ -4381,12 +4381,12 @@ AND oa.c.exist(''(action[@name="query_hash_signed"]/value[. != 0])'') = 1; ' FROM #human_events_xml_internal AS xet OUTER APPLY xet.human_events_xml.nodes(''//event'') AS oa(c) WHERE 1 = 1 ' - + CONVERT(nvarchar(MAX), CASE WHEN @compile_events = 1 /*Same here, where we need to filter data*/ + + CONVERT(nvarchar(max), CASE WHEN @compile_events = 1 /*Same here, where we need to filter data*/ THEN N' AND oa.c.exist(''(data[@name="is_recompile"]/value[. = "false"])'') = 0 ' ELSE N'' - END) + CONVERT(nvarchar(MAX), N' + END) + CONVERT(nvarchar(max), N' AND oa.c.exist(''@timestamp[. > sql:variable("@date_filter")]'') = 1 ORDER BY event_time;' @@ -4395,12 +4395,12 @@ ORDER BY THEN CONVERT ( - nvarchar(MAX), + nvarchar(max), N'INSERT INTO ' + REPLACE(@object_name_check, N'_parameterization', N'') + N' WITH(TABLOCK) ' + @nc10 + N'( server_name, event_time, event_type, ' + @nc10 + N' database_name, object_name, statement_text ' - + CONVERT(nvarchar(MAX), CASE WHEN @compile_events = 1 THEN N', compile_cpu_ms, compile_duration_ms )' ELSE N' )' END) + @nc10 + - CONVERT(nvarchar(MAX), N'SELECT + + CONVERT(nvarchar(max), CASE WHEN @compile_events = 1 THEN N', compile_cpu_ms, compile_duration_ms )' ELSE N' )' END) + @nc10 + + CONVERT(nvarchar(max), N'SELECT server_name = @@SERVERNAME, event_time = DATEADD @@ -4417,8 +4417,8 @@ ORDER BY event_type = oa.c.value(''@name'', ''sysname''), database_name = oa.c.value(''(action[@name="database_name"]/value/text())[1]'', ''sysname''), [object_name] = oa.c.value(''(data[@name="object_name"]/value/text())[1]'', ''sysname''), - statement_text = oa.c.value(''(data[@name="statement"]/value/text())[1]'', ''nvarchar(MAX)'')' - + CONVERT(nvarchar(MAX), CASE WHEN @compile_events = 1 /*Only get these columns if we're using the newer XE: sql_statement_post_compile*/ + statement_text = oa.c.value(''(data[@name="statement"]/value/text())[1]'', ''nvarchar(max)'')' + + CONVERT(nvarchar(max), CASE WHEN @compile_events = 1 /*Only get these columns if we're using the newer XE: sql_statement_post_compile*/ THEN N' , compile_cpu_ms = oa.c.value(''(data[@name="cpu_time"]/value/text())[1]'', ''bigint''), @@ -4428,12 +4428,12 @@ ORDER BY FROM #human_events_xml_internal AS xet OUTER APPLY xet.human_events_xml.nodes(''//event'') AS oa(c) WHERE 1 = 1 ' - + CONVERT(nvarchar(MAX), CASE WHEN @compile_events = 1 /*Just like above*/ + + CONVERT(nvarchar(max), CASE WHEN @compile_events = 1 /*Just like above*/ THEN N' AND oa.c.exist(''(data[@name="is_recompile"]/value[. = "false"])'') = 1 ' ELSE N'' - END) + CONVERT(nvarchar(MAX), N' + END) + CONVERT(nvarchar(max), N' AND oa.c.exist(''@name[.= "sql_statement_post_compile"]'') = 1 AND oa.c.exist(''@timestamp[. > sql:variable("@date_filter")]'') = 1 ORDER BY @@ -4444,12 +4444,12 @@ ORDER BY @nc10 + CONVERT ( - nvarchar(MAX), + nvarchar(max), N'INSERT INTO ' + REPLACE(@object_name_check, N'_parameterization', N'') + N'_parameterization' + N' WITH(TABLOCK) ' + @nc10 + N'( server_name, event_time, event_type, database_name, sql_text, compile_cpu_time_ms, ' + @nc10 + N' compile_duration_ms, query_param_type, is_cached, is_recompiled, compile_code, has_literals, ' + @nc10 + N' is_parameterizable, parameterized_values_count, query_plan_hash, query_hash, plan_handle, statement_sql_hash ) ' + @nc10 + - CONVERT(nvarchar(MAX), N'SELECT + CONVERT(nvarchar(max), N'SELECT server_name = @@SERVERNAME, event_time = DATEADD @@ -4465,7 +4465,7 @@ ORDER BY ), event_type = oa.c.value(''@name'', ''sysname''), database_name = oa.c.value(''(action[@name="database_name"]/value/text())[1]'', ''sysname''), - sql_text = oa.c.value(''(action[@name="sql_text"]/value/text())[1]'', ''nvarchar(MAX)''), + sql_text = oa.c.value(''(action[@name="sql_text"]/value/text())[1]'', ''nvarchar(max)''), compile_cpu_time_ms = oa.c.value(''(data[@name="compile_cpu_time"]/value/text())[1]'', ''bigint'') / 1000., compile_duration_ms = oa.c.value(''(data[@name="compile_duration"]/value/text())[1]'', ''bigint'') / 1000., query_param_type = oa.c.value(''(data[@name="query_param_type"]/value/text())[1]'', ''integer''), @@ -4710,7 +4710,7 @@ BEGIN EXECUTE sys.sp_executesql @cleanup_tables, - N'@i_cleanup_tables nvarchar(MAX) OUTPUT', + N'@i_cleanup_tables nvarchar(max) OUTPUT', @i_cleanup_tables = @drop_holder OUTPUT; IF @debug = 1 @@ -4741,7 +4741,7 @@ BEGIN EXECUTE sys.sp_executesql @cleanup_views, - N'@i_cleanup_views nvarchar(MAX) OUTPUT', + N'@i_cleanup_views nvarchar(max) OUTPUT', @i_cleanup_views = @drop_holder OUTPUT; IF @debug = 1 diff --git a/sp_HumanEvents/sp_HumanEventsBlockViewer.sql b/sp_HumanEvents/sp_HumanEventsBlockViewer.sql index 94ffb11e..faca0bb1 100644 --- a/sp_HumanEvents/sp_HumanEventsBlockViewer.sql +++ b/sp_HumanEvents/sp_HumanEventsBlockViewer.sql @@ -50,7 +50,7 @@ EXECUTE sp_HumanEventsBlockViewer @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ IF OBJECT_ID(N'dbo.sp_HumanEventsBlockViewer', N'P') IS NULL @@ -62,28 +62,38 @@ GO ALTER PROCEDURE dbo.sp_HumanEventsBlockViewer ( - @session_name nvarchar(256) = N'keeper_HumanEvents_blocking', - @target_type sysname = NULL, - @start_date datetime2 = NULL, - @end_date datetime2 = NULL, - @database_name sysname = NULL, - @object_name sysname = NULL, - @help bit = 0, - @debug bit = 0, - @version varchar(30) = NULL OUTPUT, - @version_date datetime = NULL OUTPUT + @session_name sysname = N'keeper_HumanEvents_blocking', /*Event session name*/ + @target_type sysname = NULL, /*ring buffer, file, or table*/ + @start_date datetime2 = NULL, /*when to start looking for blocking*/ + @end_date datetime2 = NULL, /*when to stop looking for blocking*/ + @database_name sysname = NULL, /*target a specific database*/ + @object_name sysname = NULL, /*target a specific schema-prefixed table*/ + @target_database sysname = NULL, /*database containing the table with BPR data*/ + @target_schema sysname = NULL, /*schema of the table*/ + @target_table sysname = NULL, /*table name*/ + @target_column sysname = NULL, /*column containing XML data*/ + @timestamp_column sysname = NULL, /*column containing timestamp (optional)*/ + @log_to_table bit = 0, /*enable logging to permanent tables*/ + @log_database_name sysname = NULL, /*database to store logging tables*/ + @log_schema_name sysname = NULL, /*schema to store logging tables*/ + @log_table_name_prefix sysname = 'HumanEventsBlockViewer', /*prefix for all logging tables*/ + @log_retention_days integer = 30, /*Number of days to keep logs, 0 = keep indefinitely*/ + @help bit = 0, /*get help with this procedure*/ + @debug bit = 0, /*print dynamic sql and select temp table contents*/ + @version varchar(30) = NULL OUTPUT, /*check the version number*/ + @version_date datetime = NULL OUTPUT /*check the version date*/ ) WITH RECOMPILE AS BEGIN SET STATISTICS XML OFF; SET NOCOUNT ON; -SET XACT_ABORT ON; +SET XACT_ABORT OFF; SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT - @version = '4.1', - @version_date = '20250101'; + @version = '4.4', + @version_date = '20250401'; IF @help = 1 BEGIN @@ -105,11 +115,21 @@ BEGIN description = CASE ap.name WHEN N'@session_name' THEN 'name of the extended event session to pull from' - WHEN N'@target_type' THEN 'target of the extended event session' + WHEN N'@target_type' THEN 'target type of the extended event session (ring buffer, file) or ''table'' to read from a table' WHEN N'@start_date' THEN 'filter by date' WHEN N'@end_date' THEN 'filter by date' WHEN N'@database_name' THEN 'filter by database name' WHEN N'@object_name' THEN 'filter by table name' + WHEN N'@target_database' THEN 'database containing the table with blocked process report data' + WHEN N'@target_schema' THEN 'schema of the table containing blocked process report data' + WHEN N'@target_table' THEN 'table containing blocked process report data' + WHEN N'@target_column' THEN 'column containing blocked process report XML' + WHEN N'@timestamp_column' THEN 'column containing timestamp for filtering (optional)' + WHEN N'@log_to_table' THEN N'enable logging to permanent tables instead of returning results' + WHEN N'@log_database_name' THEN N'database to store logging tables' + WHEN N'@log_schema_name' THEN N'schema to store logging tables' + WHEN N'@log_table_name_prefix' THEN N'prefix for all logging tables' + WHEN N'@log_retention_days' THEN N'how many days of data to retain' WHEN N'@help' THEN 'how you got here' WHEN N'@debug' THEN 'dumps raw temp table contents' WHEN N'@version' THEN 'OUTPUT; for support' @@ -123,6 +143,16 @@ BEGIN WHEN N'@end_date' THEN 'a reasonable date' WHEN N'@database_name' THEN 'a database that exists on this server' WHEN N'@object_name' THEN 'a schema-prefixed table name' + WHEN N'@target_database' THEN 'a database that exists on this server' + WHEN N'@target_schema' THEN 'a schema in the target database' + WHEN N'@target_table' THEN 'a table in the target schema' + WHEN N'@target_column' THEN 'an XML column containing blocked process report data' + WHEN N'@timestamp_column' THEN 'a datetime column for filtering by date range' + WHEN N'@log_to_table' THEN N'0 or 1' + WHEN N'@log_database_name' THEN N'any valid database name' + WHEN N'@log_schema_name' THEN N'any valid schema name' + WHEN N'@log_table_name_prefix' THEN N'any valid identifier' + WHEN N'@log_retention_days' THEN N'a positive integer' WHEN N'@help' THEN '0 or 1' WHEN N'@debug' THEN '0 or 1' WHEN N'@version' THEN 'none; OUTPUT' @@ -136,6 +166,16 @@ BEGIN WHEN N'@end_date' THEN 'NULL' WHEN N'@database_name' THEN 'NULL' WHEN N'@object_name' THEN 'NULL' + WHEN N'@target_database' THEN 'NULL' + WHEN N'@target_schema' THEN 'NULL' + WHEN N'@target_table' THEN 'NULL' + WHEN N'@target_column' THEN 'NULL' + WHEN N'@timestamp_column' THEN 'NULL' + WHEN N'@log_to_table' THEN N'0' + WHEN N'@log_database_name' THEN N'NULL (current database)' + WHEN N'@log_schema_name' THEN N'NULL (dbo)' + WHEN N'@log_table_name_prefix' THEN N'HumanEventsBlockViewer' + WHEN N'@log_retention_days' THEN N'30' WHEN N'@help' THEN '0' WHEN N'@debug' THEN '0' WHEN N'@version' THEN 'none; OUTPUT' @@ -284,7 +324,19 @@ DECLARE @inputbuf_bom nvarchar(1) = CONVERT(nvarchar(1), 0x0a00, 0), @start_date_original datetime2 = @start_date, - @end_date_original datetime2 = @end_date; + @end_date_original datetime2 = @end_date, + @validation_sql nvarchar(max), + @extract_sql nvarchar(max), + /*Log to table stuff*/ + @log_table_blocking sysname, + @cleanup_date datetime2(7), + @check_sql nvarchar(max) = N'', + @create_sql nvarchar(max) = N'', + @insert_sql nvarchar(max) = N'', + @log_database_schema nvarchar(1024), + @max_event_time datetime2(7), + @dsql nvarchar(max) = N'', + @mdsql nvarchar(max) = N''; /*Use some sane defaults for input parameters*/ IF @debug = 1 @@ -358,7 +410,34 @@ SELECT WHEN @session_name LIKE N'system%health' THEN 1 ELSE 0 - END; + END, + @mdsql = N' +IF OBJECT_ID(''{table_check}'', ''U'') IS NOT NULL +BEGIN + SELECT + @max_event_time = + ISNULL + ( + MAX({date_column}), + DATEADD + ( + MINUTE, + DATEDIFF + ( + MINUTE, + SYSDATETIME(), + GETUTCDATE() + ), + DATEADD + ( + DAY, + -1, + SYSDATETIME() + ) + ) + ) + FROM {table_check}; +END;'; SELECT @azure_msg = @@ -366,6 +445,358 @@ SELECT @is_system_health_msg = CONVERT(nchar(1), @is_system_health); +/*Change this here in case someone leave it NULL*/ +IF ISNULL(@target_database, DB_NAME()) IS NOT NULL +AND ISNULL(@target_schema, N'dbo') IS NOT NULL +AND @target_table IS NOT NULL +AND @target_column IS NOT NULL +BEGIN + SET @target_type = N'table'; +END; + +/* Check for table input early and validate */ +IF LOWER(@target_type) = N'table' +BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Table source detected, validating parameters', 0, 1) WITH NOWAIT; + END; + + IF @target_database IS NULL + BEGIN + SET @target_database = DB_NAME(); + END; + + IF @target_schema IS NULL + BEGIN + SET @target_schema = N'dbo' + END; + + /* Parameter validation */ + IF @target_table IS NULL + OR @target_column IS NULL + BEGIN + RAISERROR(N' + When @target_type is ''table'', you must specify @target_table and @target_column. + When @target_database or @target_schema is NULL, they default to DB_NAME() and dbo. + ', + 11, 1) WITH NOWAIT; + RETURN; + END; + + /* Check if target database exists */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM sys.databases AS d + WHERE d.name = @target_database + ) + BEGIN + RAISERROR(N'The specified @target_database ''%s'' does not exist.', 11, 1, @target_database) WITH NOWAIT; + RETURN; + END; + + /* Use dynamic SQL to validate schema, table, and column existence */ + SET @validation_sql = N' + /*Validate schema exists*/ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@target_database) + N'.sys.schemas AS s + WHERE s.name = @schema + ) + BEGIN + RAISERROR(N''The specified @target_schema %s does not exist in @database %s'', 11, 1, @schema, @database) WITH NOWAIT; + RETURN; + END; + + /*Validate table exists*/ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@target_database) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@target_database) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table + AND s.name = @schema + ) + BEGIN + RAISERROR(N''The specified @target_table %s does not exist in @schema %s in database %s'', 11, 1, @table, @schema, @database) WITH NOWAIT; + RETURN; + END; + + /*Validate column name exists*/ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@target_database) + N'.sys.columns AS c + JOIN ' + QUOTENAME(@target_database) + N'.sys.tables AS t + ON c.object_id = t.object_id + JOIN ' + QUOTENAME(@target_database) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE c.name = @column + AND t.name = @table + AND s.name = @schema + ) + BEGIN + RAISERROR(N''The specified @target_column %s does not exist in table %s.%s in database %s'', 11, 1, @column, @schema, @table, @database) WITH NOWAIT; + RETURN; + END; + + /* Validate column is XML type */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@target_database) + N'.sys.columns AS c + JOIN ' + QUOTENAME(@target_database) + N'.sys.types AS ty + ON c.user_type_id = ty.user_type_id + JOIN ' + QUOTENAME(@target_database) + N'.sys.tables AS t + ON c.object_id = t.object_id + JOIN ' + QUOTENAME(@target_database) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE c.name = @column + AND t.name = @table + AND s.name = @schema + AND ty.name = ''xml'' + ) + BEGIN + RAISERROR(N''The specified @target_column %s must be of XML data type.'', 11, 1, @column) WITH NOWAIT; + RETURN; + END; + '; + + /* Validate timestamp_column if specified */ + IF @timestamp_column IS NOT NULL + BEGIN + SET @validation_sql = @validation_sql + N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@target_database) + N'.sys.columns AS c + JOIN ' + QUOTENAME(@target_database) + N'.sys.tables AS t + ON c.object_id = t.object_id + JOIN ' + QUOTENAME(@target_database) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE c.name = @timestamp_column + AND t.name = @table + AND s.name = @schema + ) + BEGIN + RAISERROR(N''The specified @timestamp_column %s does not exist in table %s.%s in database %s'', 11, 1, @timestamp_column, @schema, @table, @database) WITH NOWAIT; + RETURN; + END; + + /* Validate timestamp column is date-ish type */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@target_database) + N'.sys.columns AS c + JOIN ' + QUOTENAME(@target_database) + N'.sys.types AS ty + ON c.user_type_id = ty.user_type_id + JOIN ' + QUOTENAME(@target_database) + N'.sys.tables AS t + ON c.object_id = t.object_id + JOIN ' + QUOTENAME(@target_database) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE c.name = @timestamp_column + AND t.name = @table + AND s.name = @schema + AND ty.name LIKE N''%date%'' + ) + BEGIN + RAISERROR(N''The specified @timestamp_column %s must be of datetime data type.'', 11, 1, @timestamp_column) WITH NOWAIT; + RETURN; + END;'; + END; + + IF @debug = 1 + BEGIN + PRINT @validation_sql; + END; + + EXECUTE sys.sp_executesql + @validation_sql, + N' + @database sysname, + @schema sysname, + @table sysname, + @column sysname, + @timestamp_column sysname + ', + @target_database, + @target_schema, + @target_table, + @target_column, + @timestamp_column; +END; + +/* Validate logging parameters */ +IF @log_to_table = 1 +BEGIN + SELECT + /* Default database name to current database if not specified */ + @log_database_name = ISNULL(@log_database_name, DB_NAME()), + /* Default schema name to dbo if not specified */ + @log_schema_name = ISNULL(@log_schema_name, N'dbo'), + @log_retention_days = + CASE + WHEN @log_retention_days < 0 + THEN ABS(@log_retention_days) + ELSE @log_retention_days + END; + + /* Validate database exists */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM sys.databases AS d + WHERE d.name = @log_database_name + ) + BEGIN + RAISERROR('The specified logging database %s does not exist. Logging will be disabled.', 11, 1, @log_database_name) WITH NOWAIT; + RETURN; + END; + + SET + @log_database_schema = + QUOTENAME(@log_database_name) + + N'.' + + QUOTENAME(@log_schema_name) + + N'.'; + + /* Generate fully qualified table names */ + SELECT + @log_table_blocking = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_BlockedProcessReport'); + + /* Check if schema exists and create it if needed */ + SET @check_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + WHERE s.name = @schema_name + ) + BEGIN + DECLARE + @create_schema_sql nvarchar(max) = N''CREATE SCHEMA '' + QUOTENAME(@schema_name); + + EXECUTE ' + QUOTENAME(@log_database_name) + N'.sys.sp_executesql @create_schema_sql; + IF @debug = 1 BEGIN RAISERROR(''Created schema %s in database %s for logging.'', 0, 1, @schema_name, @db_name) WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @check_sql, + N'@schema_name sysname, + @db_name sysname, + @debug bit', + @log_schema_name, + @log_database_name, + @debug; + + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_BlockedProcessReport'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_blocking + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + blocked_process_report varchar(22) NOT NULL, + event_time datetime2(7) NULL, + database_name nvarchar(128) NULL, + currentdbname nvarchar(256) NULL, + contentious_object nvarchar(4000) NULL, + activity varchar(8) NULL, + blocking_tree varchar(8000) NULL, + spid int NULL, + ecid int NULL, + query_text xml NULL, + wait_time_ms bigint NULL, + status nvarchar(10) NULL, + isolation_level nvarchar(50) NULL, + lock_mode nvarchar(10) NULL, + resource_owner_type nvarchar(256) NULL, + transaction_count int NULL, + transaction_name nvarchar(512) NULL, + last_transaction_started datetime2(7) NULL, + last_transaction_completed datetime2(7) NULL, + client_option_1 varchar(261) NULL, + client_option_2 varchar(307) NULL, + wait_resource nvarchar(100) NULL, + priority int NULL, + log_used bigint NULL, + client_app nvarchar(256) NULL, + host_name nvarchar(256) NULL, + login_name nvarchar(256) NULL, + transaction_id bigint NULL, + blocked_process_report_xml xml NULL + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for significant waits logging.'', 0, 1, ''' + @log_table_blocking + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Handle log retention if specified */ + IF @log_to_table = 1 AND @log_retention_days > 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Cleaning up log tables older than %i days', 0, 1, @log_retention_days) WITH NOWAIT; + END; + + SET @cleanup_date = + DATEADD + ( + DAY, + -@log_retention_days, + SYSDATETIME() + ); + + /* Clean up each log table */ + SET @dsql = N' + DELETE FROM ' + @log_table_blocking + ' + WHERE collection_time < @cleanup_date;'; + + IF @debug = 1 BEGIN PRINT @dsql; END; + + EXECUTE sys.sp_executesql + @dsql, + N'@cleanup_date datetime2(7)', + @cleanup_date; + + IF @debug = 1 + BEGIN + RAISERROR('Log cleanup complete', 0, 1) WITH NOWAIT; + END; + END; +END; + /*Temp tables for staging results*/ IF @debug = 1 BEGIN @@ -386,8 +817,8 @@ CREATE TABLE CREATE TABLE #block_findings ( - id int IDENTITY PRIMARY KEY, - check_id int NOT NULL, + id integer IDENTITY PRIMARY KEY CLUSTERED, + check_id integer NOT NULL, database_name nvarchar(256) NULL, object_name nvarchar(1000) NULL, finding_group nvarchar(100) NULL, @@ -395,6 +826,12 @@ CREATE TABLE sort_order bigint ); +IF LOWER(@target_type) = N'table' +BEGIN + GOTO TableMode; + RETURN; +END; + /*Look to see if the session exists and is running*/ IF @debug = 1 BEGIN @@ -441,7 +878,8 @@ IF @debug = 1 BEGIN RAISERROR('What kind of target does %s have?', 0, 1, @session_name) WITH NOWAIT; END; -IF @target_type IS NULL AND @is_system_health = 0 +IF @target_type IS NULL +AND @is_system_health = 0 BEGIN IF @azure = 0 BEGIN @@ -471,7 +909,8 @@ BEGIN END; /* Dump whatever we got into a temp table */ -IF @target_type = N'ring_buffer' AND @is_system_health = 0 +IF LOWER(@target_type) = N'ring_buffer' +AND @is_system_health = 0 BEGIN IF @azure = 0 BEGIN @@ -520,7 +959,8 @@ BEGIN END; END; -IF @target_type = N'event_file' AND @is_system_health = 0 +IF LOWER(@target_type) = N'event_file' +AND @is_system_health = 0 BEGIN IF @azure = 0 BEGIN @@ -635,7 +1075,8 @@ BEGIN END; -IF @target_type = N'ring_buffer' AND @is_system_health = 0 +IF LOWER(@target_type) = N'ring_buffer' +AND @is_system_health = 0 BEGIN IF @debug = 1 BEGIN @@ -643,7 +1084,9 @@ BEGIN END; INSERT - #blocking_xml WITH(TABLOCKX) + #blocking_xml + WITH + (TABLOCKX) ( human_events_xml ) @@ -656,7 +1099,8 @@ BEGIN OPTION(RECOMPILE); END; -IF @target_type = N'event_file' AND @is_system_health = 0 +IF LOWER(@target_type) = N'event_file' +AND @is_system_health = 0 BEGIN IF @debug = 1 BEGIN @@ -664,7 +1108,9 @@ BEGIN END; INSERT - #blocking_xml WITH(TABLOCKX) + #blocking_xml + WITH + (TABLOCKX) ( human_events_xml ) @@ -680,8 +1126,17 @@ END; /* This section is special for the well-hidden and much less comprehensive blocked process report stored in the system health extended event session + +Note: I do not allow logging to a table from this, because the set of columns +and available data is too incomplete, and I don't want to juggle multiple +table definitions. + +Logging to a table is only allowed from the a blocked_process_report Extended Event, +but it can either be ring buffer or file target. I don't care about that. */ -IF @is_system_health = 1 +IF @is_system_health = 1 +AND LOWER(@target_type) <> N'table' +AND @log_to_table = 0 BEGIN IF @debug = 1 BEGIN @@ -748,16 +1203,16 @@ BEGIN SELECT bx.event_time, currentdbname = bd.value('(process/@currentdbname)[1]', 'nvarchar(128)'), - spid = bd.value('(process/@spid)[1]', 'int'), - ecid = bd.value('(process/@ecid)[1]', 'int'), - query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + spid = bd.value('(process/@spid)[1]', 'integer'), + ecid = bd.value('(process/@ecid)[1]', 'integer'), + query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bd.value('(process/@waittime)[1]', 'bigint'), lastbatchstarted = bd.value('(process/@lastbatchstarted)[1]', 'datetime2'), lastbatchcompleted = bd.value('(process/@lastbatchcompleted)[1]', 'datetime2'), wait_resource = bd.value('(process/@waitresource)[1]', 'nvarchar(100)'), status = bd.value('(process/@status)[1]', 'nvarchar(10)'), - priority = bd.value('(process/@priority)[1]', 'int'), - transaction_count = bd.value('(process/@trancount)[1]', 'int'), + priority = bd.value('(process/@priority)[1]', 'integer'), + transaction_count = bd.value('(process/@trancount)[1]', 'integer'), client_app = bd.value('(process/@clientapp)[1]', 'nvarchar(256)'), host_name = bd.value('(process/@hostname)[1]', 'nvarchar(256)'), login_name = bd.value('(process/@loginname)[1]', 'nvarchar(256)'), @@ -805,16 +1260,16 @@ BEGIN SELECT bx.event_time, currentdbname = bg.value('(process/@currentdbname)[1]', 'nvarchar(128)'), - spid = bg.value('(process/@spid)[1]', 'int'), - ecid = bg.value('(process/@ecid)[1]', 'int'), - query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + spid = bg.value('(process/@spid)[1]', 'integer'), + ecid = bg.value('(process/@ecid)[1]', 'integer'), + query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bg.value('(process/@waittime)[1]', 'bigint'), last_transaction_started = bg.value('(process/@lastbatchstarted)[1]', 'datetime2'), last_transaction_completed = bg.value('(process/@lastbatchcompleted)[1]', 'datetime2'), wait_resource = bg.value('(process/@waitresource)[1]', 'nvarchar(100)'), status = bg.value('(process/@status)[1]', 'nvarchar(10)'), - priority = bg.value('(process/@priority)[1]', 'int'), - transaction_count = bg.value('(process/@trancount)[1]', 'int'), + priority = bg.value('(process/@priority)[1]', 'integer'), + transaction_count = bg.value('(process/@trancount)[1]', 'integer'), client_app = bg.value('(process/@clientapp)[1]', 'nvarchar(256)'), host_name = bg.value('(process/@hostname)[1]', 'nvarchar(256)'), login_name = bg.value('(process/@loginname)[1]', 'nvarchar(256)'), @@ -1051,13 +1506,13 @@ BEGIN 'available_plans', b.currentdbname, query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), + TRY_CAST(b.query_text AS nvarchar(max)), sql_handle = CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), stmtstart = - ISNULL(n.c.value('@stmtstart', 'int'), 0), + ISNULL(n.c.value('@stmtstart', 'integer'), 0), stmtend = - ISNULL(n.c.value('@stmtend', 'int'), -1) + ISNULL(n.c.value('@stmtend', 'integer'), -1) FROM #blocks_sh AS b CROSS APPLY b.blocked_process_report.nodes('/blocked-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) WHERE (b.currentdbname = @database_name @@ -1070,13 +1525,13 @@ BEGIN 'available_plans', b.currentdbname, query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), + TRY_CAST(b.query_text AS nvarchar(max)), sql_handle = CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), stmtstart = - ISNULL(n.c.value('@stmtstart', 'int'), 0), + ISNULL(n.c.value('@stmtstart', 'integer'), 0), stmtend = - ISNULL(n.c.value('@stmtend', 'int'), -1) + ISNULL(n.c.value('@stmtend', 'integer'), -1) FROM #blocks_sh AS b CROSS APPLY b.blocked_process_report.nodes('/blocking-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) WHERE (b.currentdbname = @database_name @@ -1159,7 +1614,8 @@ BEGIN FROM #available_plans_sh AS ap WHERE ap.sql_handle = deqs.sql_handle ) - AND deqs.query_hash IS NOT NULL; + AND deqs.query_hash IS NOT NULL + OPTION(RECOMPILE); IF @debug = 1 BEGIN @@ -1259,6 +1715,72 @@ BEGIN /*End system health section, skips checks because most of them won't run*/ END; +TableMode: +IF LOWER(@target_type) = N'table' +BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Extracting blocked process reports from table %s.%s.%s', 0, 1, @target_database, @target_schema, @target_table) WITH NOWAIT; + END; + + /* Build dynamic SQL to extract the XML */ + SET @extract_sql = N' + SELECT + human_events_xml = ' + + QUOTENAME(@target_column) + + N' + FROM ' + + QUOTENAME(@target_database) + + N'.' + + QUOTENAME(@target_schema) + + N'.' + + QUOTENAME(@target_table) + + N' AS x + CROSS APPLY x.' + + QUOTENAME(@target_column) + + N'.nodes(''/event'') AS e(x) + WHERE e.x.exist(''@name[ .= "blocked_process_report"]'') = 1'; + + /* Add timestamp filtering if specified*/ + IF @timestamp_column IS NOT NULL + BEGIN + SET @extract_sql = @extract_sql + N' + AND x.' + QUOTENAME(@timestamp_column) + N' >= @start_date + AND x.' + QUOTENAME(@timestamp_column) + N' < @end_date'; + END; + + IF @timestamp_column IS NULL + BEGIN + BEGIN + SET @extract_sql = @extract_sql + N' + AND e.x.exist(''@timestamp[. >= sql:variable("@start_date") and . < sql:variable("@end_date")]'') = 1'; + END; + END; + + SET @extract_sql = @extract_sql + N' + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT @extract_sql; + END; + + /* Execute the dynamic SQL*/ + INSERT + #blocking_xml + WITH + (TABLOCKX) + ( + human_events_xml + ) + EXECUTE sys.sp_executesql + @extract_sql, + N'@start_date datetime2, + @end_date datetime2', + @start_date, + @end_date; +END; IF @debug = 1 BEGIN @@ -1284,17 +1806,17 @@ SELECT ), c.value('@timestamp', 'datetime2') ), - database_name = DB_NAME(c.value('(data[@name="database_id"]/value/text())[1]', 'int')), - database_id = c.value('(data[@name="database_id"]/value/text())[1]', 'int'), - object_id = c.value('(data[@name="object_id"]/value/text())[1]', 'int'), + database_name = DB_NAME(c.value('(data[@name="database_id"]/value/text())[1]', 'integer')), + database_id = c.value('(data[@name="database_id"]/value/text())[1]', 'integer'), + object_id = c.value('(data[@name="object_id"]/value/text())[1]', 'integer'), transaction_id = c.value('(data[@name="transaction_id"]/value/text())[1]', 'bigint'), resource_owner_type = c.value('(data[@name="resource_owner_type"]/text)[1]', 'nvarchar(256)'), - monitor_loop = c.value('(//@monitorLoop)[1]', 'int'), - blocking_spid = bg.value('(process/@spid)[1]', 'int'), - blocking_ecid = bg.value('(process/@ecid)[1]', 'int'), - blocked_spid = bd.value('(process/@spid)[1]', 'int'), - blocked_ecid = bd.value('(process/@ecid)[1]', 'int'), - query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + monitor_loop = c.value('(//@monitorLoop)[1]', 'integer'), + blocking_spid = bg.value('(process/@spid)[1]', 'integer'), + blocking_ecid = bg.value('(process/@ecid)[1]', 'integer'), + blocked_spid = bd.value('(process/@spid)[1]', 'integer'), + blocked_ecid = bd.value('(process/@ecid)[1]', 'integer'), + query_text_pre = bd.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bd.value('(process/@waittime)[1]', 'bigint'), transaction_name = bd.value('(process/@transactionname)[1]', 'nvarchar(512)'), last_transaction_started = bd.value('(process/@lasttranstarted)[1]', 'datetime2'), @@ -1302,8 +1824,8 @@ SELECT wait_resource = bd.value('(process/@waitresource)[1]', 'nvarchar(100)'), lock_mode = bd.value('(process/@lockMode)[1]', 'nvarchar(10)'), status = bd.value('(process/@status)[1]', 'nvarchar(10)'), - priority = bd.value('(process/@priority)[1]', 'int'), - transaction_count = bd.value('(process/@trancount)[1]', 'int'), + priority = bd.value('(process/@priority)[1]', 'integer'), + transaction_count = bd.value('(process/@trancount)[1]', 'integer'), client_app = bd.value('(process/@clientapp)[1]', 'nvarchar(256)'), host_name = bd.value('(process/@hostname)[1]', 'nvarchar(256)'), login_name = bd.value('(process/@loginname)[1]', 'nvarchar(256)'), @@ -1312,7 +1834,7 @@ SELECT clientoption1 = bd.value('(process/@clientoption1)[1]', 'bigint'), clientoption2 = bd.value('(process/@clientoption1)[1]', 'bigint'), currentdbname = bd.value('(process/@currentdbname)[1]', 'nvarchar(256)'), - currentdbid = bd.value('(process/@currentdb)[1]', 'int'), + currentdbid = bd.value('(process/@currentdb)[1]', 'integer'), blocking_level = 0, sort_order = CAST('' AS varchar(400)), activity = CASE WHEN oa.c.exist('//blocked-process-report/blocked-process') = 1 THEN 'blocked' END, @@ -1404,17 +1926,17 @@ SELECT ), c.value('@timestamp', 'datetime2') ), - database_name = DB_NAME(c.value('(data[@name="database_id"]/value/text())[1]', 'int')), - database_id = c.value('(data[@name="database_id"]/value/text())[1]', 'int'), - object_id = c.value('(data[@name="object_id"]/value/text())[1]', 'int'), + database_name = DB_NAME(c.value('(data[@name="database_id"]/value/text())[1]', 'integer')), + database_id = c.value('(data[@name="database_id"]/value/text())[1]', 'integer'), + object_id = c.value('(data[@name="object_id"]/value/text())[1]', 'integer'), transaction_id = c.value('(data[@name="transaction_id"]/value/text())[1]', 'bigint'), resource_owner_type = c.value('(data[@name="resource_owner_type"]/text)[1]', 'nvarchar(256)'), - monitor_loop = c.value('(//@monitorLoop)[1]', 'int'), - blocking_spid = bg.value('(process/@spid)[1]', 'int'), - blocking_ecid = bg.value('(process/@ecid)[1]', 'int'), - blocked_spid = bd.value('(process/@spid)[1]', 'int'), - blocked_ecid = bd.value('(process/@ecid)[1]', 'int'), - query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(MAX)'), + monitor_loop = c.value('(//@monitorLoop)[1]', 'integer'), + blocking_spid = bg.value('(process/@spid)[1]', 'integer'), + blocking_ecid = bg.value('(process/@ecid)[1]', 'integer'), + blocked_spid = bd.value('(process/@spid)[1]', 'integer'), + blocked_ecid = bd.value('(process/@ecid)[1]', 'integer'), + query_text_pre = bg.value('(process/inputbuf/text())[1]', 'nvarchar(max)'), wait_time = bg.value('(process/@waittime)[1]', 'bigint'), transaction_name = bg.value('(process/@transactionname)[1]', 'nvarchar(512)'), last_transaction_started = bg.value('(process/@lastbatchstarted)[1]', 'datetime2'), @@ -1422,8 +1944,8 @@ SELECT wait_resource = bg.value('(process/@waitresource)[1]', 'nvarchar(100)'), lock_mode = bg.value('(process/@lockMode)[1]', 'nvarchar(10)'), status = bg.value('(process/@status)[1]', 'nvarchar(10)'), - priority = bg.value('(process/@priority)[1]', 'int'), - transaction_count = bg.value('(process/@trancount)[1]', 'int'), + priority = bg.value('(process/@priority)[1]', 'integer'), + transaction_count = bg.value('(process/@trancount)[1]', 'integer'), client_app = bg.value('(process/@clientapp)[1]', 'nvarchar(256)'), host_name = bg.value('(process/@hostname)[1]', 'nvarchar(256)'), login_name = bg.value('(process/@loginname)[1]', 'nvarchar(256)'), @@ -1432,7 +1954,7 @@ SELECT clientoption1 = bg.value('(process/@clientoption1)[1]', 'bigint'), clientoption2 = bg.value('(process/@clientoption1)[1]', 'bigint'), currentdbname = bg.value('(process/@currentdbname)[1]', 'nvarchar(128)'), - currentdbid = bg.value('(process/@currentdb)[1]', 'int'), + currentdbid = bg.value('(process/@currentdb)[1]', 'integer'), blocking_level = 0, sort_order = CAST('' AS varchar(400)), activity = CASE WHEN oa.c.exist('//blocked-process-report/blocking-process') = 1 THEN 'blocking' END, @@ -1523,15 +2045,15 @@ WITH CAST ( blocking_desc + - ' <-- ' + + ' @max_event_time'; +END; + +/* Add the ORDER BY clause */ +SET @dsql += N' ORDER BY b.event_time, b.sort_order, CASE - WHEN b.activity = 'blocking' + WHEN b.activity = ''blocking'' THEN -1 ELSE +1 END -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #available_plans', 0, 1) WITH NOWAIT; -END; - -SELECT DISTINCT - b.* -INTO #available_plans -FROM +OPTION(RECOMPILE);'; + +/* Handle table logging */ +IF @log_to_table = 1 +BEGIN + SET @insert_sql = N' +INSERT INTO + ' + @log_table_blocking + N' ( - SELECT - available_plans = - 'available_plans', - b.database_name, - b.database_id, - b.currentdbname, - b.currentdbid, - b.contentious_object, - query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), - sql_handle = - CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), - stmtstart = - ISNULL(n.c.value('@stmtstart', 'int'), 0), - stmtend = - ISNULL(n.c.value('@stmtend', 'int'), -1) - FROM #blocks AS b - CROSS APPLY b.blocked_process_report.nodes('/event/data/value/blocked-process-report/blocking-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) - WHERE (b.database_name = @database_name - OR @database_name IS NULL) - AND (b.contentious_object = @object_name - OR @object_name IS NULL) - - UNION ALL - - SELECT - available_plans = - 'available_plans', - b.database_name, - b.database_id, - b.currentdbname, - b.currentdbid, - b.contentious_object, - query_text = - TRY_CAST(b.query_text AS nvarchar(MAX)), - sql_handle = - CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), - stmtstart = - ISNULL(n.c.value('@stmtstart', 'int'), 0), - stmtend = - ISNULL(n.c.value('@stmtend', 'int'), -1) - FROM #blocks AS b - CROSS APPLY b.blocked_process_report.nodes('/event/data/value/blocked-process-report/blocking-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) - WHERE (b.database_name = @database_name - OR @database_name IS NULL) - AND (b.contentious_object = @object_name - OR @object_name IS NULL) -) AS b -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - SELECT - '#available_plans' AS table_name, - ap.* - FROM #available_plans AS ap - OPTION(RECOMPILE); - - RAISERROR('Inserting #dm_exec_query_stats', 0, 1) WITH NOWAIT; + blocked_process_report, + event_time, + database_name, + currentdbname, + contentious_object, + activity, + blocking_tree, + spid, + ecid, + query_text, + wait_time_ms, + status, + isolation_level, + lock_mode, + resource_owner_type, + transaction_count, + transaction_name, + last_transaction_started, + last_transaction_completed, + client_option_1, + client_option_2, + wait_resource, + priority, + log_used, + client_app, + host_name, + login_name, + transaction_id, + blocked_process_report_xml +)' + + @dsql; + + IF @debug = 1 BEGIN PRINT @insert_sql; END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@max_event_time datetime2(7), + @object_name sysname', + @max_event_time, + @object_name; END; -SELECT - deqs.sql_handle, - deqs.plan_handle, - deqs.statement_start_offset, - deqs.statement_end_offset, - deqs.creation_time, - deqs.last_execution_time, - deqs.execution_count, - total_worker_time_ms = - deqs.total_worker_time / 1000., - avg_worker_time_ms = - CONVERT(decimal(38, 6), deqs.total_worker_time / 1000. / deqs.execution_count), - total_elapsed_time_ms = - deqs.total_elapsed_time / 1000., - avg_elapsed_time_ms = - CONVERT(decimal(38, 6), deqs.total_elapsed_time / 1000. / deqs.execution_count), - executions_per_second = - ISNULL - ( - deqs.execution_count / - NULLIF - ( - DATEDIFF - ( - SECOND, - deqs.creation_time, - NULLIF(deqs.last_execution_time, '1900-01-01 00:00:00.000') - ), - 0 - ), - 0 - ), - total_physical_reads_mb = - deqs.total_physical_reads * 8. / 1024., - total_logical_writes_mb = - deqs.total_logical_writes * 8. / 1024., - total_logical_reads_mb = - deqs.total_logical_reads * 8. / 1024., - min_grant_mb = - deqs.min_grant_kb * 8. / 1024., - max_grant_mb = - deqs.max_grant_kb * 8. / 1024., - min_used_grant_mb = - deqs.min_used_grant_kb * 8. / 1024., - max_used_grant_mb = - deqs.max_used_grant_kb * 8. / 1024., - deqs.min_reserved_threads, - deqs.max_reserved_threads, - deqs.min_used_threads, - deqs.max_used_threads, - deqs.total_rows, - max_worker_time_ms = - deqs.max_worker_time / 1000., - max_elapsed_time_ms = - deqs.max_elapsed_time / 1000. -INTO #dm_exec_query_stats -FROM sys.dm_exec_query_stats AS deqs -WHERE EXISTS -( - SELECT - 1/0 - FROM #available_plans AS ap - WHERE ap.sql_handle = deqs.sql_handle -) -AND deqs.query_hash IS NOT NULL; - -IF @debug = 1 -BEGIN - RAISERROR('Creating index on #dm_exec_query_stats', 0, 1) WITH NOWAIT; +/* Execute the query for client results */ +IF @log_to_table = 0 +BEGIN + + IF @debug = 1 BEGIN PRINT @dsql; END; + + EXECUTE sys.sp_executesql + @dsql, + N'@object_name sysname', + @object_name; END; -CREATE CLUSTERED INDEX - deqs -ON #dm_exec_query_stats -( - sql_handle, - plan_handle -); - -SELECT - ap.available_plans, - ap.database_name, - ap.currentdbname, - query_text = - TRY_CAST(ap.query_text AS xml), - ap.query_plan, - ap.creation_time, - ap.last_execution_time, - ap.execution_count, - ap.executions_per_second, - ap.total_worker_time_ms, - ap.avg_worker_time_ms, - ap.max_worker_time_ms, - ap.total_elapsed_time_ms, - ap.avg_elapsed_time_ms, - ap.max_elapsed_time_ms, - ap.total_logical_reads_mb, - ap.total_physical_reads_mb, - ap.total_logical_writes_mb, - ap.min_grant_mb, - ap.max_grant_mb, - ap.min_used_grant_mb, - ap.max_used_grant_mb, - ap.min_reserved_threads, - ap.max_reserved_threads, - ap.min_used_threads, - ap.max_used_threads, - ap.total_rows, - ap.sql_handle, - ap.statement_start_offset, - ap.statement_end_offset -FROM -( - - SELECT - ap.*, - c.statement_start_offset, - c.statement_end_offset, - c.creation_time, - c.last_execution_time, - c.execution_count, - c.total_worker_time_ms, - c.avg_worker_time_ms, - c.total_elapsed_time_ms, - c.avg_elapsed_time_ms, - c.executions_per_second, - c.total_physical_reads_mb, - c.total_logical_writes_mb, - c.total_logical_reads_mb, - c.min_grant_mb, - c.max_grant_mb, - c.min_used_grant_mb, - c.max_used_grant_mb, - c.min_reserved_threads, - c.max_reserved_threads, - c.min_used_threads, - c.max_used_threads, - c.total_rows, - c.query_plan, - c.max_worker_time_ms, - c.max_elapsed_time_ms - FROM #available_plans AS ap - OUTER APPLY +/* +Only run query plan and check stuff +when not logging to a table +*/ +IF @log_to_table = 0 +BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Inserting #available_plans', 0, 1) WITH NOWAIT; + END; + + SELECT DISTINCT + b.* + INTO #available_plans + FROM ( SELECT - deqs.*, - query_plan = - TRY_CAST(deps.query_plan AS xml) - FROM #dm_exec_query_stats deqs - OUTER APPLY sys.dm_exec_text_query_plan + available_plans = + 'available_plans', + b.database_name, + b.database_id, + b.currentdbname, + b.currentdbid, + b.contentious_object, + query_text = + TRY_CAST(b.query_text AS nvarchar(max)), + sql_handle = + CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), + stmtstart = + ISNULL(n.c.value('@stmtstart', 'integer'), 0), + stmtend = + ISNULL(n.c.value('@stmtend', 'integer'), -1) + FROM #blocks AS b + CROSS APPLY b.blocked_process_report.nodes('/event/data/value/blocked-process-report/blocking-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) + WHERE ( - deqs.plan_handle, - deqs.statement_start_offset, - deqs.statement_end_offset - ) AS deps - WHERE deqs.sql_handle = ap.sql_handle - AND deps.dbid IN (ap.database_id, ap.currentdbid) - ) AS c -) AS ap -WHERE ap.query_plan IS NOT NULL -ORDER BY - ap.avg_worker_time_ms DESC -OPTION(RECOMPILE, LOOP JOIN, HASH JOIN); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id -1', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = -1, - database_name = N'erikdarling.com', - object_name = N'sp_HumanEventsBlockViewer version ' + CONVERT(nvarchar(30), @version) + N'.', - finding_group = N'https://github.com/erikdarlingdata/DarlingData', - finding = N'blocking for period ' + CONVERT(nvarchar(30), @start_date_original, 126) + N' through ' + CONVERT(nvarchar(30), @end_date_original, 126) + N'.', - 1; - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 1', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 1, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Database Locks', - finding = - N'The database ' + - b.database_name + - N' has been involved in ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' blocking sessions.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 2', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 2, - database_name = - b.database_name, - object_name = - b.contentious_object, - finding_group = - N'Object Locks', - finding = - N'The object ' + - b.contentious_object + - CASE - WHEN b.contentious_object LIKE N'Unresolved%' - THEN N'' - ELSE N' in database ' + - b.database_name - END + - N' has been involved in ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' blocking sessions.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name, - b.contentious_object -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 3', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 3, - database_name = - b.database_name, - object_name = - CASE - WHEN EXISTS - ( - SELECT - 1/0 - FROM sys.databases AS d - WHERE d.name COLLATE DATABASE_DEFAULT = b.database_name COLLATE DATABASE_DEFAULT - AND d.is_read_committed_snapshot_on = 1 - ) - THEN N'You already enabled RCSI, but...' - ELSE N'You Might Need RCSI' - END, - finding_group = - N'Blocking Involving Selects', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' select queries involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.lock_mode IN - ( - N'S', - N'IS' - ) -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -HAVING - COUNT_BIG(DISTINCT b.transaction_id) > 1 -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 4', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 4, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Repeatable Read Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' repeatable read queries involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.isolation_level LIKE N'repeatable%' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 5', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 5, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Serializable Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' serializable queries involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.isolation_level LIKE N'serializable%' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 6.1', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 6, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Sleeping Query Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' sleeping queries involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.status = N'sleeping' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 6.2', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 6, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Background Query Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' background tasks involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.status = N'background' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 6.3', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 6, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Done Query Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' background tasks involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.status = N'done' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 6.4', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 6, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Compile Lock Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' compile locks blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.wait_resource LIKE N'%COMPILE%' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 6.5', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 6, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Application Lock Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' application locks blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.wait_resource LIKE N'APPLICATION%' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 7.1', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 7, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Implicit Transaction Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' implicit transaction queries involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.transaction_name = N'implicit_transaction' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 7.2', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 7, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'User Transaction Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' user transaction queries involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.transaction_name = N'user_transaction' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 7.3', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 7, - database_name = - b.database_name, - object_name = - N'-', - finding_group = - N'Auto-Stats Update Blocking', - finding = - N'There have been ' + - CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + - N' user transaction queries involved in blocking sessions in ' + - b.database_name + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE b.transaction_name = N'sqlsource_transform' -AND (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 8', 0, 1) WITH NOWAIT; -END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = 8, - b.database_name, - object_name = N'-', - finding_group = N'Login, App, and Host blocking', - finding = - N'This database has had ' + - CONVERT - ( - nvarchar(20), - COUNT_BIG(DISTINCT b.transaction_id) - ) + - N' instances of blocking involving the login ' + - ISNULL - ( - b.login_name, - N'UNKNOWN' - ) + - N' from the application ' + - ISNULL + (b.database_name = @database_name + OR @database_name IS NULL) + OR (b.currentdbname = @database_name + OR @database_name IS NULL) + ) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + + UNION ALL + + SELECT + available_plans = + 'available_plans', + b.database_name, + b.database_id, + b.currentdbname, + b.currentdbid, + b.contentious_object, + query_text = + TRY_CAST(b.query_text AS nvarchar(max)), + sql_handle = + CONVERT(varbinary(64), n.c.value('@sqlhandle', 'varchar(130)'), 1), + stmtstart = + ISNULL(n.c.value('@stmtstart', 'integer'), 0), + stmtend = + ISNULL(n.c.value('@stmtend', 'integer'), -1) + FROM #blocks AS b + CROSS APPLY b.blocked_process_report.nodes('/event/data/value/blocked-process-report/blocking-process/process/executionStack/frame[not(@sqlhandle = "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")]') AS n(c) + WHERE ( - b.client_app, - N'UNKNOWN' - ) + - N' on host ' + - ISNULL + (b.database_name = @database_name + OR @database_name IS NULL) + OR (b.currentdbname = @database_name + OR @database_name IS NULL) + ) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + ) AS b + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + '#available_plans' AS table_name, + ap.* + FROM #available_plans AS ap + OPTION(RECOMPILE); + + RAISERROR('Inserting #dm_exec_query_stats', 0, 1) WITH NOWAIT; + END; + + SELECT + deqs.sql_handle, + deqs.plan_handle, + deqs.statement_start_offset, + deqs.statement_end_offset, + deqs.creation_time, + deqs.last_execution_time, + deqs.execution_count, + total_worker_time_ms = + deqs.total_worker_time / 1000., + avg_worker_time_ms = + CONVERT(decimal(38, 6), deqs.total_worker_time / 1000. / deqs.execution_count), + total_elapsed_time_ms = + deqs.total_elapsed_time / 1000., + avg_elapsed_time_ms = + CONVERT(decimal(38, 6), deqs.total_elapsed_time / 1000. / deqs.execution_count), + executions_per_second = + ISNULL + ( + deqs.execution_count / + NULLIF + ( + DATEDIFF + ( + SECOND, + deqs.creation_time, + NULLIF(deqs.last_execution_time, '1900-01-01 00:00:00.000') + ), + 0 + ), + 0 + ), + total_physical_reads_mb = + deqs.total_physical_reads * 8. / 1024., + total_logical_writes_mb = + deqs.total_logical_writes * 8. / 1024., + total_logical_reads_mb = + deqs.total_logical_reads * 8. / 1024., + min_grant_mb = + deqs.min_grant_kb * 8. / 1024., + max_grant_mb = + deqs.max_grant_kb * 8. / 1024., + min_used_grant_mb = + deqs.min_used_grant_kb * 8. / 1024., + max_used_grant_mb = + deqs.max_used_grant_kb * 8. / 1024., + deqs.min_reserved_threads, + deqs.max_reserved_threads, + deqs.min_used_threads, + deqs.max_used_threads, + deqs.total_rows, + max_worker_time_ms = + deqs.max_worker_time / 1000., + max_elapsed_time_ms = + deqs.max_elapsed_time / 1000. + INTO #dm_exec_query_stats + FROM sys.dm_exec_query_stats AS deqs + WHERE EXISTS + ( + SELECT + 1/0 + FROM #available_plans AS ap + WHERE ap.sql_handle = deqs.sql_handle + ) + AND deqs.query_hash IS NOT NULL + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Creating index on #dm_exec_query_stats', 0, 1) WITH NOWAIT; + END; + + CREATE CLUSTERED INDEX + deqs + ON #dm_exec_query_stats + ( + sql_handle, + plan_handle + ); + + SELECT + ap.available_plans, + ap.database_name, + ap.currentdbname, + query_text = + TRY_CAST(ap.query_text AS xml), + ap.query_plan, + ap.creation_time, + ap.last_execution_time, + ap.execution_count, + ap.executions_per_second, + ap.total_worker_time_ms, + ap.avg_worker_time_ms, + ap.max_worker_time_ms, + ap.total_elapsed_time_ms, + ap.avg_elapsed_time_ms, + ap.max_elapsed_time_ms, + ap.total_logical_reads_mb, + ap.total_physical_reads_mb, + ap.total_logical_writes_mb, + ap.min_grant_mb, + ap.max_grant_mb, + ap.min_used_grant_mb, + ap.max_used_grant_mb, + ap.min_reserved_threads, + ap.max_reserved_threads, + ap.min_used_threads, + ap.max_used_threads, + ap.total_rows, + ap.sql_handle, + ap.statement_start_offset, + ap.statement_end_offset + FROM + ( + + SELECT + ap.*, + c.statement_start_offset, + c.statement_end_offset, + c.creation_time, + c.last_execution_time, + c.execution_count, + c.total_worker_time_ms, + c.avg_worker_time_ms, + c.total_elapsed_time_ms, + c.avg_elapsed_time_ms, + c.executions_per_second, + c.total_physical_reads_mb, + c.total_logical_writes_mb, + c.total_logical_reads_mb, + c.min_grant_mb, + c.max_grant_mb, + c.min_used_grant_mb, + c.max_used_grant_mb, + c.min_reserved_threads, + c.max_reserved_threads, + c.min_used_threads, + c.max_used_threads, + c.total_rows, + c.query_plan, + c.max_worker_time_ms, + c.max_elapsed_time_ms + FROM #available_plans AS ap + OUTER APPLY ( - b.host_name, - N'UNKNOWN' - ) + - N'.', - sort_order = - ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) -FROM #blocks AS b -WHERE (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name, - b.login_name, - b.client_app, - b.host_name -OPTION(RECOMPILE); + SELECT + deqs.*, + query_plan = + TRY_CAST(deps.query_plan AS xml) + FROM #dm_exec_query_stats deqs + OUTER APPLY sys.dm_exec_text_query_plan + ( + deqs.plan_handle, + deqs.statement_start_offset, + deqs.statement_end_offset + ) AS deps + WHERE deqs.sql_handle = ap.sql_handle + AND deps.dbid IN (ap.database_id, ap.currentdbid) + ) AS c + ) AS ap + WHERE ap.query_plan IS NOT NULL + ORDER BY + ap.avg_worker_time_ms DESC + OPTION(RECOMPILE, LOOP JOIN, HASH JOIN); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id -1', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = -1, + database_name = N'erikdarling.com', + object_name = N'sp_HumanEventsBlockViewer version ' + CONVERT(nvarchar(30), @version) + N'.', + finding_group = N'https://code.erikdarling.com', + finding = N'blocking for period ' + CONVERT(nvarchar(30), @start_date_original, 126) + N' through ' + CONVERT(nvarchar(30), @end_date_original, 126) + N'.', + 1; + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 1', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 1, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Database Locks', + finding = + N'The database ' + + b.database_name + + N' has been involved in ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' blocking sessions.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 2', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 2, + database_name = + b.database_name, + object_name = + b.contentious_object, + finding_group = + N'Object Locks', + finding = + N'The object ' + + b.contentious_object + + CASE + WHEN b.contentious_object LIKE N'Unresolved%' + THEN N'' + ELSE N' in database ' + + b.database_name + END + + N' has been involved in ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' blocking sessions.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name, + b.contentious_object + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 3', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 3, + database_name = + b.database_name, + object_name = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM sys.databases AS d + WHERE d.name COLLATE DATABASE_DEFAULT = b.database_name COLLATE DATABASE_DEFAULT + AND d.is_read_committed_snapshot_on = 1 + ) + THEN N'You already enabled RCSI, but...' + ELSE N'You Might Need RCSI' + END, + finding_group = + N'Blocking Involving Selects', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' select queries involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.lock_mode IN + ( + N'S', + N'IS' + ) + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + HAVING + COUNT_BIG(DISTINCT b.transaction_id) > 1 + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 4', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 4, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Repeatable Read Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' repeatable read queries involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.isolation_level LIKE N'repeatable%' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 5', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 5, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Serializable Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' serializable queries involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.isolation_level LIKE N'serializable%' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 6.1', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 6, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Sleeping Query Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' sleeping queries involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.status = N'sleeping' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 1000', 0, 1) WITH NOWAIT; -END; + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 6.2', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 6, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Background Query Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' background tasks involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.status = N'background' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 6.3', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 6, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Done Query Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' background tasks involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.status = N'done' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 6.4', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 6, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Compile Lock Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' compile locks blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.wait_resource LIKE N'%COMPILE%' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 6.5', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 6, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Application Lock Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' application locks blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.wait_resource LIKE N'APPLICATION%' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 7.1', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 7, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Implicit Transaction Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' implicit transaction queries involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.transaction_name = N'implicit_transaction' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 7.2', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 7, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'User Transaction Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' user transaction queries involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.transaction_name = N'user_transaction' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 7.3', 0, 1) WITH NOWAIT; + END; -WITH - b AS -( + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 7, + database_name = + b.database_name, + object_name = + N'-', + finding_group = + N'Auto-Stats Update Blocking', + finding = + N'There have been ' + + CONVERT(nvarchar(20), COUNT_BIG(DISTINCT b.transaction_id)) + + N' auto stats updates involved in blocking sessions in ' + + b.database_name + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) + FROM #blocks AS b + WHERE b.transaction_name = N'sqlsource_transform' + AND (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 8', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) SELECT + check_id = 8, b.database_name, - b.transaction_id, - wait_time_ms = - MAX(b.wait_time_ms) + object_name = N'-', + finding_group = N'Login, App, and Host blocking', + finding = + N'This database has had ' + + CONVERT + ( + nvarchar(20), + COUNT_BIG(DISTINCT b.transaction_id) + ) + + N' instances of blocking involving the login ' + + ISNULL + ( + b.login_name, + N'UNKNOWN' + ) + + N' from the application ' + + ISNULL + ( + b.client_app, + N'UNKNOWN' + ) + + N' on host ' + + ISNULL + ( + b.host_name, + N'UNKNOWN' + ) + + N'.', + sort_order = + ROW_NUMBER() OVER (ORDER BY COUNT_BIG(DISTINCT b.transaction_id) DESC) FROM #blocks AS b WHERE (b.database_name = @database_name OR @database_name IS NULL) @@ -2774,202 +3407,228 @@ WITH OR @object_name IS NULL) GROUP BY b.database_name, - b.transaction_id -) -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 1000, - b.database_name, - object_name = - N'-', - finding_group = - N'Total database block wait time', - finding = - N'This database has had ' + - CONVERT - ( - nvarchar(30), + b.login_name, + b.client_app, + b.host_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 1000', 0, 1) WITH NOWAIT; + END; + + WITH + b AS + ( + SELECT + b.database_name, + b.transaction_id, + wait_time_ms = + MAX(b.wait_time_ms) + FROM #blocks AS b + WHERE (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name, + b.transaction_id + ) + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 1000, + b.database_name, + object_name = + N'-', + finding_group = + N'Total database block wait time', + finding = + N'This database has had ' + + CONVERT ( - SUM + nvarchar(30), ( - CONVERT + SUM ( - bigint, - b.wait_time_ms - ) - ) / 1000 / 86400 - ) - ) + - N' ' + - CONVERT - ( - nvarchar(30), - DATEADD + CONVERT + ( + bigint, + b.wait_time_ms + ) + ) / 1000 / 86400 + ) + ) + + N' ' + + CONVERT ( - MILLISECOND, + nvarchar(30), + DATEADD ( - SUM + MILLISECOND, ( - CONVERT + SUM ( - bigint, - b.wait_time_ms + CONVERT + ( + bigint, + b.wait_time_ms + ) ) - ) + ), + '19000101' ), - '19000101' - ), - 14 - ) + - N' [dd hh:mm:ss:ms] of lock wait time.', - sort_order = - ROW_NUMBER() OVER (ORDER BY SUM(CONVERT(bigint, b.wait_time_ms)) DESC) -FROM b AS b -WHERE (b.database_name = @database_name - OR @database_name IS NULL) -GROUP BY - b.database_name -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 1001', 0, 1) WITH NOWAIT; -END; - -WITH - b AS -( - SELECT - b.database_name, - b.transaction_id, - b.contentious_object, - wait_time_ms = - MAX(b.wait_time_ms) - FROM #blocks AS b + 14 + ) + + N' [dd hh:mm:ss:ms] of lock wait time.', + sort_order = + ROW_NUMBER() OVER (ORDER BY SUM(CONVERT(bigint, b.wait_time_ms)) DESC) + FROM b AS b WHERE (b.database_name = @database_name OR @database_name IS NULL) - AND (b.contentious_object = @object_name - OR @object_name IS NULL) GROUP BY + b.database_name + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 1001', 0, 1) WITH NOWAIT; + END; + + WITH + b AS + ( + SELECT + b.database_name, + b.transaction_id, + b.contentious_object, + wait_time_ms = + MAX(b.wait_time_ms) + FROM #blocks AS b + WHERE (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY + b.database_name, + b.contentious_object, + b.transaction_id + ) + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = + 1001, b.database_name, - b.contentious_object, - b.transaction_id -) -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = - 1001, - b.database_name, - object_name = - b.contentious_object, - finding_group = - N'Total database and object block wait time', - finding = - N'This object has had ' + - CONVERT - ( - nvarchar(30), + object_name = + b.contentious_object, + finding_group = + N'Total database and object block wait time', + finding = + N'This object has had ' + + CONVERT ( - SUM + nvarchar(30), ( - CONVERT + SUM ( - bigint, - b.wait_time_ms - ) - ) / 1000 / 86400 - ) - ) + - N' ' + - CONVERT - ( - nvarchar(30), - DATEADD + CONVERT + ( + bigint, + b.wait_time_ms + ) + ) / 1000 / 86400 + ) + ) + + N' ' + + CONVERT ( - MILLISECOND, + nvarchar(30), + DATEADD ( - SUM + MILLISECOND, ( - CONVERT + SUM ( - bigint, - b.wait_time_ms + CONVERT + ( + bigint, + b.wait_time_ms + ) ) - ) + ), + '19000101' ), - '19000101' - ), - 14 - ) + - N' [dd hh:mm:ss:ms] of lock wait time in database ' + + 14 + ) + + N' [dd hh:mm:ss:ms] of lock wait time in database ' + + b.database_name, + sort_order = + ROW_NUMBER() OVER (ORDER BY SUM(CONVERT(bigint, b.wait_time_ms)) DESC) + FROM b AS b + WHERE (b.database_name = @database_name + OR @database_name IS NULL) + AND (b.contentious_object = @object_name + OR @object_name IS NULL) + GROUP BY b.database_name, - sort_order = - ROW_NUMBER() OVER (ORDER BY SUM(CONVERT(bigint, b.wait_time_ms)) DESC) -FROM b AS b -WHERE (b.database_name = @database_name - OR @database_name IS NULL) -AND (b.contentious_object = @object_name - OR @object_name IS NULL) -GROUP BY - b.database_name, - b.contentious_object -OPTION(RECOMPILE); - -IF @debug = 1 -BEGIN - RAISERROR('Inserting #block_findings, check_id 2147483647', 0, 1) WITH NOWAIT; + b.contentious_object + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + RAISERROR('Inserting #block_findings, check_id 2147483647', 0, 1) WITH NOWAIT; + END; + + INSERT + #block_findings + ( + check_id, + database_name, + object_name, + finding_group, + finding, + sort_order + ) + SELECT + check_id = 2147483647, + database_name = N'erikdarling.com', + object_name = N'sp_HumanEventsBlockViewer version ' + CONVERT(nvarchar(30), @version) + N'.', + finding_group = N'https://code.erikdarling.com', + finding = N'thanks for using me!', + 2147483647; + + SELECT + findings = + 'findings', + bf.check_id, + bf.database_name, + bf.object_name, + bf.finding_group, + bf.finding + FROM #block_findings AS bf + ORDER BY + bf.check_id, + bf.finding_group, + bf.sort_order + OPTION(RECOMPILE); END; - -INSERT - #block_findings -( - check_id, - database_name, - object_name, - finding_group, - finding, - sort_order -) -SELECT - check_id = 2147483647, - database_name = N'erikdarling.com', - object_name = N'sp_HumanEventsBlockViewer version ' + CONVERT(nvarchar(30), @version) + N'.', - finding_group = N'https://github.com/erikdarlingdata/DarlingData', - finding = N'thanks for using me!', - 2147483647; - -SELECT - findings = - 'findings', - bf.check_id, - bf.database_name, - bf.object_name, - bf.finding_group, - bf.finding -FROM #block_findings AS bf -ORDER BY - bf.check_id, - bf.finding_group, - bf.sort_order -OPTION(RECOMPILE); END; --Final End GO diff --git a/sp_IndexCleanup BETA/README.md b/sp_IndexCleanup BETA/README.md deleted file mode 100644 index ceff4d92..00000000 --- a/sp_IndexCleanup BETA/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# This is the BETA VERSION of sp_IndexCleanup - -It needs lots of love and testing in real environments with real indexes to fix many issues: - * Data collection - * Deduping logic - * Result correctness - * Edge cases - - If you run this, only use the output to debug validate result correctness. - - Do not run any of the output scripts, period. Doing so may be harmful. \ No newline at end of file diff --git a/sp_IndexCleanup BETA/sp_IndexCleanup BETA.sql b/sp_IndexCleanup BETA/sp_IndexCleanup BETA.sql deleted file mode 100644 index 92c406cb..00000000 --- a/sp_IndexCleanup BETA/sp_IndexCleanup BETA.sql +++ /dev/null @@ -1,1654 +0,0 @@ -/* -EXEC sp_IndexCleanup - @database_name = 'StackOverflow2013', - @debug = 1; - -EXEC sp_IndexCleanup - @database_name = 'StackOverflow2013', - @table_name = 'Users', - @debug = 1 -*/ - -SET ANSI_WARNINGS ON; -SET ARITHABORT ON; -SET CONCAT_NULL_YIELDS_NULL ON; -SET QUOTED_IDENTIFIER ON; -SET NUMERIC_ROUNDABORT OFF; -SET IMPLICIT_TRANSACTIONS OFF; -SET STATISTICS TIME, IO OFF; -GO - -IF OBJECT_ID('dbo.sp_IndexCleanup', 'P') IS NULL -BEGIN - EXECUTE ('CREATE PROCEDURE dbo.sp_IndexCleanup AS RETURN 138;'); -END; -GO - -ALTER PROCEDURE - dbo.sp_IndexCleanup -( - @database_name sysname = NULL, - @schema_name sysname = NULL, - @table_name sysname = NULL, - @help bit = 'false', - @debug bit = 'true', - @version varchar(20) = NULL OUTPUT, - @version_date datetime = NULL OUTPUT -) -WITH RECOMPILE -AS -BEGIN -SET NOCOUNT ON; - -BEGIN TRY - SELECT - @version = '-2147483648', - @version_date = '17530101'; - - SELECT - warning = N'Read the messages pane carefully!' - - PRINT ' -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- -This is the BETA VERSION of sp_IndexCleanup - -It needs lots of love and testing in real environments with real indexes to fix many issues: - * Data collection - * Deduping logic - * Result correctness - * Edge cases - - If you run this, only use the output to debug and validate result correctness. - - Do not run any of the output scripts, period. Doing so may be harmful. - ------------------------------------------------------------------------------------------- - ------------------------------------------------------------------------------------------- - ------------------------------------------------------------------------------------------- - - '; - - - /* - Help section, for help. - Will become more helpful when out of beta. - */ - IF @help = 1 - BEGIN - SELECT - help = N'hello, i am sp_IndexCleanup - BETA' - UNION ALL - SELECT - help = N'this is a script to help clean up unused and duplicate indexes' - UNION ALL - SELECT - help = N'you are currently using a beta version, and the advice should not be followed' - UNION ALL - SELECT - help = N'without careful analysis and consideration. it may be harmful.' - - - /* - Parameters - */ - SELECT - parameter_name = - ap.name, - data_type = - t.name, - description = - CASE - ap.name - WHEN ap.name - THEN ap.name - END, - valid_inputs = - CASE - ap.name - WHEN ap.name - THEN ap.name - END, - defaults = - CASE - ap.name - WHEN ap.name - THEN ap.name - END - FROM sys.all_parameters AS ap - INNER JOIN sys.all_objects AS o - ON ap.object_id = o.object_id - INNER JOIN sys.types AS t - ON ap.system_type_id = t.system_type_id - AND ap.user_type_id = t.user_type_id - WHERE o.name = N'sp_IndexCleanup' - OPTION(MAXDOP 1, RECOMPILE); - - SELECT - mit_license_yo = 'i am MIT licensed, so like, do whatever' - - UNION ALL - - SELECT - mit_license_yo = 'see printed messages for full license'; - - RAISERROR(' -MIT License - -Copyright 2024 Darling Data, LLC - -https://www.erikdarling.com/ - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), -to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, -sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the -following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE -FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -', 0, 1) WITH NOWAIT; - - RETURN; - END; - - DECLARE - /*general script variables*/ - @sql nvarchar(MAX) = N'', - @database_id integer = NULL, - @object_id integer = NULL, - @full_object_name nvarchar(768) = NULL, - @final_script nvarchar(MAX) = '', - /*cursor variables*/ - @c_database_id integer, - @c_schema_name sysname, - @c_table_name sysname, - @c_index_name sysname, - @c_is_unique bit, - @c_filter_definition nvarchar(MAX), - /*print variables*/ - @helper integer = 0, - @sql_len integer, - @sql_debug nvarchar(MAX) = N''; - - /* - Initial checks for object validity - */ - IF @database_name IS NULL - AND DB_NAME() NOT IN - ( - N'master', - N'model', - N'msdb', - N'tempdb', - N'rdsadmin' - ) - BEGIN - SELECT - @database_name = DB_NAME(); - END; - - IF @database_name IS NOT NULL - BEGIN - SELECT - @database_id = d.database_id - FROM sys.databases AS d - WHERE d.name = @database_name; - END; - - IF @schema_name IS NULL - AND @table_name IS NOT NULL - BEGIN - SELECT - @schema_name = N'dbo'; - END; - - IF @schema_name IS NOT NULL - AND @table_name IS NOT NULL - BEGIN - SELECT - @full_object_name = - QUOTENAME(@database_name) + - N'.' + - QUOTENAME(@schema_name) + - N'.' + - QUOTENAME(@table_name); - - SELECT - @object_id = - OBJECT_ID(@full_object_name); - - IF @object_id IS NULL - BEGIN - RAISERROR('The object %s doesn''t seem to exist', 16, 1, @full_object_name) WITH NOWAIT; - RETURN; - END; - END; - - /* - Temp tables! - */ - CREATE TABLE - #operational_stats - ( - database_id integer NOT NULL, - object_id integer NOT NULL, - index_id integer NOT NULL, - range_scan_count bigint NULL, - singleton_lookup_count bigint NULL, - forwarded_fetch_count bigint NULL, - lob_fetch_in_pages bigint NULL, - row_overflow_fetch_in_pages bigint NULL, - leaf_insert_count bigint NULL, - leaf_update_count bigint NULL, - leaf_delete_count bigint NULL, - leaf_ghost_count bigint NULL, - nonleaf_insert_count bigint NULL, - nonleaf_update_count bigint NULL, - nonleaf_delete_count bigint NULL, - leaf_allocation_count bigint NULL, - nonleaf_allocation_count bigint NULL, - row_lock_count bigint NULL, - row_lock_wait_count bigint NULL, - row_lock_wait_in_ms bigint NULL, - page_lock_count bigint NULL, - page_lock_wait_count bigint NULL, - page_lock_wait_in_ms bigint NULL, - index_lock_promotion_attempt_count bigint NULL, - index_lock_promotion_count bigint NULL, - page_latch_wait_count bigint NULL, - page_latch_wait_in_ms bigint NULL, - tree_page_latch_wait_count bigint NULL, - tree_page_latch_wait_in_ms bigint NULL, - page_io_latch_wait_count bigint NULL, - page_io_latch_wait_in_ms bigint NULL, - page_compression_attempt_count bigint NULL, - page_compression_success_count bigint NULL, - PRIMARY KEY CLUSTERED(database_id, object_id, index_id) - ); - - CREATE TABLE - #index_details - ( - database_id integer NOT NULL, - object_id integer NOT NULL, - index_id integer NOT NULL, - schema_name sysname NOT NULL, - table_name sysname NOT NULL, - index_name sysname NULL, - column_name sysname NOT NULL, - is_primary_key bit NULL, - is_unique bit NULL, - is_unique_constraint bit NULL, - is_indexed_view integer NOT NULL, - is_foreign_key bit NULL, - is_foreign_key_reference bit NULL, - key_ordinal tinyint NOT NULL, - index_column_id integer NOT NULL, - is_descending_key bit NOT NULL, - is_included_column bit NULL, - filter_definition nvarchar(MAX) NULL, - is_max_length integer NOT NULL, - user_seeks bigint NOT NULL, - user_scans bigint NOT NULL, - user_lookups bigint NOT NULL, - user_updates bigint NOT NULL, - last_user_seek datetime NULL, - last_user_scan datetime NULL, - last_user_lookup datetime NULL, - last_user_update datetime NULL, - PRIMARY KEY CLUSTERED(database_id, object_id, index_id, column_name) - ); - - CREATE TABLE - #partition_stats - ( - database_id integer NOT NULL, - object_id integer NOT NULL, - index_id integer NOT NULL, - schema_name sysname NOT NULL, - table_name sysname NOT NULL, - index_name sysname NULL, - partition_id bigint NOT NULL, - partition_number int NOT NULL, - total_rows bigint NULL, - total_space_mb decimal(38, 2) NULL, - reserved_lob_mb decimal(38, 2) NULL, - reserved_row_overflow_mb decimal(38, 2) NULL, - data_compression_desc nvarchar(60) NULL, - built_on sysname NULL, - partition_function_name sysname NULL, - partition_columns nvarchar(MAX) - PRIMARY KEY CLUSTERED(database_id, object_id, index_id, partition_id) - ); - - CREATE TABLE - #index_analysis - ( - database_id integer NOT NULL, - schema_name sysname NOT NULL, - table_name sysname NOT NULL, - index_name sysname NOT NULL, - is_unique bit NULL, - key_columns nvarchar(MAX) NULL, - included_columns nvarchar(MAX) NULL, - filter_definition nvarchar(MAX) NULL, - is_redundant bit NULL, - superseded_by sysname NULL, - missing_columns nvarchar(MAX) NULL, - action nvarchar(MAX) NULL, - INDEX c CLUSTERED - (database_id, schema_name, table_name, index_name) - ); - - CREATE TABLE - #index_cleanup_report - ( - database_name sysname NOT NULL, - table_name sysname NOT NULL, - index_name sysname NOT NULL, - action nvarchar(MAX) NULL, - cleanup_script nvarchar(MAX) NULL, - original_definition nvarchar(MAX) NULL, - /*Usage details*/ - user_seeks bigint NULL, - user_scans bigint NULL, - user_lookups bigint NULL, - user_updates bigint NULL, - last_user_seek datetime NULL, - last_user_scan datetime NULL, - last_user_lookup datetime NULL, - last_user_update datetime NULL, - /*Operational stats*/ - range_scan_count bigint NULL, - singleton_lookup_count bigint NULL, - leaf_insert_count bigint NULL, - leaf_update_count bigint NULL, - leaf_delete_count bigint NULL, - page_lock_count bigint NULL, - page_lock_wait_count bigint NULL, - page_lock_wait_in_ms bigint NULL - ); - - CREATE TABLE - #index_cleanup_summary - ( - database_name sysname NOT NULL, - table_name sysname NOT NULL, - index_name sysname NOT NULL, - action nvarchar(MAX) NOT NULL, - details nvarchar(MAX) NULL, - current_definition nvarchar(MAX) NOT NULL, - proposed_definition nvarchar(MAX) NULL, - usage_summary nvarchar(MAX) NULL, - operational_summary nvarchar(MAX) NULL - ); - - CREATE TABLE - #final_index_actions - ( - database_name sysname NOT NULL, - table_name sysname NOT NULL, - index_name sysname NOT NULL, - action nvarchar(MAX) NOT NULL, - script nvarchar(MAX) NOT NULL - ); - - /* - Start insert queries - */ - SELECT - @sql = N' - SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; - - SELECT - @sql += N' - SELECT - os.database_id, - os.object_id, - os.index_id, - range_scan_count = SUM(os.range_scan_count), - singleton_lookup_count = SUM(os.singleton_lookup_count), - forwarded_fetch_count = SUM(os.forwarded_fetch_count), - lob_fetch_in_pages = SUM(os.lob_fetch_in_pages), - row_overflow_fetch_in_pages = SUM(os.row_overflow_fetch_in_pages), - leaf_insert_count = SUM(os.leaf_insert_count), - leaf_update_count = SUM(os.leaf_update_count), - leaf_delete_count = SUM(os.leaf_delete_count), - leaf_ghost_count = SUM(os.leaf_ghost_count), - nonleaf_insert_count = SUM(os.nonleaf_insert_count), - nonleaf_update_count = SUM(os.nonleaf_update_count), - nonleaf_delete_count = SUM(os.nonleaf_delete_count), - leaf_allocation_count = SUM(os.leaf_allocation_count), - nonleaf_allocation_count = SUM(os.nonleaf_allocation_count), - row_lock_count = SUM(os.row_lock_count), - row_lock_wait_count = SUM(os.row_lock_wait_count), - row_lock_wait_in_ms = SUM(os.row_lock_wait_in_ms), - page_lock_count = SUM(os.page_lock_count), - page_lock_wait_count = SUM(os.page_lock_wait_count), - page_lock_wait_in_ms = SUM(os.page_lock_wait_in_ms), - index_lock_promotion_attempt_count = SUM(os.index_lock_promotion_attempt_count), - index_lock_promotion_count = SUM(os.index_lock_promotion_count), - page_latch_wait_count = SUM(os.page_latch_wait_count), - page_latch_wait_in_ms = SUM(os.page_latch_wait_in_ms), - tree_page_latch_wait_count = SUM(os.tree_page_latch_wait_count), - tree_page_latch_wait_in_ms = SUM(os.tree_page_latch_wait_in_ms), - page_io_latch_wait_count = SUM(os.page_io_latch_wait_count), - page_io_latch_wait_in_ms = SUM(os.page_io_latch_wait_in_ms), - page_compression_attempt_count = SUM(os.page_compression_attempt_count), - page_compression_success_count = SUM(os.page_compression_success_count) - FROM ' + QUOTENAME(@database_name) + N'.sys.dm_db_index_operational_stats - ( - @database_id, - @object_id, - NULL, - NULL - ) AS os - WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + QUOTENAME(@database_name) + N'.sys.tables AS t - WHERE t.object_id = os.object_id - AND t.is_ms_shipped = 0 - ) - AND os.index_id > 1 - GROUP BY - os.database_id, - os.object_id, - os.index_id - OPTION(RECOMPILE);'; - - IF @debug = 1 - BEGIN - PRINT @sql; - END; - - INSERT - #operational_stats - WITH - (TABLOCK) - ( - database_id, - object_id, - index_id, - range_scan_count, - singleton_lookup_count, - forwarded_fetch_count, - lob_fetch_in_pages, - row_overflow_fetch_in_pages, - leaf_insert_count, - leaf_update_count, - leaf_delete_count, - leaf_ghost_count, - nonleaf_insert_count, - nonleaf_update_count, - nonleaf_delete_count, - leaf_allocation_count, - nonleaf_allocation_count, - row_lock_count, - row_lock_wait_count, - row_lock_wait_in_ms, - page_lock_count, - page_lock_wait_count, - page_lock_wait_in_ms, - index_lock_promotion_attempt_count, - index_lock_promotion_count, - page_latch_wait_count, - page_latch_wait_in_ms, - tree_page_latch_wait_count, - tree_page_latch_wait_in_ms, - page_io_latch_wait_count, - page_io_latch_wait_in_ms, - page_compression_attempt_count, - page_compression_success_count - ) - EXEC sys.sp_executesql - @sql, - N'@database_id integer, - @object_id integer', - @database_id, - @object_id; - - IF @debug = 1 - BEGIN - SELECT - table_name = '#operational_stats', - os.* - FROM #operational_stats AS os; - END; - - SELECT - @sql = N' - SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; - - SELECT - @sql += N' - SELECT - database_id = @database_id, - t.object_id, - i.index_id, - schema_name = s.name, - table_name = t.name, - index_name = i.name, - column_name = c.name, - i.is_primary_key, - i.is_unique, - i.is_unique_constraint, - is_indexed_view = - CASE - WHEN EXISTS - ( - SELECT - 1/0 - FROM ' + QUOTENAME(@database_name) + N'.sys.objects AS so - WHERE i.object_id = so.object_id - AND so.is_ms_shipped = 0 - AND so.type = ''V'' - ) - THEN 1 - ELSE 0 - END, - is_foreign_key = - CASE - WHEN EXISTS - ( - SELECT - 1/0 - FROM ' + QUOTENAME(@database_name) + N'.sys.foreign_key_columns AS f - WHERE f.parent_column_id = c.column_id - AND f.parent_object_id = c.object_id - ) - THEN 1 - ELSE 0 - END, - is_foreign_key_reference = - CASE - WHEN EXISTS - ( - SELECT - 1/0 - FROM ' + QUOTENAME(@database_name) + N'.sys.foreign_key_columns AS f - WHERE f.referenced_column_id = c.column_id - AND f.referenced_object_id = c.object_id - ) - THEN 1 - ELSE 0 - END, - ic.key_ordinal, - ic.index_column_id, - ic.is_descending_key, - ic.is_included_column, - i.filter_definition, - is_max_length = - CASE - WHEN EXISTS - ( - SELECT - 1/0 - FROM ' + QUOTENAME(@database_name) + N'.sys.types AS t - WHERE c.system_type_id = t.system_type_id - AND c.user_type_id = t.user_type_id - AND t.name IN (N''varchar'', N''nvarchar'') - AND t.max_length = -1 - ) - THEN 1 - ELSE 0 - END, - user_seeks = ISNULL(us.user_seeks, 0), - user_scans = ISNULL(us.user_scans, 0), - user_lookups = ISNULL(us.user_lookups, 0), - user_updates = ISNULL(us.user_updates, 0), - us.last_user_seek, - us.last_user_scan, - us.last_user_lookup, - us.last_user_update - FROM ' + QUOTENAME(@database_name) + N'.sys.tables AS t - JOIN ' + QUOTENAME(@database_name) + N'.sys.schemas AS s - ON t.schema_id = s.schema_id - JOIN ' + QUOTENAME(@database_name) + N'.sys.indexes AS i - ON t.object_id = i.object_id - JOIN ' + QUOTENAME(@database_name) + N'.sys.index_columns AS ic - ON i.object_id = ic.object_id - AND i.index_id = ic.index_id - JOIN ' + QUOTENAME(@database_name) + N'.sys.columns AS c - ON ic.object_id = c.object_id - AND ic.column_id = c.column_id - LEFT JOIN sys.dm_db_index_usage_stats AS us - ON i.object_id = us.object_id - AND i.index_id = us.index_id - AND us.database_id = @database_id - WHERE t.is_ms_shipped = 0 - AND i.type = 2 - AND i.is_disabled = 0 - AND i.is_hypothetical = 0'; - - IF @object_id IS NOT NULL - BEGIN - SELECT @sql += N' - AND t.object_id = @object_id'; - END; - - SELECT - @sql += N' - AND NOT EXISTS - ( - SELECT - 1/0 - FROM ' + QUOTENAME(@database_name) + N'.sys.objects AS so - WHERE i.object_id = so.object_id - AND so.is_ms_shipped = 0 - AND so.type = ''TF'' - ) - OPTION(RECOMPILE);'; - - IF @debug = 1 - BEGIN - PRINT @sql; - END; - - INSERT - #index_details - WITH - (TABLOCK) - ( - database_id, - object_id, - index_id, - schema_name, - table_name, - index_name, - column_name, - is_primary_key, - is_unique, - is_unique_constraint, - is_indexed_view, - is_foreign_key, - is_foreign_key_reference, - key_ordinal, - index_column_id, - is_descending_key, - is_included_column, - filter_definition, - is_max_length, - user_seeks, - user_scans, - user_lookups, - user_updates, - last_user_seek, - last_user_scan, - last_user_lookup, - last_user_update - ) - EXEC sys.sp_executesql - @sql, - N'@database_id integer, - @object_id integer', - @database_id, - @object_id; - - IF @debug = 1 - BEGIN - SELECT - table_name = '#index_details', - * - FROM #index_details AS id; - END; - - SELECT - @sql = N' - SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; - - SELECT - @sql += N' - SELECT - database_id = @database_id, - x.object_id, - x.index_id, - x.schema_name, - x.table_name, - x.index_name, - x.partition_id, - x.partition_number, - x.total_rows, - x.total_space_mb, - x.reserved_lob_mb, - x.reserved_row_overflow_mb, - x.data_compression_desc, - built_on = - ISNULL - ( - psfg.partition_scheme_name, - psfg.filegroup_name - ), - psfg.partition_function_name, - pc.partition_columns - FROM - ( - SELECT - ps.object_id, - ps.index_id, - schema_name = s.name, - table_name = t.name, - index_name = i.name, - ps.partition_id, - p.partition_number, - total_rows = SUM(ps.row_count), - total_space_mb = SUM(a.total_pages) * 8 / 1024.0, - reserved_lob_mb = SUM(ps.lob_reserved_page_count) * 8. / 1024., - reserved_row_overflow_mb = SUM(ps.row_overflow_reserved_page_count) * 8. / 1024., - p.data_compression_desc, - i.data_space_id - FROM ' + QUOTENAME(@database_name) + N'.sys.tables AS t - JOIN ' + QUOTENAME(@database_name) + N'.sys.indexes AS i - ON t.object_id = i.object_id - JOIN ' + QUOTENAME(@database_name) + N'.sys.schemas AS s - ON t.schema_id = s.schema_id - JOIN ' + QUOTENAME(@database_name) + N'.sys.partitions AS p - ON i.object_id = p.object_id - AND i.index_id = p.index_id - JOIN ' + QUOTENAME(@database_name) + N'.sys.allocation_units AS a - ON p.partition_id = a.container_id - LEFT HASH JOIN ' + QUOTENAME(@database_name) + N'.sys.dm_db_partition_stats AS ps - ON p.partition_id = ps.partition_id - WHERE t.type <> ''TF'' - AND i.type = 2'; - - IF @object_id IS NOT NULL - BEGIN - SELECT @sql += N' - AND t.object_id = @object_id'; - END; - - SELECT - @sql += N' - GROUP BY - t.name, - i.name, - i.data_space_id, - s.name, - p.partition_number, - p.data_compression_desc, - ps.object_id, - ps.index_id, - ps.partition_id - ) AS x - OUTER APPLY - ( - SELECT - filegroup_name = - fg.name, - partition_scheme_name = - ps.name, - partition_function_name = - pf.name - FROM ' + QUOTENAME(@database_name) + N'.sys.filegroups AS fg - FULL JOIN ' + QUOTENAME(@database_name) + N'.sys.partition_schemes AS ps - ON ps.data_space_id = fg.data_space_id - LEFT JOIN ' + QUOTENAME(@database_name) + N'.sys.partition_functions AS pf - ON pf.function_id = ps.function_id - WHERE x.data_space_id = fg.data_space_id - OR x.data_space_id = ps.data_space_id - ) AS psfg - OUTER APPLY - ( - SELECT - partition_columns = - STUFF - ( - ( - SELECT - N'', '' + - c.name - FROM ' + QUOTENAME(@database_name) + N'.sys.index_columns AS ic - JOIN ' + QUOTENAME(@database_name) + N'.sys.columns AS c - ON c.object_id = ic.object_id - AND c.column_id = ic.column_id - WHERE ic.object_id = x.object_id - AND ic.index_id = x.index_id - AND ic.partition_ordinal > 0 - ORDER BY - ic.partition_ordinal - FOR XML - PATH(''''), - TYPE - ).value(''.'', ''nvarchar(MAX)''), - 1, - 2, - '''' - ) - ) AS pc - OPTION(RECOMPILE);'; - - IF @debug = 1 - BEGIN - PRINT @sql; - END; - - INSERT - #partition_stats WITH(TABLOCK) - ( - database_id, - object_id, - index_id, - schema_name, - table_name, - index_name, - partition_id, - partition_number, - total_rows, - total_space_mb, - reserved_lob_mb, - reserved_row_overflow_mb, - data_compression_desc, - built_on, - partition_function_name, - partition_columns - ) - EXEC sys.sp_executesql - @sql, - N'@database_id integer, - @object_id integer', - @database_id, - @object_id; - - IF @debug = 1 - BEGIN - SELECT - table_name = '#partition_stats', - * - FROM #partition_stats AS ps; - END; - - INSERT INTO - #index_analysis - WITH - (TABLOCK) - ( - database_id, - schema_name, - table_name, - index_name, - is_unique, - key_columns, - included_columns, - filter_definition - ) - SELECT - @database_id, - id1.schema_name, - id1.table_name, - id1.index_name, - id1.is_unique, - key_columns = - STUFF - ( - ( - SELECT - N', ' + - id2.column_name + - CASE - WHEN id2.is_descending_key = 1 - THEN N' DESC' - ELSE N'' - END - FROM #index_details id2 - WHERE id2.object_id = id1.object_id - AND id2.index_id = id1.index_id - AND id2.is_included_column = 0 - ORDER BY - id2.key_ordinal - FOR XML - PATH(''), - TYPE - ).value('text()[1]','nvarchar(max)'), - 1, - 2, - '' - ), - included_columns = - STUFF - ( - ( - SELECT - N', ' + - id2.column_name - FROM #index_details id2 - WHERE id2.object_id = id1.object_id - AND id2.index_id = id1.index_id - AND id2.is_included_column = 1 - ORDER BY - id2.column_name - FOR XML - PATH(''), - TYPE - ).value('text()[1]','nvarchar(max)'), - 1, - 2, - '' - ), - id1.filter_definition - FROM #index_details id1 - GROUP BY - id1.schema_name, - id1.table_name, - id1.index_name, - id1.is_unique, - id1.object_id, - id1.index_id, - id1.filter_definition - OPTION(RECOMPILE); - - IF @debug = 1 - BEGIN - SELECT - table_name = '#index_analysis', - ia.* - FROM #index_analysis AS ia; - END; - - /*Analyze indexes*/ - DECLARE - @index_cursor CURSOR; - - SET @index_cursor = CURSOR - LOCAL - STATIC - FORWARD_ONLY - READ_ONLY - FOR - SELECT DISTINCT - ia.database_id, - ia.schema_name, - ia.table_name, - ia.index_name, - ia.is_unique, - ia.filter_definition - FROM #index_analysis AS ia - ORDER BY - ia.table_name, - ia.index_name; - - OPEN @index_cursor; - - FETCH NEXT - FROM @index_cursor - INTO - @c_database_id, - @c_schema_name, - @c_table_name, - @c_index_name, - @c_is_unique, - @c_filter_definition; - - WHILE @@FETCH_STATUS = 0 - BEGIN - WITH - IndexColumns AS - ( - SELECT - id.database_id, - id.schema_name, - id.table_name, - id.index_name, - id.column_name, - id.is_included_column, - id.key_ordinal - FROM #index_details id - WHERE id.database_id = @c_database_id - AND id.schema_name = @c_schema_name - AND id.table_name = @c_table_name - ), - CurrentIndexColumns AS - ( - SELECT - ic.* - FROM IndexColumns AS ic - WHERE ic.index_name = @c_index_name - ), - OtherIndexColumns AS - ( - SELECT - ic.* - FROM IndexColumns AS ic - WHERE ic.index_name <> @c_index_name - ) - UPDATE - ia - SET - ia.is_redundant = - CASE - WHEN NOT EXISTS - ( - SELECT - 1/0 - FROM CurrentIndexColumns cic - WHERE cic.is_included_column = 0 -- Only check key columns - AND NOT EXISTS - ( - SELECT - 1/0 - FROM OtherIndexColumns oic - WHERE oic.column_name = cic.column_name - AND oic.is_included_column = 0 -- Must be in key columns - AND oic.key_ordinal <= cic.key_ordinal -- Check leading edge - ) - ) - AND - ( - -- Check included columns separately since order doesn't matter - NOT EXISTS - ( - SELECT - 1/0 - FROM CurrentIndexColumns cic - WHERE cic.is_included_column = 1 - AND NOT EXISTS - ( - SELECT - 1/0 - FROM OtherIndexColumns oic - WHERE oic.column_name = cic.column_name - AND - ( - oic.is_included_column = 1 - OR oic.is_included_column = 0 -- Include cols can be covered by key cols - ) - ) - ) - ) - AND ISNULL(ia.filter_definition, '') = ISNULL(@c_filter_definition, '') - AND - ( - ia.is_unique = 0 - OR @c_is_unique = 1 - ) - THEN 1 - ELSE 0 - END, - ia.superseded_by = - CASE - WHEN NOT EXISTS - ( - SELECT - 1/0 - FROM CurrentIndexColumns cic - WHERE cic.is_included_column = 0 -- Only check key columns - AND NOT EXISTS - ( - SELECT - 1/0 - FROM OtherIndexColumns oic - WHERE oic.column_name = cic.column_name - AND oic.is_included_column = 0 -- Must be in key columns - AND oic.key_ordinal <= cic.key_ordinal -- Check leading edge - ) - ) - AND - ( - -- Check included columns separately since order doesn't matter - NOT EXISTS - ( - SELECT - 1/0 - FROM CurrentIndexColumns cic - WHERE cic.is_included_column = 1 - AND NOT EXISTS - ( - SELECT - 1/0 - FROM OtherIndexColumns oic - WHERE oic.column_name = cic.column_name - AND - ( - oic.is_included_column = 1 - OR oic.is_included_column = 0 -- Include cols can be covered by key cols - ) - ) - ) - ) - AND ISNULL(ia.filter_definition, '') = ISNULL(@c_filter_definition, '') - AND - ( - ia.is_unique = 0 - OR @c_is_unique = 1 - ) - THEN @c_index_name - ELSE ia.superseded_by - END, - ia.missing_columns = - STUFF - ( - ( - SELECT - N', ' + - oic.column_name - FROM OtherIndexColumns oic - WHERE NOT EXISTS - ( - SELECT - 1/0 - FROM CurrentIndexColumns cic - WHERE cic.column_name = oic.column_name - ) - FOR XML - PATH(''), - TYPE - ).value('.', 'nvarchar(MAX)'), - 1, - 2, - '' - ) - FROM #index_analysis ia - WHERE ia.database_id = @c_database_id - AND ia.schema_name = @c_schema_name - AND ia.table_name = @c_table_name - AND ia.index_name <> @c_index_name; - - FETCH NEXT - FROM @index_cursor - INTO - @c_database_id, - @c_schema_name, - @c_table_name, - @c_index_name, - @c_is_unique, - @c_filter_definition; - END; - - /*Determine actions*/ - UPDATE - #index_analysis - SET - action = - CASE - WHEN is_redundant = 1 - THEN N'DROP' - WHEN superseded_by IS NOT NULL - AND missing_columns IS NULL - THEN N'MERGE INTO ' + - superseded_by - WHEN superseded_by IS NOT NULL - AND missing_columns IS NOT NULL - THEN N'MERGE INTO ' + - superseded_by + - N' (ADD ' + - missing_columns + - N')' - ELSE N'KEEP' - END; - - IF @debug = 1 - BEGIN - SELECT - table_name = '#index_analysis', - ia.* - FROM #index_analysis AS ia; - END; - - INSERT INTO - #index_cleanup_report - WITH - (TABLOCK) - ( - database_name, - table_name, - index_name, - action, - cleanup_script, - original_definition, - user_seeks, - user_scans, - user_lookups, - user_updates, - last_user_seek, - last_user_scan, - last_user_lookup, - last_user_update, - range_scan_count, - singleton_lookup_count, - leaf_insert_count, - leaf_update_count, - leaf_delete_count, - page_lock_count, - page_lock_wait_count, - page_lock_wait_in_ms - ) - SELECT - @database_name, - ia.table_name, - ia.index_name, - ia.action, - cleanup_script = - CASE - WHEN ia.action = N'DROP' - THEN N'DROP INDEX ' + - QUOTENAME(ia.index_name) + - N' ON ' + - QUOTENAME(ia.table_name) + - N';' - WHEN ia.action LIKE N'MERGE INTO%' - THEN N'CREATE ' + - CASE - WHEN ia.is_unique = 1 - THEN N'UNIQUE ' - ELSE N'' - END + - N'INDEX ' + - QUOTENAME(ia.superseded_by) + - N' ON ' + - QUOTENAME(ia.table_name) + - N'(' + - ISNULL(superseding.key_columns, ia.key_columns) + - N')' + - CASE - WHEN ISNULL(superseding.included_columns, ia.included_columns) IS NOT NULL - OR ia.missing_columns IS NOT NULL - THEN N' INCLUDE (' + - STUFF(( - SELECT DISTINCT N',' + c.value('.', 'nvarchar(128)') - FROM ( - SELECT CAST(N'' + - REPLACE(ISNULL(superseding.included_columns, ia.included_columns), N', ', N'') - + N'' AS xml) - ) AS x(c) - FOR XML PATH('') - ), 1, 1, '') + - CASE - WHEN ia.missing_columns IS NOT NULL - THEN N', ' + ia.missing_columns - ELSE N'' - END + - N')' - ELSE N'' - END + - CASE - WHEN ps.partition_function_name IS NOT NULL - THEN N' ON ' + - QUOTENAME(ps.partition_function_name) + - N'(' + - ps.partition_columns + - N')' - ELSE N'' - END + - CASE - WHEN ia.filter_definition IS NOT NULL - THEN N' WHERE ' + - ia.filter_definition - ELSE N'' - END + - N' WITH (DROP_EXISTING = ON' + - CASE - WHEN ps.data_compression_desc <> N'NONE' - THEN N', DATA_COMPRESSION = ' + - ps.data_compression_desc - ELSE N'' - END + - N');' + - NCHAR(13) + NCHAR(10) + - N'ALTER INDEX ' + - QUOTENAME(ia.index_name) + - N' ON ' + - QUOTENAME(ia.table_name) + - N' DISABLE;' - ELSE N'' - END, - original_definition = - N'CREATE ' + - CASE - WHEN ia.is_unique = 1 - THEN N'UNIQUE ' - ELSE N'' - END + - N'INDEX ' + - QUOTENAME(ia.index_name) + - N' ON ' + - QUOTENAME(ia.table_name) + - N'(' + - ia.key_columns + - N')' + - CASE - WHEN ia.included_columns IS NOT NULL - THEN N' INCLUDE (' + - ia.included_columns + - N')' - ELSE N'' - END + - CASE - WHEN ps.partition_function_name IS NOT NULL - THEN N' ON ' + - QUOTENAME(ps.partition_function_name) + - N'(' + - ps.partition_columns + - N')' - ELSE N'' - END + - CASE - WHEN ia.filter_definition IS NOT NULL - THEN N' WHERE ' + - ia.filter_definition - ELSE N'' - END, - id.user_seeks, - id.user_scans, - id.user_lookups, - id.user_updates, - id.last_user_seek, - id.last_user_scan, - id.last_user_lookup, - id.last_user_update, - os.range_scan_count, - os.singleton_lookup_count, - os.leaf_insert_count, - os.leaf_update_count, - os.leaf_delete_count, - os.page_lock_count, - os.page_lock_wait_count, - os.page_lock_wait_in_ms - FROM #index_analysis ia - LEFT JOIN #partition_stats ps - ON ia.table_name = ps.table_name - AND ia.index_name = ps.index_name - LEFT JOIN #index_details id - ON ia.table_name = id.table_name - AND ia.index_name = id.index_name - LEFT JOIN #operational_stats os - ON id.object_id = os.object_id - AND id.index_id = os.index_id - LEFT JOIN #index_analysis superseding - ON ia.superseded_by = superseding.index_name - AND ia.table_name = superseding.table_name; - - IF @debug = 1 - BEGIN - SELECT - table_name = '#index_cleanup_report', - icr.* - FROM #index_cleanup_report AS icr; - END; - - INSERT INTO - #index_cleanup_summary - WITH - (TABLOCK) - ( - database_name, - table_name, - index_name, - action, - details, - current_definition, - proposed_definition, - usage_summary, - operational_summary - ) - SELECT - icr.database_name, - icr.table_name, - icr.index_name, - action = - CASE - WHEN icr.action = N'KEEP' - THEN N'Keep' - WHEN icr.action = N'DROP' - THEN N'Drop' - WHEN icr.action LIKE N'MERGE INTO%' - THEN N'Merge' - ELSE N'???' - END, - details = - CASE - WHEN icr.action = N'KEEP' - THEN N'No action needed' - WHEN icr.action = N'DROP' - THEN N'Index is redundant and can be safely dropped' - WHEN icr.action LIKE N'MERGE INTO%' - THEN N'Merge into index: ' + - SUBSTRING - ( - icr.action, - 12, - CHARINDEX(N' ', icr.action, 12) - 12 - ) - ELSE N'???' - END, - current_definition = icr.original_definition, - proposed_definition = - CASE - WHEN icr.action LIKE N'MERGE INTO%' - THEN icr.cleanup_script - ELSE NULL - END, - usage_summary = - N'Seeks: ' + CONVERT(nvarchar(20), icr.user_seeks) + - N', Scans: ' + CONVERT(nvarchar(20), icr.user_scans) + - N', Lookups: ' + CONVERT(nvarchar(20), icr.user_lookups) + - N', Updates: ' + CONVERT(nvarchar(20), icr.user_updates) + - N', Last used: ' + - ISNULL - ( - CONVERT - ( - nvarchar(30), - NULLIF - ( - DATEADD - ( - SECOND, - -1, - CASE - WHEN icr.last_user_seek > icr.last_user_scan - AND icr.last_user_seek > icr.last_user_lookup - THEN icr.last_user_seek - WHEN icr.last_user_scan > icr.last_user_lookup - THEN icr.last_user_scan - ELSE icr.last_user_lookup - END - ), - N'1900-01-01' - ), 120 - ), - N'Unknown' - ), - operational_summary = - N'Range scans: ' + CONVERT(nvarchar(20), icr.range_scan_count) + - N', Lookups: ' + CONVERT(nvarchar(20), icr.singleton_lookup_count) + - N', Inserts: ' + CONVERT(nvarchar(20), icr.leaf_insert_count) + - N', Updates: ' + CONVERT(nvarchar(20), icr.leaf_update_count) + - N', Deletes: ' + CONVERT(nvarchar(20), icr.leaf_delete_count) - FROM #index_cleanup_report AS icr; - - IF @debug = 1 - BEGIN - SELECT - table_name = '#index_cleanup_summary', - ics.* - FROM #index_cleanup_summary AS ics; - END; - - SELECT - ics.database_name, - ics.table_name, - ics.index_name, - ics.action, - ics.details, - ics.current_definition, - ics.proposed_definition, - ics.usage_summary, - ics.operational_summary - FROM #index_cleanup_summary AS ics - ORDER BY - CASE ics.action - WHEN N'Drop' THEN 1 - WHEN N'Merge' THEN 2 - WHEN N'Keep' THEN 3 - ELSE 999 - END, - ics.table_name, - ics.index_name; - - WITH - IndexActions AS - ( - SELECT - icr.database_name, - icr.table_name, - icr.index_name, - icr.action, - icr.cleanup_script, - n = ROW_NUMBER() OVER - ( - PARTITION BY - icr.table_name, - icr.index_name - ORDER BY - CASE - WHEN icr.action LIKE N'MERGE INTO%' - THEN 1 - WHEN icr.action = N'DROP' - THEN 2 - ELSE 3 - END - ) - FROM #index_cleanup_report icr - ) - INSERT INTO - #final_index_actions - WITH - (TABLOCK) - ( - database_name, - table_name, - index_name, - action, - script - ) - SELECT - database_name, - table_name, - index_name, - action, - CASE - WHEN action LIKE N'MERGE INTO%' - THEN cleanup_script - WHEN action = N'DROP' - THEN N'ALTER INDEX ' + - QUOTENAME(index_name) + - N' ON ' + - QUOTENAME(table_name) + - N' DISABLE;' - ELSE N'???' - END AS script - FROM IndexActions - WHERE n = 1; - - SELECT - f.database_name, - f.table_name, - f.index_name, - f.action, - f.script, - sort_order = - CASE action - WHEN N'MERGE INTO' THEN 2 - WHEN N'DROP' THEN 3 - ELSE 999 - END - FROM #final_index_actions AS f - WHERE f.action <> N'KEEP' - - UNION ALL - - SELECT - r.database_name, - r.table_name, - r.index_name, - action = - N'DISABLE (Unused)', - script = - N'ALTER INDEX ' + - QUOTENAME(r.index_name) + - N' ON ' + - QUOTENAME(r.table_name) + - N' DISABLE;', - sort_order = 1 - FROM #index_cleanup_report AS r - WHERE r.user_seeks = 0 - AND r.user_scans = 0 - AND r.user_lookups = 0 - AND r.user_updates = 0 - ORDER BY - table_name, - index_name, - sort_order; - - SELECT - @final_script += f.script + NCHAR(13) + NCHAR(10) - FROM #final_index_actions AS f - WHERE f.action LIKE N'MERGE INTO%' - ORDER BY - f.table_name, - f.index_name; - - SELECT - @final_script += f.script + NCHAR(13) + NCHAR(10) - FROM #final_index_actions AS f - WHERE f.action IN - ( - N'DROP', - N'MERGE INTO' - ) - ORDER BY - f.table_name, - f.index_name; - - SELECT - @final_script += - N'ALTER INDEX ' + - QUOTENAME(i.index_name) + - N' ON ' + - QUOTENAME(i.table_name) + - N' DISABLE;' + - NCHAR(13) + NCHAR(10) - FROM #index_cleanup_report AS i - WHERE i.user_seeks = 0 - AND i.user_scans = 0 - AND i.user_lookups = 0 - AND i.user_updates = 0 - ORDER BY - i.table_name, - i.index_name; - - PRINT N'----------------------'; - PRINT N'Final script to review. DO NOT EXECUTE WITHOUT CAREFUL REVIEW.'; - PRINT N'Implementation Script:'; - PRINT N'----------------------'; - SELECT - @sql_len = LEN(@final_script); - - IF @sql_len < 4000 - BEGIN - PRINT @sql; - END - ELSE - BEGIN - WHILE @helper <= @sql_len - BEGIN - SELECT - @sql_debug = - SUBSTRING(@final_script, @helper + 1, 2000) + NCHAR(13) + NCHAR(10); - - PRINT @sql_debug; - SET @helper += 2000; - END; - END; - -END TRY -BEGIN CATCH - PRINT N'Error occurred: ' + ERROR_MESSAGE(); -END CATCH; -END; /*Final End*/ -GO diff --git a/sp_IndexCleanup/README.md b/sp_IndexCleanup/README.md new file mode 100644 index 00000000..fd781644 --- /dev/null +++ b/sp_IndexCleanup/README.md @@ -0,0 +1,90 @@ + + +# sp_IndexCleanup + +## Overview + +This stored procedure helps identify unused and duplicate indexes in your SQL Server databases that could be candidates for removal. It analyzes index usage statistics and can generate scripts for removing unnecessary indexes. + +**IMPORTANT: This is currently a BETA VERSION.** It needs extensive testing in real environments with real indexes to address several issues: +* Data collection accuracy +* Deduping logic +* Result correctness +* Edge cases + +## Warning + +Misuse of this procedure can potentially harm your database. If you run this, only use the output to validate result correctness. **Do not run any of the output scripts without thorough review and testing**, as doing so may be harmful to your database performance. + +The procedure requires SQL Server 2012 (11.0) or later due to the use of FORMAT and CONCAT functions. + +## Parameters + +| Parameter Name | Data Type | Default Value | Description | +|----------------|-----------|---------------|-------------| +| @database_name | sysname | NULL | The name of the database you wish to analyze | +| @schema_name | sysname | NULL | The schema name to filter indexes by | +| @table_name | sysname | NULL | The table name to filter indexes by | +| @min_reads | bigint | 0 | Minimum number of reads for an index to be considered used | +| @min_writes | bigint | 0 | Minimum number of writes for an index to be considered used | +| @min_size_gb | decimal(10,2) | 0 | Minimum size in GB for an index to be analyzed | +| @min_rows | bigint | 0 | Minimum number of rows for a table to be analyzed | +| @get_all_databases | bit | 0 | When set to 1, analyzes all eligible databases on the server | +| @include_databases | nvarchar(max) | NULL | Comma-separated list of databases to include (used with @get_all_databases = 1) | +| @exclude_databases | nvarchar(max) | NULL | Comma-separated list of databases to exclude (used with @get_all_databases = 1) | +| @help | bit | 0 | Displays help information | +| @debug | bit | 0 | Prints debug information during execution | +| @version | varchar(20) | NULL | OUTPUT parameter that returns the version number of the procedure | +| @version_date | datetime | NULL | OUTPUT parameter that returns the date this version was released | + +## Usage Examples + +```sql +-- Basic usage to analyze all indexes in a database +EXECUTE dbo.sp_IndexCleanup + @database_name = 'YourDatabase'; + +-- Analyze a specific table with debug information +EXECUTE dbo.sp_IndexCleanup + @database_name = 'YourDatabase', + @table_name = 'YourTable', + @debug = 1; + +-- Filter indexes by minimum usage thresholds +EXECUTE dbo.sp_IndexCleanup + @database_name = 'YourDatabase', + @min_reads = 100, + @min_writes = 10; + +-- Analyze all user databases on the server +EXECUTE dbo.sp_IndexCleanup + @get_all_databases = 1, + @debug = 1; + +-- Analyze only specific databases +EXECUTE dbo.sp_IndexCleanup + @get_all_databases = 1, + @include_databases = 'Database1,Database2,Database3'; + +-- Analyze all databases except specific ones +EXECUTE dbo.sp_IndexCleanup + @get_all_databases = 1, + @exclude_databases = 'ReportServer,TempDB2'; + +-- Show help information +EXECUTE dbo.sp_IndexCleanup + @help = 1; +``` + +## Notes + +- The procedure issues a warning when server uptime is less than 14 days, as index usage stats may not be representative +- Certain features like online index operations and compression are only available in specific SQL Server editions (Enterprise, Azure SQL DB, Managed Instance) +- It is recommended to have a recent backup before making any index changes +- The multi-database processing feature (@get_all_databases) analyzes each database sequentially for better performance and resource management +- System databases (master, model, msdb, tempdb, rdsadmin) are always excluded from processing +- When using @get_all_databases, results for all databases are combined in a single result set +- The index_count column for the SUMMARY row in the output table will likely indicate a lower number than is shown at the DATABASE level. The SUMMARY level only includes indexes that have been analyzed; excluding things like clustered indexes, heaps, xml indexes, etc. The DATABASE level index_count value is the total number of indexes in the database. + +Copyright 2024 Darling Data, LLC +Released under MIT license \ No newline at end of file diff --git a/sp_IndexCleanup/sp_IndexCleanup.sql b/sp_IndexCleanup/sp_IndexCleanup.sql new file mode 100644 index 00000000..41bc6d28 --- /dev/null +++ b/sp_IndexCleanup/sp_IndexCleanup.sql @@ -0,0 +1,5769 @@ +SET ANSI_WARNINGS ON; +SET ARITHABORT ON; +SET CONCAT_NULL_YIELDS_NULL ON; +SET QUOTED_IDENTIFIER ON; +SET NUMERIC_ROUNDABORT OFF; +SET IMPLICIT_TRANSACTIONS OFF; +SET STATISTICS TIME, IO OFF; +GO + +IF OBJECT_ID('dbo.sp_IndexCleanup', 'P') IS NULL +BEGIN + EXECUTE ('CREATE PROCEDURE dbo.sp_IndexCleanup AS RETURN 138;'); +END; +GO + +ALTER PROCEDURE + dbo.sp_IndexCleanup +( + @database_name sysname = NULL, + @schema_name sysname = NULL, + @table_name sysname = NULL, + @min_reads bigint = 0, + @min_writes bigint = 0, + @min_size_gb decimal(10,2) = 0, + @min_rows bigint = 0, + @get_all_databases bit = 0, /*looks for all accessible user databases and returns combined results*/ + @include_databases nvarchar(max) = NULL, /*comma-separated list of databases to include (only when @get_all_databases = 1)*/ + @exclude_databases nvarchar(max) = NULL, /*comma-separated list of databases to exclude (only when @get_all_databases = 1)*/ + @help bit = 'false', + @debug bit = 'false', + @version varchar(20) = NULL OUTPUT, + @version_date datetime = NULL OUTPUT +) +WITH RECOMPILE +AS +BEGIN +SET NOCOUNT ON; + +BEGIN TRY + /* Check for SQL Server 2012 (11.0) or later for FORMAT and CONCAT functions*/ + + IF + /* Check SQL Server 2012+ for FORMAT and CONCAT functions */ + ( + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) NOT IN (5, 8) /* Not Azure SQL DB or Managed Instance */ + AND CONVERT + ( + integer, + SUBSTRING + ( + CONVERT + ( + varchar(20), + SERVERPROPERTY('ProductVersion') + ), + 1, + 2 + ) + ) < 11) /* Pre-2012 */ + BEGIN + RAISERROR('This procedure requires SQL Server 2012 (11.0) or later due to the use of FORMAT and CONCAT functions.', 11, 1); + RETURN; + END; + + SELECT + @version = '1.4', + @version_date = '20250401'; + + /* + Help section, for help. + Will become more helpful when out of beta. + */ + IF @help = 1 + BEGIN + SELECT + help = N'hello, i am sp_IndexCleanup' + UNION ALL + SELECT + help = N'this is a script to help clean up unused and duplicate indexes.' + UNION ALL + SELECT + help = N'it will also help you add page compression to uncompressed indexes.' + UNION ALL + SELECT + help = N'always validate all changes against a non-production environment!' + UNION ALL + SELECT + help = N'please test carefully.'; + + /* + Parameters + */ + SELECT + parameter_name = + ap.name, + data_type = + t.name, + description = + CASE + ap.name + WHEN N'@database_name' THEN 'the name of the database you wish to analyze' + WHEN N'@schema_name' THEN 'the schema name to filter indexes by' + WHEN N'@table_name' THEN 'the table name to filter indexes by' + WHEN N'@min_reads' THEN 'minimum number of reads for an index to be considered used' + WHEN N'@min_writes' THEN 'minimum number of writes for an index to be considered used' + WHEN N'@min_size_gb' THEN 'minimum size in GB for an index to be analyzed' + WHEN N'@min_rows' THEN 'minimum number of rows for a table to be analyzed' + WHEN N'@get_all_databases' THEN 'set to 1 to analyze all accessible user databases' + WHEN N'@include_databases' THEN 'comma-separated list of databases to include when @get_all_databases = 1' + WHEN N'@exclude_databases' THEN 'comma-separated list of databases to exclude when @get_all_databases = 1' + WHEN N'@help' THEN 'displays this help information' + WHEN N'@debug' THEN 'prints debug information during execution' + WHEN N'@version' THEN 'returns the version number of the procedure' + WHEN N'@version_date' THEN 'returns the date this version was released' + ELSE NULL + END, + valid_inputs = + CASE + ap.name + WHEN N'@database_name' THEN 'the name of a database you care about indexes in' + WHEN N'@schema_name' THEN 'schema name or NULL for all schemas' + WHEN N'@table_name' THEN 'table name or NULL for all tables' + WHEN N'@min_reads' THEN 'any positive integer or 0' + WHEN N'@min_writes' THEN 'any positive integer or 0' + WHEN N'@min_size_gb' THEN 'any positive decimal or 0' + WHEN N'@min_rows' THEN 'any positive integer or 0' + WHEN N'@get_all_databases' THEN '0 or 1' + WHEN N'@include_databases' THEN 'comma-separated list of database names' + WHEN N'@exclude_databases' THEN 'comma-separated list of database names' + WHEN N'@help' THEN '0 or 1' + WHEN N'@debug' THEN '0 or 1' + WHEN N'@version' THEN 'OUTPUT parameter' + WHEN N'@version_date' THEN 'OUTPUT parameter' + ELSE NULL + END, + defaults = + CASE + ap.name + WHEN N'@database_name' THEN 'NULL' + WHEN N'@schema_name' THEN 'NULL' + WHEN N'@table_name' THEN 'NULL' + WHEN N'@min_reads' THEN '0' + WHEN N'@min_writes' THEN '0' + WHEN N'@min_size_gb' THEN '0' + WHEN N'@min_rows' THEN '0' + WHEN N'@get_all_databases' THEN '0' + WHEN N'@include_databases' THEN 'NULL' + WHEN N'@exclude_databases' THEN 'NULL' + WHEN N'@help' THEN 'false' + WHEN N'@debug' THEN 'true' + WHEN N'@version' THEN 'NULL' + WHEN N'@version_date' THEN 'NULL' + ELSE NULL + END + FROM sys.all_parameters AS ap + JOIN sys.all_objects AS o + ON ap.object_id = o.object_id + JOIN sys.types AS t + ON ap.system_type_id = t.system_type_id + AND ap.user_type_id = t.user_type_id + WHERE o.name = N'sp_IndexCleanup' + OPTION(MAXDOP 1, RECOMPILE); + + SELECT + mit_license_yo = 'i am MIT licensed, so like, do whatever' + + UNION ALL + + SELECT + mit_license_yo = 'see printed messages for full license'; + + RAISERROR(' +MIT License + +Copyright 2025 Darling Data, LLC + +https://www.erikdarling.com/ + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, +sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the +following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE +FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +', 0, 1) WITH NOWAIT; + + RETURN; + END; + + IF @debug = 1 + BEGIN + RAISERROR('Declaring variables', 0, 0) WITH NOWAIT; + END; + + DECLARE + /*general script variables*/ + @sql nvarchar(max) = N'', + @object_id integer = NULL, + @full_object_name nvarchar(768) = NULL, + @uptime_warning bit = 0, /* Will set after @uptime_days is calculated */ + /*print variables*/ + @online bit = + CASE + WHEN + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) IN (3, 5, 8) + THEN 'true' /* Enterprise, Azure SQL DB, Managed Instance */ + ELSE 'false' + END, + /* Compression variables */ + @can_compress bit = + CASE + WHEN + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) IN (3, 5, 8) + OR + ( + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) = 2 + AND CONVERT + ( + integer, + SUBSTRING + ( + CONVERT + ( + varchar(20), + SERVERPROPERTY('ProductVersion') + ), + 1, + 2 + ) + ) >= 13 + ) + THEN 1 + ELSE 0 + END, + @uptime_days nvarchar(10) = + ( + SELECT + DATEDIFF + ( + DAY, + osi.sqlserver_start_time, + SYSDATETIME() + ) + FROM sys.dm_os_sys_info AS osi + ), + @database_cursor CURSOR, + @current_database_name sysname, + @current_database_id integer, + @error_msg nvarchar(2048), + @conflict_list nvarchar(max) = N'', + @rc bigint; + + /* Set uptime warning flag after @uptime_days is calculated */ + SELECT + @uptime_warning = + CASE + WHEN CONVERT(integer, @uptime_days) < 14 + THEN 1 + ELSE 0 + END; + + /* + Initial checks for object validity + */ + IF @debug = 1 + BEGIN + RAISERROR('Checking paramaters...', 0, 0) WITH NOWAIT; + END; + + IF @schema_name IS NULL + AND @table_name IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @schema_name cannot be NULL when specifying a table, defaulting to dbo', 10, 1) WITH NOWAIT; + END; + + SELECT + @schema_name = N'dbo'; + END; + + IF @min_reads < 0 + OR @min_reads IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_reads cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_reads = 0; + END; + + IF @min_writes < 0 + OR @min_writes IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_writes cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_writes = 0; + END; + + IF @min_size_gb < 0 + OR @min_size_gb IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_size_gb cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_size_gb = 0; + END; + + IF @min_rows < 0 + OR @min_rows IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_rows cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_rows = 0; + END; + + /* + Temp tables! + */ + + IF @debug = 1 + BEGIN + RAISERROR('Creating temp tables', 0, 0) WITH NOWAIT; + END; + + CREATE TABLE + #filtered_objects + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + can_compress bit NOT NULL + PRIMARY KEY CLUSTERED(database_id, schema_id, object_id, index_id) + ); + + CREATE TABLE + #operational_stats + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + range_scan_count bigint NULL, + singleton_lookup_count bigint NULL, + forwarded_fetch_count bigint NULL, + lob_fetch_in_pages bigint NULL, + row_overflow_fetch_in_pages bigint NULL, + leaf_insert_count bigint NULL, + leaf_update_count bigint NULL, + leaf_delete_count bigint NULL, + leaf_ghost_count bigint NULL, + nonleaf_insert_count bigint NULL, + nonleaf_update_count bigint NULL, + nonleaf_delete_count bigint NULL, + leaf_allocation_count bigint NULL, + nonleaf_allocation_count bigint NULL, + row_lock_count bigint NULL, + row_lock_wait_count bigint NULL, + row_lock_wait_in_ms bigint NULL, + page_lock_count bigint NULL, + page_lock_wait_count bigint NULL, + page_lock_wait_in_ms bigint NULL, + index_lock_promotion_attempt_count bigint NULL, + index_lock_promotion_count bigint NULL, + page_latch_wait_count bigint NULL, + page_latch_wait_in_ms bigint NULL, + tree_page_latch_wait_count bigint NULL, + tree_page_latch_wait_in_ms bigint NULL, + page_io_latch_wait_count bigint NULL, + page_io_latch_wait_in_ms bigint NULL, + page_compression_attempt_count bigint NULL, + page_compression_success_count bigint NULL, + PRIMARY KEY CLUSTERED (database_id, schema_id, object_id, index_id) + ); + + CREATE TABLE + #partition_stats + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NULL, + partition_id bigint NOT NULL, + partition_number integer NOT NULL, + total_rows bigint NULL, + total_space_gb decimal(38, 4) NULL, /* Using 4 decimal places for GB to maintain precision */ + reserved_lob_gb decimal(38, 4) NULL, /* Using 4 decimal places for GB to maintain precision */ + reserved_row_overflow_gb decimal(38, 4) NULL, /* Using 4 decimal places for GB to maintain precision */ + data_compression_desc nvarchar(60) NULL, + built_on sysname NULL, + partition_function_name sysname NULL, + partition_columns nvarchar(max) + PRIMARY KEY CLUSTERED(database_id, schema_id, object_id, index_id, partition_id) + ); + + CREATE TABLE + #index_details + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NULL, + column_name sysname NOT NULL, + column_id int NOT NULL, + is_primary_key bit NULL, + is_unique bit NULL, + is_unique_constraint bit NULL, + is_indexed_view integer NOT NULL, + is_foreign_key bit NULL, + is_foreign_key_reference bit NULL, + key_ordinal tinyint NOT NULL, + index_column_id integer NOT NULL, + is_descending_key bit NOT NULL, + is_included_column bit NULL, + filter_definition nvarchar(max) NULL, + is_max_length integer NOT NULL, + user_seeks bigint NOT NULL, + user_scans bigint NOT NULL, + user_lookups bigint NOT NULL, + user_updates bigint NOT NULL, + last_user_seek datetime NULL, + last_user_scan datetime NULL, + last_user_lookup datetime NULL, + last_user_update datetime NULL, + is_eligible_for_dedupe bit NOT NULL + PRIMARY KEY CLUSTERED(database_id, schema_id, object_id, index_id, column_id) + ); + + CREATE TABLE + #index_analysis + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + is_unique bit NULL, + key_columns nvarchar(MAX) NULL, + included_columns nvarchar(MAX) NULL, + filter_definition nvarchar(MAX) NULL, + /* Query plan for original CREATE INDEX statement */ + original_index_definition nvarchar(MAX) NULL, + /* + Consolidation rule that matched (e.g., Key Duplicate, Key Subset, etc) + For exact duplicates, use one of: Exact Duplicate, Reverse Duplicate, or Equal Except For Filter + */ + consolidation_rule nvarchar(256) NULL, + /* + Action to take (e.g., DISABLE, MERGE INCLUDES, KEEP) + If NULL, no action to be taken + */ + action nvarchar(100) NULL, + /* Target index to merge with or use instead of this one */ + target_index_name sysname NULL, + /* When this is a target, the index which points to it as a supersedes in consolidation */ + superseded_by nvarchar(4000) NULL, + /* Priority score from 0-1 to determine which index to keep (higher is better) */ + index_priority decimal(10,6) NULL + PRIMARY KEY CLUSTERED(database_id, schema_id, object_id, index_id) + ); + + CREATE TABLE + #compression_eligibility + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + can_compress bit NOT NULL, + reason nvarchar(200) NULL, + PRIMARY KEY CLUSTERED(database_id, schema_id, object_id, index_id) + ); + + CREATE TABLE + #index_cleanup_results + ( + result_type varchar(100) NOT NULL, + sort_order integer NOT NULL, + database_name sysname NULL, + schema_name sysname NULL, + table_name sysname NULL, + index_name sysname NULL, + script_type nvarchar(60) NULL, /* Type of script (e.g., MERGE SCRIPT, DISABLE SCRIPT, etc.) */ + consolidation_rule nvarchar(256) NULL, /* Reason for action (e.g., Exact Duplicate, Key Subset) */ + target_index_name sysname NULL, /* If this index is a duplicate, indicates which index is the preferred one */ + superseded_info nvarchar(4000) NULL, /* If this is a kept index, indicates which indexes it supersedes */ + additional_info nvarchar(max) NULL, /* Additional information about the action */ + original_index_definition nvarchar(max) NULL, /* Original statement to create the index */ + index_size_gb decimal(38, 4) NULL, /* Size of the index in GB */ + index_rows bigint NULL, /* Number of rows in the index */ + index_reads bigint NULL, /* Total reads (seeks + scans + lookups) */ + index_writes bigint NULL, /* Total writes */ + script nvarchar(max) NULL /* Script to execute the action */ + ); + + CREATE TABLE + #key_duplicate_dedupe + ( + database_id integer NOT NULL, + object_id integer NOT NULL, + database_name sysname NOT NULL, + schema_name sysname NOT NULL, + table_name sysname NOT NULL, + base_key_columns nvarchar(max) NULL, + filter_definition nvarchar(max) NULL, + winning_index_name sysname NULL, + index_list nvarchar(max) NULL + ); + + CREATE TABLE + #include_subset_dedupe + ( + database_id integer NOT NULL, + object_id integer NOT NULL, + subset_index_name sysname NULL, + superset_index_name sysname NULL, + subset_included_columns nvarchar(max) NULL, + superset_included_columns nvarchar(max) NULL + ); + + /* Create a new temp table for detailed reporting statistics */ + CREATE TABLE + #index_reporting_stats + ( + summary_level varchar(20) NOT NULL, /* 'DATABASE', 'TABLE', 'INDEX', 'SUMMARY' */ + database_name sysname NULL, + schema_name sysname NULL, + table_name sysname NULL, + index_name sysname NULL, + server_uptime_days integer NULL, + uptime_warning bit NULL, + tables_analyzed integer NULL, + index_count integer NULL, + total_size_gb decimal(38, 4) NULL, + total_rows bigint NULL, + unused_indexes integer NULL, + unused_size_gb decimal(38, 4) NULL, + indexes_to_disable integer NULL, + indexes_to_merge integer NULL, + avg_indexes_per_table decimal(10, 2) NULL, + space_saved_gb decimal(10, 4) NULL, + compression_min_savings_gb decimal(10, 4) NULL, + compression_max_savings_gb decimal(10, 4) NULL, + total_min_savings_gb decimal(10, 4) NULL, + total_max_savings_gb decimal(10, 4) NULL, + /* Index usage metrics */ + total_reads bigint NULL, + total_writes bigint NULL, + user_seeks bigint NULL, + user_scans bigint NULL, + user_lookups bigint NULL, + user_updates bigint NULL, + /* Operational stats */ + range_scan_count bigint NULL, + singleton_lookup_count bigint NULL, + /* Lock stats */ + row_lock_count bigint NULL, + row_lock_wait_count bigint NULL, + row_lock_wait_in_ms bigint NULL, + page_lock_count bigint NULL, + page_lock_wait_count bigint NULL, + page_lock_wait_in_ms bigint NULL, + /* Latch stats */ + page_latch_wait_count bigint NULL, + page_latch_wait_in_ms bigint NULL, + page_io_latch_wait_count bigint NULL, + page_io_latch_wait_in_ms bigint NULL, + /* Misc stats */ + forwarded_fetch_count bigint NULL, + leaf_insert_count bigint NULL, + leaf_update_count bigint NULL, + leaf_delete_count bigint NULL + ); + + /* Create temp tables for database filtering */ + CREATE TABLE + #include_databases + ( + database_name sysname NOT NULL PRIMARY KEY + ); + + CREATE TABLE + #exclude_databases + ( + database_name sysname NOT NULL PRIMARY KEY + ); + + CREATE TABLE + #databases + ( + database_name sysname NOT NULL PRIMARY KEY, + database_id int NOT NULL + ); + + CREATE TABLE + #requested_but_skipped_databases + ( + database_name sysname NOT NULL PRIMARY KEY, + reason nvarchar(100) NOT NULL + ); + + /* Parse @include_databases comma-separated list */ + IF @get_all_databases = 1 + AND @include_databases IS NOT NULL + BEGIN + INSERT + #include_databases + WITH + (TABLOCK) + ( + database_name + ) + SELECT DISTINCT + database_name = + LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) + FROM + ( + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE + ( + @include_databases, + N',', + N'' + ) + + N'' + ) + ) AS a + CROSS APPLY x.nodes(N'//i') AS t(c) + WHERE LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) <> N'' + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#include_databases', + id.* + FROM #include_databases AS id + OPTION(RECOMPILE); + END; + END; + + IF @get_all_databases = 1 + AND @include_databases IS NOT NULL + BEGIN + INSERT + #requested_but_skipped_databases + WITH + (TABLOCK) + ( + database_name, + reason + ) + SELECT + id.database_name, + reason = + CASE + WHEN d.name IS NULL + THEN 'Database does not exist' + WHEN d.state <> 0 + THEN 'Database not online' + WHEN d.is_in_standby = 1 + THEN 'Database is in standby' + WHEN d.is_read_only = 1 + THEN 'Database is read-only' + WHEN d.database_id <= 4 + THEN 'System database' + ELSE 'Other issue' + END + FROM #include_databases AS id + LEFT JOIN sys.databases AS d + ON id.database_name = d.name + WHERE NOT EXISTS + ( + SELECT + 1/0 + FROM #databases AS db + WHERE db.database_name = id.database_name + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#requested_but_skipped_databases', + rbsd.* + FROM #requested_but_skipped_databases AS rbsd + OPTION(RECOMPILE); + END; + END; + + /* Parse @exclude_databases comma-separated list */ + IF @get_all_databases = 1 + AND @exclude_databases IS NOT NULL + BEGIN + INSERT + #exclude_databases + WITH + (TABLOCK) + ( + database_name + ) + SELECT DISTINCT + database_name = + LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) + FROM + ( + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE + ( + @exclude_databases, + N',', + N'' + ) + + N'' + ) + ) AS a + CROSS APPLY x.nodes(N'//i') AS t(c) + WHERE LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) <> N'' + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#exclude_databases', + ed.* + FROM #exclude_databases AS ed + OPTION(RECOMPILE); + END; + END; + + /* Check for conflicts between include and exclude lists */ + IF @get_all_databases = 1 + AND @include_databases IS NOT NULL + AND @exclude_databases IS NOT NULL + BEGIN + SELECT + @conflict_list = + @conflict_list + + ed.database_name + N', ' + FROM #exclude_databases AS ed + WHERE EXISTS + ( + SELECT + 1/0 + FROM #include_databases AS id + WHERE id.database_name = ed.database_name + ) + OPTION(RECOMPILE); + + /* If we found any conflicts, raise an error */ + IF LEN(@conflict_list) > 0 + BEGIN + /* Remove trailing comma and space */ + SET @conflict_list = LEFT(@conflict_list, LEN(@conflict_list) - 2); + + SET @error_msg = + N'The following databases appear in both @include_databases and @exclude_databases, which creates ambiguity: ' + + @conflict_list + N'. Please remove these databases from one of the lists.'; + + RAISERROR(@error_msg, 16, 1); + RETURN; + END; + END; + + /* Handle contradictory parameters */ + IF @get_all_databases = 1 + AND @database_name IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR(N'@database name being ignored since @get_all_databases is set to 1', 0, 0) WITH NOWAIT; + END; + SET @database_name = NULL; + END; + + /* Build the #databases table */ + IF @get_all_databases = 0 + BEGIN + /* Default to current database if not system db */ + IF @database_name IS NULL + AND DB_NAME() NOT IN + ( + N'master', + N'model', + N'msdb', + N'tempdb', + N'rdsadmin' + ) + BEGIN + SELECT + @database_name = DB_NAME(); + END; + + /* Single database mode */ + IF @database_name IS NOT NULL + BEGIN + INSERT + #databases + WITH + (TABLOCK) + ( + database_name, + database_id + ) + SELECT + d.name, + d.database_id + FROM sys.databases AS d + WHERE d.database_id = DB_ID(@database_name) + AND d.state = 0 + AND d.is_in_standby = 0 + AND d.is_read_only = 0 + OPTION(RECOMPILE); + + /* Get the database_id for backwards compatibility */ + SELECT + @current_database_id = d.database_id + FROM #databases AS d; + END; + END + ELSE + BEGIN + /* Multi-database mode */ + INSERT + #databases + WITH + (TABLOCK) + ( + database_name, + database_id + ) + SELECT + d.name, + d.database_id + FROM sys.databases AS d + WHERE d.database_id > 4 /* Skip system databases */ + AND d.state = 0 + AND d.is_in_standby = 0 + AND d.is_read_only = 0 + AND ( + @include_databases IS NULL + OR EXISTS (SELECT 1/0 FROM #include_databases AS id WHERE id.database_name = d.name) + ) + AND ( + @exclude_databases IS NULL + OR NOT EXISTS (SELECT 1/0 FROM #exclude_databases AS ed WHERE ed.database_name = d.name) + ) + OPTION(RECOMPILE); + END; + + /* Check for empty database list */ + IF (SELECT COUNT_BIG(*) FROM #databases AS d) = 0 + BEGIN + RAISERROR('No valid databases found to process.', 16, 1); + RETURN; + END; + + /* Show database list in debug mode */ + IF @debug = 1 + BEGIN + SELECT + table_name = '#databases', + d.* + FROM #databases AS d + OPTION(RECOMPILE); + END; + + /* + Set up database cursor processing + */ + + /* Create a cursor to process each database */ + SET @database_cursor = + CURSOR + LOCAL + SCROLL + DYNAMIC + READ_ONLY + FOR + SELECT + d.database_name, + d.database_id + FROM #databases AS d + ORDER BY + d.database_id; + + OPEN @database_cursor; + + FETCH FIRST + FROM @database_cursor + INTO + @current_database_name, + @current_database_id; + + /* + Start insert queries + */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #filtered_object insert', 0, 0) WITH NOWAIT; + END; + + WHILE @@FETCH_STATUS = 0 + BEGIN + /*Truncate temp tables between database iterations*/ + IF @debug = 1 + BEGIN + RAISERROR('Truncating per-database temp tables for the next iteration', 0, 0) WITH NOWAIT; + END; + + TRUNCATE TABLE + #filtered_objects; + TRUNCATE TABLE + #operational_stats; + TRUNCATE TABLE + #partition_stats; + TRUNCATE TABLE + #index_details; + TRUNCATE TABLE + #compression_eligibility; + TRUNCATE TABLE + #key_duplicate_dedupe; + TRUNCATE TABLE + #include_subset_dedupe; + + /*Validate searched objects per-database*/ + IF @schema_name IS NOT NULL + AND @table_name IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('validating object existence for %s.%s.%s.', 0, 0, @current_database_name, @schema_name, @table_name) WITH NOWAIT; + END; + + SELECT + @full_object_name = + QUOTENAME(@current_database_name) + + N'.' + + QUOTENAME(@schema_name) + + N'.' + + QUOTENAME(@table_name); + + SELECT + @object_id = + OBJECT_ID(@full_object_name); + + IF @object_id IS NULL + BEGIN + RAISERROR('The object %s doesn''t seem to exist', 10, 1, @full_object_name) WITH NOWAIT; + + IF @get_all_databases = 0 + BEGIN + RETURN; + END; + + /* Get the next database and continue the loop */ + FETCH NEXT + FROM @database_cursor + INTO + @current_database_name, + @current_database_id; + CONTINUE; + END; + END; + + /* Process current database */ + IF @debug = 1 + BEGIN + RAISERROR('Processing @current_database_name: %s and @current_database_id: %d', 0, 0, @current_database_name, @current_database_id) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql = N' + SELECT DISTINCT + @database_id, + database_name = DB_NAME(@database_id), + schema_id = t.schema_id, + schema_name = s.name, + object_id = t.object_id, + table_name = t.name, + index_id = i.index_id, + index_name = ISNULL(i.name, t.name + N''.Heap''), + can_compress = + CASE + WHEN p.index_id > 0 + AND p.data_compression = 0 + THEN 1 + ELSE 0 + END + FROM ' + QUOTENAME(@current_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.indexes AS i + ON t.object_id = i.object_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.partitions AS p + ON i.object_id = p.object_id + AND i.index_id = p.index_id + LEFT JOIN ' + QUOTENAME(@current_database_name) + N'.sys.dm_db_index_usage_stats AS us + ON t.object_id = us.object_id + AND us.database_id = @database_id + WHERE t.is_ms_shipped = 0 + AND t.type <> N''TF'' + AND NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.views AS v + WHERE v.object_id = i.object_id + )'; + + IF /* Check for temporal tables support */ + ( + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) IN (5, 8) /* Azure SQL DB or Managed Instance */ + OR CONVERT + ( + integer, + SUBSTRING + ( + CONVERT + ( + varchar(20), + SERVERPROPERTY('ProductVersion') + ), + 1, + 2 + ) + ) >= 13 + ) /* SQL 2016+ */ + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding temporal table screening', 0, 0) WITH NOWAIT; + END; + + SET @sql += N' + AND NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.tables AS t + WHERE t.object_id = i.object_id + AND t.temporal_type > 0 + )'; + END; + + + IF @object_id IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding object_id filter', 0, 0) WITH NOWAIT; + END; + + SELECT @sql += N' + AND t.object_id = @object_id'; + END; + + SET @sql += N' + AND EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.dm_db_partition_stats AS ps + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.allocation_units AS au + ON ps.partition_id = au.container_id + WHERE ps.object_id = t.object_id + GROUP BY + ps.object_id + HAVING + SUM(au.total_pages) * 8.0 / 1048576.0 >= @min_size_gb + ) + AND EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.dm_db_partition_stats AS ps + WHERE ps.object_id = t.object_id + AND ps.index_id IN (0, 1) + GROUP BY + ps.object_id + HAVING + SUM(ps.row_count) >= @min_rows + ) + AND EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.dm_db_index_usage_stats AS ius + WHERE ius.object_id = t.object_id + AND ius.database_id = @database_id + GROUP BY + ius.object_id + HAVING + SUM(ius.user_seeks + ius.user_scans + ius.user_lookups) >= @min_reads + OR + SUM(ius.user_updates) >= @min_writes + ) + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT @sql; + END; + + INSERT + #filtered_objects + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + object_id, + table_name, + index_id, + index_name, + can_compress + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id integer, + @min_reads bigint, + @min_writes bigint, + @min_size_gb decimal(10,2), + @min_rows bigint, + @object_id integer', + @current_database_id, + @min_reads, + @min_writes, + @min_size_gb, + @min_rows, + @object_id; + + SET @rc = ROWCOUNT_BIG(); + + IF @rc = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #filtered_objects from %s, continuing to next database...', 10, 0, @current_database_name) WITH NOWAIT; + END; + + IF @get_all_databases = 0 + BEGIN + RETURN; + END; + + /* Get the next database and continue the loop */ + FETCH NEXT + FROM @database_cursor + INTO + @current_database_name, + @current_database_id; + CONTINUE; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#filtered_objects', + fo.* + FROM #filtered_objects AS fo + OPTION(RECOMPILE); + + RAISERROR('Generating #compression_eligibility insert', 0, 0) WITH NOWAIT; + END; + + /* Populate compression eligibility table */ + INSERT INTO + #compression_eligibility + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + object_id, + table_name, + index_id, + index_name, + can_compress, + reason + ) + SELECT + fo.database_id, + fo.database_name, + fo.schema_id, + fo.schema_name, + fo.object_id, + fo.table_name, + fo.index_id, + fo.index_name, + can_compress = + CASE + @can_compress + WHEN 0 + THEN 0 + ELSE 1 + END, + reason = + CASE + @can_compress + WHEN 0 + THEN N'SQL Server edition or version does not support compression' + ELSE NULL + END + FROM #filtered_objects AS fo + WHERE fo.can_compress = 1 + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#compression_eligibility before update', + ce.* + FROM #compression_eligibility AS ce + OPTION(RECOMPILE); + END; + + /* Check for sparse columns or incompatible data types */ + IF @can_compress = 1 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Updating #compression_eligibility', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + + UPDATE + ce + SET + ce.can_compress = 0, + ce.reason = ''Table contains sparse columns or incompatible data types'' + FROM #compression_eligibility AS ce + WHERE EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.columns AS c + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.types AS t + ON c.user_type_id = t.user_type_id + WHERE c.object_id = ce.object_id + AND + ( + c.is_sparse = 1 + OR t.name IN (N''text'', N''ntext'', N''image'') + ) + ) + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#compression_eligibility after update', + ce.* + FROM #compression_eligibility AS ce + OPTION(RECOMPILE); + + RAISERROR('Generating #operational_stats insert', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql += N' + SELECT + os.database_id, + database_name = DB_NAME(os.database_id), + schema_id = s.schema_id, + schema_name = s.name, + os.object_id, + table_name = t.name, + os.index_id, + index_name = ISNULL(i.name, t.name + N''.Heap''), + range_scan_count = SUM(os.range_scan_count), + singleton_lookup_count = SUM(os.singleton_lookup_count), + forwarded_fetch_count = SUM(os.forwarded_fetch_count), + lob_fetch_in_pages = SUM(os.lob_fetch_in_pages), + row_overflow_fetch_in_pages = SUM(os.row_overflow_fetch_in_pages), + leaf_insert_count = SUM(os.leaf_insert_count), + leaf_update_count = SUM(os.leaf_update_count), + leaf_delete_count = SUM(os.leaf_delete_count), + leaf_ghost_count = SUM(os.leaf_ghost_count), + nonleaf_insert_count = SUM(os.nonleaf_insert_count), + nonleaf_update_count = SUM(os.nonleaf_update_count), + nonleaf_delete_count = SUM(os.nonleaf_delete_count), + leaf_allocation_count = SUM(os.leaf_allocation_count), + nonleaf_allocation_count = SUM(os.nonleaf_allocation_count), + row_lock_count = SUM(os.row_lock_count), + row_lock_wait_count = SUM(os.row_lock_wait_count), + row_lock_wait_in_ms = SUM(os.row_lock_wait_in_ms), + page_lock_count = SUM(os.page_lock_count), + page_lock_wait_count = SUM(os.page_lock_wait_count), + page_lock_wait_in_ms = SUM(os.page_lock_wait_in_ms), + index_lock_promotion_attempt_count = SUM(os.index_lock_promotion_attempt_count), + index_lock_promotion_count = SUM(os.index_lock_promotion_count), + page_latch_wait_count = SUM(os.page_latch_wait_count), + page_latch_wait_in_ms = SUM(os.page_latch_wait_in_ms), + tree_page_latch_wait_count = SUM(os.tree_page_latch_wait_count), + tree_page_latch_wait_in_ms = SUM(os.tree_page_latch_wait_in_ms), + page_io_latch_wait_count = SUM(os.page_io_latch_wait_count), + page_io_latch_wait_in_ms = SUM(os.page_io_latch_wait_in_ms), + page_compression_attempt_count = SUM(os.page_compression_attempt_count), + page_compression_success_count = SUM(os.page_compression_success_count) + FROM ' + QUOTENAME(@current_database_name) + N'.sys.dm_db_index_operational_stats + ( + @database_id, + @object_id, + NULL, + NULL + ) AS os + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.tables AS t + ON os.object_id = t.object_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.indexes AS i + ON os.object_id = i.object_id + AND os.index_id = i.index_id + WHERE EXISTS + ( + SELECT + 1/0 + FROM #filtered_objects AS fo + WHERE fo.database_id = os.database_id + AND fo.object_id = os.object_id + ) + GROUP BY + os.database_id, + DB_NAME(os.database_id), + s.schema_id, + s.name, + os.object_id, + t.name, + os.index_id, + i.name + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT @sql; + END; + + INSERT + #operational_stats + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + object_id, + table_name, + index_id, + index_name, + range_scan_count, + singleton_lookup_count, + forwarded_fetch_count, + lob_fetch_in_pages, + row_overflow_fetch_in_pages, + leaf_insert_count, + leaf_update_count, + leaf_delete_count, + leaf_ghost_count, + nonleaf_insert_count, + nonleaf_update_count, + nonleaf_delete_count, + leaf_allocation_count, + nonleaf_allocation_count, + row_lock_count, + row_lock_wait_count, + row_lock_wait_in_ms, + page_lock_count, + page_lock_wait_count, + page_lock_wait_in_ms, + index_lock_promotion_attempt_count, + index_lock_promotion_count, + page_latch_wait_count, + page_latch_wait_in_ms, + tree_page_latch_wait_count, + tree_page_latch_wait_in_ms, + page_io_latch_wait_count, + page_io_latch_wait_in_ms, + page_compression_attempt_count, + page_compression_success_count + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id integer, + @object_id integer', + @current_database_id, + @object_id; + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #operational_stats', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#operational_stats', + os.* + FROM #operational_stats AS os + OPTION(RECOMPILE); + + RAISERROR('Generating #index_details insert', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql += N' + SELECT + database_id = @database_id, + database_name = DB_NAME(@database_id), + t.object_id, + i.index_id, + s.schema_id, + schema_name = s.name, + table_name = t.name, + index_name = ISNULL(i.name, t.name + N''.Heap''), + column_name = c.name, + column_id = c.column_id, + i.is_primary_key, + i.is_unique, + i.is_unique_constraint, + is_indexed_view = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.objects AS so + WHERE i.object_id = so.object_id + AND so.is_ms_shipped = 0 + AND so.type = ''V'' + ) + THEN 1 + ELSE 0 + END, + is_foreign_key = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.foreign_key_columns AS f + WHERE f.parent_column_id = c.column_id + AND f.parent_object_id = c.object_id + ) + THEN 1 + ELSE 0 + END, + is_foreign_key_reference = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.foreign_key_columns AS f + WHERE f.referenced_column_id = c.column_id + AND f.referenced_object_id = c.object_id + ) + THEN 1 + ELSE 0 + END, + ic.key_ordinal, + ic.index_column_id, + ic.is_descending_key, + ic.is_included_column, + i.filter_definition, + is_max_length = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.types AS t + WHERE c.system_type_id = t.system_type_id + AND c.user_type_id = t.user_type_id + AND t.name IN (N''varchar'', N''nvarchar'') + AND t.max_length = -1 + ) + THEN 1 + ELSE 0 + END, + user_seeks = ISNULL(us.user_seeks, 0), + user_scans = ISNULL(us.user_scans, 0), + user_lookups = ISNULL(us.user_lookups, 0), + user_updates = ISNULL(us.user_updates, 0), + us.last_user_seek, + us.last_user_scan, + us.last_user_lookup, + us.last_user_update, + is_eligible_for_dedupe = + CASE + WHEN i.type = 2 + THEN 1 + WHEN + ( + i.type = 1 + OR i.is_primary_key = 1 + ) + THEN 0 + END + FROM ' + QUOTENAME(@current_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.indexes AS i + ON t.object_id = i.object_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.index_columns AS ic + ON i.object_id = ic.object_id + AND i.index_id = ic.index_id + JOIN ' + QUOTENAME(@current_database_name) + + CONVERT + ( + nvarchar(MAX), + N'.sys.columns AS c + ON ic.object_id = c.object_id + AND ic.column_id = c.column_id + LEFT JOIN sys.dm_db_index_usage_stats AS us + ON i.object_id = us.object_id + AND i.index_id = us.index_id + AND us.database_id = @database_id + WHERE t.is_ms_shipped = 0 + AND i.type IN (1, 2) + AND i.is_disabled = 0 + AND i.is_hypothetical = 0 + AND EXISTS + ( + SELECT + 1/0 + FROM #filtered_objects AS fo + WHERE fo.database_id = @database_id + AND fo.object_id = t.object_id + ) + AND EXISTS + ( + SELECT + 1/0 + FROM ' + ) + QUOTENAME(@current_database_name) + + CONVERT + ( + nvarchar(MAX), + N'.sys.dm_db_partition_stats ps + WHERE ps.object_id = t.object_id + AND ps.index_id = 1 + AND ps.row_count >= @min_rows + )' + ); + + IF @object_id IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding object+id filter', 0, 0) WITH NOWAIT; + END; + + SELECT @sql += N' + AND t.object_id = @object_id'; + END; + + SELECT + @sql += CONVERT + ( + nvarchar(max), + N' + AND NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@current_database_name) + N'.sys.objects AS so + WHERE i.object_id = so.object_id + AND so.is_ms_shipped = 0 + AND so.type = N''TF'' + ) + OPTION(RECOMPILE); + ' + ); + + IF @debug = 1 + BEGIN + PRINT SUBSTRING(@sql, 1, 4000); + PRINT SUBSTRING(@sql, 4000, 8000); + END; + + INSERT + #index_details + WITH + (TABLOCK) + ( + database_id, + database_name, + object_id, + index_id, + schema_id, + schema_name, + table_name, + index_name, + column_name, + column_id, + is_primary_key, + is_unique, + is_unique_constraint, + is_indexed_view, + is_foreign_key, + is_foreign_key_reference, + key_ordinal, + index_column_id, + is_descending_key, + is_included_column, + filter_definition, + is_max_length, + user_seeks, + user_scans, + user_lookups, + user_updates, + last_user_seek, + last_user_scan, + last_user_lookup, + last_user_update, + is_eligible_for_dedupe + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id integer, + @object_id integer, + @min_rows integer', + @current_database_id, + @object_id, + @min_rows; + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #index_details', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_details', + * + FROM #index_details AS id; + + RAISERROR('Generating #partition_stats insert', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql += N' + SELECT + database_id = @database_id, + database_name = DB_NAME(@database_id), + x.object_id, + x.index_id, + x.schema_id, + x.schema_name, + x.table_name, + x.index_name, + x.partition_id, + x.partition_number, + x.total_rows, + x.total_space_gb, + x.reserved_lob_gb, + x.reserved_row_overflow_gb, + x.data_compression_desc, + built_on = + ISNULL + ( + psfg.partition_scheme_name, + psfg.filegroup_name + ), + psfg.partition_function_name, + pc.partition_columns + FROM + ( + SELECT DISTINCT + ps.object_id, + ps.index_id, + s.schema_id, + schema_name = s.name, + table_name = t.name, + index_name = ISNULL(i.name, t.name + N''.Heap''), + ps.partition_id, + p.partition_number, + total_rows = ps.row_count, + total_space_gb = SUM(a.total_pages) * 8 / 1024.0 / 1024.0, /* Convert directly to GB */ + reserved_lob_gb = SUM(ps.lob_reserved_page_count) * 8. / 1024. / 1024.0, /* Convert directly to GB */ + reserved_row_overflow_gb = SUM(ps.row_overflow_reserved_page_count) * 8. / 1024. / 1024.0, /* Convert directly to GB */ + p.data_compression_desc, + i.data_space_id + FROM ' + QUOTENAME(@current_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.indexes AS i + ON t.object_id = i.object_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.partitions AS p + ON i.object_id = p.object_id + AND i.index_id = p.index_id + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.allocation_units AS a + ON p.partition_id = a.container_id + LEFT HASH JOIN ' + QUOTENAME(@current_database_name) + N'.sys.dm_db_partition_stats AS ps + ON p.partition_id = ps.partition_id + WHERE t.type <> N''TF'' + AND i.type IN (1, 2) + AND EXISTS + ( + SELECT + 1/0 + FROM #filtered_objects AS fo + WHERE fo.database_id = @database_id + AND fo.object_id = t.object_id + )'; + + IF @object_id IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding in object_id filter', 0, 0) WITH NOWAIT; + END; + + SELECT @sql += N' + AND t.object_id = @object_id'; + END; + + SELECT + @sql += N' + GROUP BY + ps.object_id, + ps.index_id, + s.schema_id, + s.name, + t.name, + i.name, + ps.partition_id, + p.partition_number, + ps.row_count, + p.data_compression_desc, + i.data_space_id + ) AS x + OUTER APPLY + ( + SELECT + filegroup_name = + fg.name, + partition_scheme_name = + ps.name, + partition_function_name = + pf.name + FROM ' + QUOTENAME(@current_database_name) + N'.sys.filegroups AS fg + FULL JOIN ' + QUOTENAME(@current_database_name) + N'.sys.partition_schemes AS ps + ON ps.data_space_id = fg.data_space_id + LEFT JOIN ' + QUOTENAME(@current_database_name) + N'.sys.partition_functions AS pf + ON pf.function_id = ps.function_id + WHERE x.data_space_id = fg.data_space_id + OR x.data_space_id = ps.data_space_id + ) AS psfg + OUTER APPLY + ( + SELECT + partition_columns = + STUFF + ( + ( + SELECT + N'', '' + + c.name + FROM ' + QUOTENAME(@current_database_name) + N'.sys.index_columns AS ic + JOIN ' + QUOTENAME(@current_database_name) + N'.sys.columns AS c + ON c.object_id = ic.object_id + AND c.column_id = ic.column_id + WHERE ic.object_id = x.object_id + AND ic.index_id = x.index_id + AND ic.partition_ordinal > 0 + ORDER BY + ic.partition_ordinal + FOR + XML + PATH(''''), + TYPE + ).value(''.'', ''nvarchar(max)''), + 1, + 2, + '''' + ) + ) AS pc + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT SUBSTRING(@sql, 1, 4000); + PRINT SUBSTRING(@sql, 4000, 8000); + END; + + INSERT + #partition_stats WITH(TABLOCK) + ( + database_id, + database_name, + object_id, + index_id, + schema_id, + schema_name, + table_name, + index_name, + partition_id, + partition_number, + total_rows, + total_space_gb, + reserved_lob_gb, + reserved_row_overflow_gb, + data_compression_desc, + built_on, + partition_function_name, + partition_columns + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id integer, + @object_id integer', + @current_database_id, + @object_id; + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #partition_stats', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#partition_stats', + * + FROM #partition_stats AS ps + OPTION(RECOMPILE); + + RAISERROR('Performing #index_analysis insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_analysis + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + table_name, + object_id, + index_id, + index_name, + is_unique, + key_columns, + included_columns, + filter_definition, + original_index_definition + ) + SELECT + @current_database_id, + database_name = DB_NAME(@current_database_id), + id1.schema_id, + id1.schema_name, + id1.table_name, + id1.object_id, + id1.index_id, + id1.index_name, + id1.is_unique, + key_columns = + STUFF + ( + ( + SELECT + N', ' + + QUOTENAME(id2.column_name) + + CASE + WHEN id2.is_descending_key = 1 + THEN N' DESC' + ELSE N'' + END + FROM #index_details id2 + WHERE id2.object_id = id1.object_id + AND id2.index_id = id1.index_id + AND id2.is_included_column = 0 + GROUP BY + id2.column_name, + id2.is_descending_key, + id2.key_ordinal + ORDER BY + id2.key_ordinal + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ), + included_columns = + STUFF + ( + ( + SELECT + N', ' + + QUOTENAME(id2.column_name) + FROM #index_details id2 + WHERE id2.object_id = id1.object_id + AND id2.index_id = id1.index_id + AND id2.is_included_column = 1 + GROUP BY + id2.column_name + ORDER BY + id2.column_name + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ), + id1.filter_definition, + /* Store the original index definition for validation */ + original_index_definition = + CASE + /* For unique constraints, use ALTER TABLE ADD CONSTRAINT syntax */ + WHEN id1.is_unique_constraint = 1 + THEN + N'ALTER TABLE ' + + QUOTENAME(DB_NAME(@current_database_id)) + + N'.' + + QUOTENAME(id1.schema_name) + + N'.' + + QUOTENAME(id1.table_name) + + N' ADD CONSTRAINT ' + + QUOTENAME(id1.index_name) + + N' UNIQUE (' + /* For regular indexes, use CREATE INDEX syntax */ + ELSE + N'CREATE ' + + CASE + WHEN id1.is_unique = 1 + THEN N'UNIQUE ' + ELSE N'' + END + + CASE + WHEN id1.index_id = 1 + THEN N'CLUSTERED ' + WHEN id1.index_id > 1 + THEN N'NONCLUSTERED ' + ELSE N'' + END + + N'INDEX ' + + QUOTENAME(id1.index_name) + + N' ON ' + + QUOTENAME(DB_NAME(@current_database_id)) + + N'.' + + QUOTENAME(id1.schema_name) + + N'.' + + QUOTENAME(id1.table_name) + + N' (' + END + + STUFF + ( + ( + SELECT + N', ' + + QUOTENAME(id2.column_name) + + CASE + WHEN id2.is_descending_key = 1 + THEN N' DESC' + ELSE N'' + END + FROM #index_details id2 + WHERE id2.object_id = id1.object_id + AND id2.index_id = id1.index_id + AND id2.is_included_column = 0 + GROUP BY + id2.column_name, + id2.is_descending_key, + id2.key_ordinal + ORDER BY + id2.key_ordinal + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ) + + N')' + + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details id3 + WHERE id3.object_id = id1.object_id + AND id3.index_id = id1.index_id + AND id3.is_included_column = 1 + ) + THEN N' INCLUDE (' + + STUFF + ( + ( + SELECT + N', ' + + QUOTENAME(id4.column_name) + FROM #index_details id4 + WHERE id4.object_id = id1.object_id + AND id4.index_id = id1.index_id + AND id4.is_included_column = 1 + GROUP BY + id4.column_id, + id4.column_name + ORDER BY + id4.column_id, + id4.column_name + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ) + + N')' + ELSE N'' + END + + CASE + WHEN id1.filter_definition IS NOT NULL + THEN N' WHERE ' + id1.filter_definition + ELSE N'' + END + + N';' + FROM #index_details id1 + WHERE id1.is_eligible_for_dedupe = 1 + GROUP BY + id1.schema_name, + id1.schema_id, + id1.table_name, + id1.index_name, + id1.index_id, + id1.is_unique, + id1.object_id, + id1.index_id, + id1.filter_definition, + id1.is_unique_constraint + OPTION(RECOMPILE); + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #index_analysis', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + + RAISERROR('Starting updates', 0, 0) WITH NOWAIT; + END; + + /* Calculate index priority scores based on actual columns that exist */ + UPDATE + #index_analysis + SET + #index_analysis.index_priority = + CASE + WHEN #index_analysis.index_id = 1 + THEN 1000 /* Clustered indexes get highest priority */ + ELSE 0 + END + + + CASE + /* Unique indexes get high priority, but reduce priority for unique constraints */ + WHEN #index_analysis.is_unique = 1 AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id_uc + WHERE id_uc.index_id = #index_analysis.index_id + AND id_uc.object_id = #index_analysis.object_id + AND id_uc.is_unique_constraint = 1 + ) THEN 500 + /* Unique constraints get lower priority */ + WHEN #index_analysis.is_unique = 1 AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id_uc + WHERE id_uc.index_id = #index_analysis.index_id + AND id_uc.object_id = #index_analysis.object_id + AND id_uc.is_unique_constraint = 1 + ) THEN 50 + ELSE 0 + END + + + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id + WHERE id.index_id = #index_analysis.index_id + AND id.object_id = #index_analysis.object_id + AND id.user_seeks > 0 + ) THEN 200 + ELSE 0 + END /* Indexes with seeks get priority */ + + + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id + WHERE id.index_id = #index_analysis.index_id + AND id.object_id = #index_analysis.object_id + AND id.user_scans > 0 + ) THEN 100 ELSE 0 + END + OPTION(RECOMPILE); /* Indexes with scans get some priority */ + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after priority score', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 1: Identify unused indexes */ + UPDATE + #index_analysis + SET + #index_analysis.consolidation_rule = + CASE + WHEN @uptime_warning = 1 + THEN 'Unused Index (WARNING: Server uptime < 14 days - usage data may be incomplete)' + ELSE 'Unused Index' + END, + #index_analysis.action = N'DISABLE' + WHERE EXISTS + ( + SELECT + 1/0 + FROM #index_details id + WHERE id.database_id = #index_analysis.database_id + AND id.object_id = #index_analysis.object_id + AND id.index_id = #index_analysis.index_id + AND id.user_seeks = 0 + AND id.user_scans = 0 + AND id.user_lookups = 0 + AND id.is_primary_key = 0 /* Don't disable primary keys */ + AND id.is_unique_constraint = 0 /* Don't disable unique constraints */ + AND id.is_eligible_for_dedupe = 1 /* Only eligible indexes */ + ) + AND #index_analysis.index_id <> 1 + OPTION(RECOMPILE); /* Don't disable clustered indexes */ + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 1', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 2: Exact duplicates - matching key columns and includes */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Exact Duplicate', + ia1.target_index_name = + CASE + WHEN ia1.index_priority > ia2.index_priority + THEN NULL /* This index is the keeper */ + WHEN ia1.index_priority = ia2.index_priority + AND ia1.index_name < ia2.index_name + THEN NULL /* When tied, use alphabetical ordering for consistency */ + ELSE ia2.index_name /* Other index is the keeper */ + END, + ia1.action = + CASE + WHEN ia1.index_priority > ia2.index_priority + THEN N'KEEP' /* This index is the keeper */ + WHEN ia1.index_priority = ia2.index_priority + AND ia1.index_name < ia2.index_name + THEN N'KEEP' /* When tied, use alphabetical ordering for consistency */ + ELSE N'DISABLE' /* Other index gets disabled */ + END + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia1.key_columns = ia2.key_columns /* Exact key match */ + AND ISNULL(ia1.included_columns, '') = ISNULL(ia2.included_columns, '') /* Exact includes match */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + /* Exclude unique constraints - we'll handle those separately in Rule 7 */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1_uc + WHERE id1_uc.database_id = ia1.database_id + AND id1_uc.object_id = ia1.object_id + AND id1_uc.index_id = ia1.index_id + AND id1_uc.is_unique_constraint = 1 + ) + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2_uc + WHERE id2_uc.database_id = ia2.database_id + AND id2_uc.object_id = ia2.object_id + AND id2_uc.index_id = ia2.index_id + AND id2_uc.is_unique_constraint = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_eligible_for_dedupe = 1 + ) + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + JOIN #index_details AS id2 + ON id2.database_id = id1.database_id + AND id2.object_id = id1.object_id + AND id2.column_name = id1.column_name + AND id2.key_ordinal = id1.key_ordinal + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id1.is_descending_key <> id2.is_descending_key /* Different sort direction */ + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 2', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + + /* Special debug for exact duplicates */ + RAISERROR('Special debug for exact duplicates after rule 2:', 0, 0) WITH NOWAIT; + SELECT + ia1.index_name AS index1_name, + ia1.action AS index1_action, + ia1.consolidation_rule AS index1_rule, + ia1.index_priority AS index1_priority, + ia1.target_index_name AS index1_target, + ia1.filter_definition AS index1_filter, + ia2.index_name AS index2_name, + ia2.action AS index2_action, + ia2.consolidation_rule AS index2_rule, + ia2.index_priority AS index2_priority, + ia2.target_index_name AS index2_target, + ia2.filter_definition AS index2_filter + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia1.key_columns = ia2.key_columns /* Exact key match */ + AND ISNULL(ia1.included_columns, '') = ISNULL(ia2.included_columns, '') /* Exact includes match */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + WHERE ia1.consolidation_rule = N'Exact Duplicate' + OR ia2.consolidation_rule = N'Exact Duplicate' + ORDER BY ia1.index_name + OPTION(RECOMPILE); + END; + + /* Rule 3: Key duplicates - matching key columns, different includes */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Key Duplicate', + ia1.target_index_name = + CASE + /* If one is unique and the other isn't, prefer the unique one */ + WHEN ia1.is_unique = 1 + AND ia2.is_unique = 0 + THEN NULL + WHEN ia1.is_unique = 0 + AND ia2.is_unique = 1 + THEN ia2.index_name + /* Otherwise use priority */ + WHEN ia1.index_priority >= ia2.index_priority + THEN NULL + ELSE ia2.index_name + END, + ia1.action = + CASE + WHEN (ia1.is_unique = 1 AND ia2.is_unique = 0) + OR + ( + ia1.index_priority >= ia2.index_priority + AND NOT (ia1.is_unique = 0 AND ia2.is_unique = 1) + ) + AND ISNULL(ia1.included_columns, N'') <> ISNULL(ia2.included_columns, N'') + THEN N'MERGE INCLUDES' /* Keep this index but merge includes */ + ELSE N'DISABLE' /* Other index is keeper, disable this one */ + END, + /* For the winning index, set clear superseded_by text for the report */ + ia1.superseded_by = + CASE + WHEN (ia1.is_unique = 1 AND ia2.is_unique = 0) + OR + ( + ia1.index_priority >= ia2.index_priority + AND NOT (ia1.is_unique = 0 AND ia2.is_unique = 1) + ) + THEN N'Supersedes ' + + ia2.index_name + ELSE NULL + END + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia1.key_columns = ia2.key_columns /* Exact key match */ + AND ISNULL(ia1.included_columns, '') <> ISNULL(ia2.included_columns, '') /* Different includes */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + /* Exclude pairs where either one is a unique constraint (we'll handle those separately in Rule 7) */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1_uc + WHERE id1_uc.database_id = ia1.database_id + AND id1_uc.object_id = ia1.object_id + AND id1_uc.index_id = ia1.index_id + AND id1_uc.is_unique_constraint = 1 + ) + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2_uc + WHERE id2_uc.database_id = ia2.database_id + AND id2_uc.object_id = ia2.object_id + AND id2_uc.index_id = ia2.index_id + AND id2_uc.is_unique_constraint = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_eligible_for_dedupe = 1 + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 3', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 4: Superset/subset key columns */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Key Subset', + ia1.target_index_name = ia2.index_name, + ia1.action = N'DISABLE' /* The narrower index gets disabled */ + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia2.key_columns LIKE (ia1.key_columns + '%') /* ia2 has wider key that starts with ia1's key */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + /* Exception: If narrower index is unique and wider is not, they should not be merged */ + AND NOT (ia1.is_unique = 1 AND ia2.is_unique = 0) + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_eligible_for_dedupe = 1 + ) + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + JOIN #index_details AS id2 + ON id2.database_id = id1.database_id + AND id2.object_id = id1.object_id + AND id2.column_name = id1.column_name + AND id2.key_ordinal = id1.key_ordinal + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id1.is_descending_key <> id2.is_descending_key /* Different sort direction */ + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 4', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 5: Mark superset indexes for merging with includes from subset */ + UPDATE + ia2 + SET + ia2.consolidation_rule = N'Key Superset', + ia2.action = N'MERGE INCLUDES', /* The wider index gets merged with includes */ + ia2.superseded_by = + ISNULL + ( + ia2.superseded_by + + ', ', + '' + ) + + N'Supersedes ' + + ia1.index_name + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.target_index_name = ia2.index_name /* Link from Rule 4 */ + WHERE ia1.consolidation_rule = N'Key Subset' + AND ia1.action = N'DISABLE' + AND ia2.consolidation_rule IS NULL /* Not already processed */ + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 5', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 6: Merge includes from subset to superset indexes */ + WITH + KeySubsetSuperset AS + ( + SELECT + superset.database_id, + superset.object_id, + superset.index_id, + superset.index_name, + superset.included_columns AS superset_includes, + subset.included_columns AS subset_includes + FROM #index_analysis AS superset + JOIN #index_analysis AS subset + ON superset.database_id = subset.database_id + AND superset.object_id = subset.object_id + AND subset.target_index_name = superset.index_name + WHERE superset.action = N'MERGE INCLUDES' + AND subset.action = N'DISABLE' + AND superset.consolidation_rule = N'Key Superset' + AND subset.consolidation_rule = N'Key Subset' + ) + UPDATE + ia + SET + ia.included_columns = + CASE + /* If both have includes, combine them without duplicates */ + WHEN kss.superset_includes IS NOT NULL + AND kss.subset_includes IS NOT NULL + THEN + /* Create combined includes using XML method that works with all SQL Server versions */ + ( + SELECT + /* Combine both sets of includes */ + combined_cols = + STUFF + ( + ( + SELECT DISTINCT + N', ' + + t.c.value('.', 'sysname') + FROM + ( + /* Create XML from superset includes */ + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE(kss.superset_includes, N', ', N'') + + N'' + ) + + UNION ALL + + /* Create XML from subset includes */ + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE(kss.subset_includes, N', ', N'') + + N'' + ) + ) AS a + /* Split XML into individual columns */ + CROSS APPLY a.x.nodes('/c') AS t(c) + FOR + XML + PATH('') + ), + 1, + 2, + '' + ) + ) + /* If only subset has includes, use those */ + WHEN kss.superset_includes IS NULL + AND kss.subset_includes IS NOT NULL + THEN kss.subset_includes + /* If only superset has includes or neither has includes, keep superset's includes */ + ELSE kss.superset_includes + END + FROM #index_analysis AS ia + JOIN KeySubsetSuperset AS kss + ON ia.database_id = kss.database_id + AND ia.object_id = kss.object_id + AND ia.index_id = kss.index_id + WHERE ia.action = N'MERGE INCLUDES' + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 6', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Update the superseded_by column for the wider index in a separate statement */ + UPDATE + ia2 + SET + ia2.superseded_by = N'Supersedes ' + ia1.index_name + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia2.key_columns LIKE (ia1.key_columns + N'%') /* ia2 has wider key that starts with ia1's key */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + /* Exception: If narrower index is unique and wider is not, they should not be merged */ + AND NOT (ia1.is_unique = 1 AND ia2.is_unique = 0) + WHERE ia1.consolidation_rule = N'Key Subset' /* Use records just processed in previous UPDATE */ + AND ia1.target_index_name = ia2.index_name /* Make sure we're updating the right wider index */ + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after update superseded', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 7: Unique constraint vs. nonclustered index handling */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Unique Constraint Replacement', + ia1.action = + CASE + WHEN ia1.is_unique = 0 + THEN 'MAKE UNIQUE' /* Convert to unique index */ + ELSE 'KEEP' /* Already unique, so just keep it */ + END + FROM #index_analysis AS ia1 + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia1.action IS NULL /* Not already processed by earlier rules */ + AND EXISTS + ( + /* Find nonclustered indexes */ + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + /* Find unique constraints with matching key columns */ + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia1.database_id + AND id2.object_id = ia1.object_id + AND id2.is_unique_constraint = 1 + AND NOT EXISTS + ( + /* Verify key columns match between index and unique constraint */ + SELECT + id2_inner.column_name + FROM #index_details AS id2_inner + WHERE id2_inner.database_id = id2.database_id + AND id2_inner.object_id = id2.object_id + AND id2_inner.index_id = id2.index_id + AND id2_inner.is_included_column = 0 + + EXCEPT + + SELECT + id1_inner.column_name + FROM #index_details AS id1_inner + WHERE id1_inner.database_id = ia1.database_id + AND id1_inner.object_id = ia1.object_id + AND id1_inner.index_id = ia1.index_id + AND id1_inner.is_included_column = 0 + ) + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 7', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 7.5: Mark unique constraints that have matching nonclustered indexes for disabling */ + /* First, mark unique constraints for disabling */ + UPDATE + ia_uc + SET + ia_uc.consolidation_rule = N'Unique Constraint Replacement', + ia_uc.action = N'DISABLE', /* Mark unique constraint for disabling */ + ia_uc.target_index_name = ia_nc.index_name /* Point to the nonclustered index that will replace it */ + FROM #index_analysis AS ia_uc /* Unique constraint */ + JOIN #index_details AS id_uc /* Join to get unique constraint details */ + ON id_uc.database_id = ia_uc.database_id + AND id_uc.object_id = ia_uc.object_id + AND id_uc.index_id = ia_uc.index_id + AND id_uc.is_unique_constraint = 1 /* This is a unique constraint */ + JOIN #index_analysis AS ia_nc /* Join to find nonclustered index */ + ON ia_nc.database_id = ia_uc.database_id + AND ia_nc.object_id = ia_uc.object_id + AND ia_nc.index_name <> ia_uc.index_name /* Different index */ + WHERE + /* Verify key columns EXACT match between index and unique constraint */ + ia_uc.key_columns = ia_nc.key_columns + OPTION(RECOMPILE); + + /* Second, mark nonclustered indexes to be made unique */ + UPDATE + ia_nc + SET + ia_nc.consolidation_rule = N'Unique Constraint Replacement', + ia_nc.action = N'MAKE UNIQUE', /* Mark nonclustered index to be made unique */ + /* CRITICAL: Set target_index_name to NULL to ensure it gets a MERGE script */ + ia_nc.target_index_name = NULL + FROM #index_analysis AS ia_nc /* Nonclustered index */ + JOIN #index_details AS id_nc /* Join to get nonclustered index details */ + ON id_nc.database_id = ia_nc.database_id + AND id_nc.object_id = ia_nc.object_id + AND id_nc.index_id = ia_nc.index_id + AND id_nc.is_unique_constraint = 0 /* This is not a unique constraint */ + WHERE + /* Two conditions for matching: + 1. Index key columns exactly match a unique constraint's key columns + 2. A unique constraint is already marked for DISABLE and has this index as target */ + EXISTS + ( + /* Find unique constraint with matching keys that should be disabled */ + SELECT + 1/0 + FROM #index_analysis AS ia_uc + JOIN #index_details AS id_uc + ON id_uc.database_id = ia_uc.database_id + AND id_uc.object_id = ia_uc.object_id + AND id_uc.index_id = ia_uc.index_id + AND id_uc.is_unique_constraint = 1 + WHERE ia_uc.database_id = ia_nc.database_id + AND ia_uc.object_id = ia_nc.object_id + /* Check that both indexes have EXACTLY the same key columns */ + AND ia_uc.key_columns = ia_nc.key_columns + ) + OPTION(RECOMPILE); + + /* CRITICAL: Ensure that only the unique constraints that exactly match get this treatment */ + /* And remove any incorrect MAKE UNIQUE actions */ + UPDATE + ia + SET + action = NULL, + consolidation_rule = NULL, + target_index_name = NULL + FROM #index_analysis AS ia + WHERE ia.action = N'MAKE UNIQUE' + AND NOT EXISTS ( + /* Check if there's a unique constraint with matching keys that points to this index */ + SELECT 1 + FROM #index_analysis AS ia_uc + WHERE ia_uc.database_id = ia.database_id + AND ia_uc.object_id = ia.object_id + AND ia_uc.key_columns = ia.key_columns + AND ia_uc.action = N'DISABLE' + AND ia_uc.target_index_name = ia.index_name + ) + OPTION(RECOMPILE); + + /* Make sure the nonclustered index has the superseded_by field set correctly */ + UPDATE + ia_nc + SET + ia_nc.superseded_by = + CASE + WHEN ia_nc.superseded_by IS NULL + THEN N'Will replace constraint ' + + ia_uc.index_name + ELSE ia_nc.superseded_by + + N', will replace constraint ' + ia_uc.index_name + END + FROM #index_analysis AS ia_nc + JOIN #index_analysis AS ia_uc + ON ia_uc.database_id = ia_nc.database_id + AND ia_uc.object_id = ia_nc.object_id + AND ia_uc.action = N'DISABLE' + AND ia_uc.target_index_name = ia_nc.index_name + WHERE ia_nc.action = N'MAKE UNIQUE' + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 7.5', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 8: Identify indexes with same keys but in different order after first column */ + /* This rule flags indexes that have the same set of key columns but ordered differently */ + /* These need manual review as they may be redundant depending on query patterns */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Same Keys Different Order', + ia1.action = N'REVIEW', /* These need manual review */ + ia1.target_index_name = ia2.index_name /* Reference the partner index */ + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name < ia2.index_name /* Only process each pair once */ + AND ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + WHERE + /* Leading columns match */ + EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + JOIN #index_details AS id2 + ON id1.database_id = id2.database_id + AND id1.object_id = id2.object_id + AND id1.column_name = id2.column_name + AND id1.key_ordinal = 1 + AND id2.key_ordinal = 1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id2.index_id = ia2.index_id + ) + /* Same set of key columns but in different order */ + AND NOT EXISTS + ( + /* Make sure the sets of key columns are exactly the same */ + SELECT + id1.column_name + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_included_column = 0 + AND id1.key_ordinal > 0 + + EXCEPT + + SELECT + id2.column_name + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_included_column = 0 + AND id2.key_ordinal > 0 + ) + /* But the order is different (excluding the first column) */ + AND EXISTS + ( + /* There's at least one column in a different position */ + SELECT + 1/0 + FROM #index_details AS id1 + JOIN #index_details AS id2 + ON id1.database_id = id2.database_id + AND id1.object_id = id2.object_id + AND id1.column_name = id2.column_name + AND id1.key_ordinal <> id2.key_ordinal + AND id1.key_ordinal > 1 /* After the first column */ + AND id2.key_ordinal > 1 /* After the first column */ + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id2.index_id = ia2.index_id + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 8', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Create a reference to the detailed summary that will appear at the end */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + consolidation_rule, + script_type, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + script, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT + result_type = 'SUMMARY', + sort_order = -1, + database_name = + N'processed databases: ' + + CASE + WHEN @get_all_databases = 0 + THEN ISNULL(@database_name, N'None') + ELSE + ISNULL + ( + STUFF + ( + ( + SELECT + N', ' + + d.database_name + FROM #databases AS d + ORDER BY + d.database_name + FOR + XML + PATH(''), + TYPE + ).value('.', 'nvarchar(max)'), + 1, + 2, + N'' + ), + N'None' + ) + END, + schema_name = + N'skipped databases: ' + + ISNULL + ( + STUFF + ( + ( + SELECT + N', ' + + rbs.database_name + + N' (' + + rbs.reason + + N')' + FROM #requested_but_skipped_databases AS rbs + ORDER BY + rbs.database_name + FOR + XML + PATH(''), + TYPE + ).value('.', 'nvarchar(MAX)'), + 1, + 2, + N'' + ), + N'None' + ), + table_name = N'brought to you by erikdarling.com', + index_name = N'for support: https://code.erikdarling.com/', + consolidation_rule = N'run date: ' + CONVERT(nvarchar(30), SYSDATETIME(), 120), + script_type = N'Index Cleanup Scripts', + additional_info = N'A detailed index analysis report appears after these scripts', + target_index_name = N'ALWAYS TEST THESE RECOMMENDATIONS', + superseded_info = N'IN A NON-PRODUCTION ENVIRONMENT FIRST!', + original_index_definition = N'please enjoy responsibly!', + script = N'happy index cleaning!', + index_size_gb = 0, + index_rows = 0, + index_reads = 0, + index_writes = 0 + OPTION(RECOMPILE); + + + /* Identify key duplicates where both indexes have MERGE INCLUDES action */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #key_duplicate_dedupe insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #key_duplicate_dedupe + WITH + (TABLOCK) + ( + database_id, + object_id, + database_name, + schema_name, + table_name, + base_key_columns, + filter_definition, + winning_index_name, + index_list + ) + SELECT + ia.database_id, + ia.object_id, + database_name = MAX(ia.database_name), + schema_name = MAX(ia.schema_name), + table_name = MAX(ia.table_name), + base_key_columns = ia.key_columns, + filter_definition = ISNULL(ia.filter_definition, N''), + /* Choose the index with most included columns as the winner (or first alphabetically if tied) */ + winning_index_name = + ( + SELECT TOP (1) + candidate.index_name + FROM #index_analysis AS candidate + WHERE candidate.database_id = ia.database_id + AND candidate.object_id = ia.object_id + AND candidate.key_columns = ia.key_columns + AND ISNULL(candidate.filter_definition, '') = ISNULL(ia.filter_definition, '') + AND candidate.action = N'MERGE INCLUDES' + AND candidate.consolidation_rule = N'Key Duplicate' + ORDER BY + /* Then prefer indexes with more included columns (by length as a proxy) */ + LEN(ISNULL(candidate.included_columns, '')) DESC, + /* Then alphabetically for stability */ + candidate.index_name + ), + /* Build a list of other indexes in this group */ + index_list = + STUFF + ( + ( + SELECT + N', ' + + inner_ia.index_name + FROM #index_analysis AS inner_ia + WHERE inner_ia.database_id = ia.database_id + AND inner_ia.object_id = ia.object_id + AND inner_ia.key_columns = ia.key_columns + AND ISNULL(inner_ia.filter_definition, '') = ISNULL(ia.filter_definition, '') + AND inner_ia.action = N'MERGE INCLUDES' + AND inner_ia.consolidation_rule = N'Key Duplicate' + GROUP BY + inner_ia.index_name + ORDER BY + inner_ia.index_name + FOR + XML + PATH(''), + TYPE + ).value('.', 'nvarchar(max)'), + 1, + 2, + '' + ) + FROM #index_analysis AS ia + WHERE ia.action = N'MERGE INCLUDES' + AND ia.consolidation_rule = N'Key Duplicate' + GROUP BY + ia.database_id, + ia.object_id, + ia.key_columns, + ia.filter_definition + HAVING + COUNT_BIG(*) > 1 + OPTION(RECOMPILE); /* Only groups with multiple MERGE INCLUDES */ + + /* Update the index_analysis table to make only one index the winner in each group */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_analysis updates', 0, 0) WITH NOWAIT; + END; + + UPDATE + ia + SET + ia.action = N'DISABLE', + ia.target_index_name = kdd.winning_index_name, + ia.superseded_by = NULL + FROM #index_analysis AS ia + JOIN #key_duplicate_dedupe AS kdd + ON ia.database_id = kdd.database_id + AND ia.object_id = kdd.object_id + AND ia.key_columns = kdd.base_key_columns + AND ISNULL(ia.filter_definition, N'') = kdd.filter_definition + WHERE ia.index_name <> kdd.winning_index_name + AND ia.action = N'MERGE INCLUDES' + AND ia.consolidation_rule = N'Key Duplicate' + OPTION(RECOMPILE); + + /* Update the winning index's superseded_by to list all other indexes */ + UPDATE + ia + SET + ia.superseded_by = N'Supersedes ' + + REPLACE + ( + kdd.index_list, + ia.index_name + N', ', + N'' + ) /* Remove self from list if present */ + FROM #index_analysis AS ia + JOIN #key_duplicate_dedupe AS kdd + ON ia.database_id = kdd.database_id + AND ia.object_id = kdd.object_id + AND ia.key_columns = kdd.base_key_columns + AND ISNULL(ia.filter_definition, '') = kdd.filter_definition + WHERE ia.index_name = kdd.winning_index_name + OPTION(RECOMPILE); + + /* Find indexes with same key columns where one has includes that are a subset of another */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #include_subset_dedupe insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #include_subset_dedupe + WITH + (TABLOCK) + ( + database_id, + object_id, + subset_index_name, + superset_index_name, + subset_included_columns, + superset_included_columns + ) + SELECT + ia1.database_id, + ia1.object_id, + ia1.index_name AS subset_index_name, + ia2.index_name AS superset_index_name, + ia1.included_columns AS subset_included_columns, + ia2.included_columns AS superset_included_columns + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.key_columns = ia2.key_columns + AND ISNULL(ia1.filter_definition, N'') = ISNULL(ia2.filter_definition, N'') + AND ia1.index_name <> ia2.index_name + AND ia1.action = N'MERGE INCLUDES' + AND ia2.action = N'MERGE INCLUDES' + AND ia1.consolidation_rule = N'Key Duplicate' + AND ia2.consolidation_rule = N'Key Duplicate' + /* Find where subset's includes are contained within superset's includes */ + AND + ( + ia1.included_columns IS NULL + OR CHARINDEX(ia1.included_columns, ia2.included_columns) > 0 + ) + /* Don't match if lengths are the same (would be exact duplicates) */ + AND + ( + ia1.included_columns IS NULL + OR ia2.included_columns IS NULL + OR LEN(ia1.included_columns) < LEN(ia2.included_columns) + ) + OPTION(RECOMPILE); + + /* Update the subset indexes to be disabled, since supersets already contain their columns */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_analysis updates', 0, 0) WITH NOWAIT; + END; + + UPDATE + ia + SET + ia.action = N'DISABLE', + ia.target_index_name = isd.superset_index_name, + ia.superseded_by = NULL + FROM #index_analysis AS ia + JOIN #include_subset_dedupe AS isd + ON ia.database_id = isd.database_id + AND ia.object_id = isd.object_id + AND ia.index_name = isd.subset_index_name + OPTION(RECOMPILE); + + /* Update the superset indexes to indicate they supersede the subset indexes */ + UPDATE + ia + SET + ia.superseded_by = + CASE + WHEN ia.superseded_by IS NULL + THEN N'Supersedes ' + + isd.subset_index_name + ELSE ia.superseded_by + + N', ' + + isd.subset_index_name + END + FROM #index_analysis AS ia + JOIN #include_subset_dedupe AS isd + ON ia.database_id = isd.database_id + AND ia.object_id = isd.object_id + AND ia.index_name = isd.superset_index_name + OPTION(RECOMPILE); + + /* Update winning indexes that don't actually need changes to have action = N'KEEP' */ + UPDATE + ia + SET + /* Change action to 'KEEP' for indexes that don't need to be modified */ + ia.action = N'KEEP' + FROM #index_analysis AS ia + WHERE ia.action = N'MERGE INCLUDES' + AND ia.superseded_by IS NOT NULL + /* This should indicate it already has all the needed includes */ + AND NOT EXISTS + ( + /* Find any indexes it supersedes that have includes not in this index */ + SELECT + 1/0 + FROM #index_analysis AS ia_subset + WHERE ia_subset.database_id = ia.database_id + AND ia_subset.object_id = ia.object_id + AND ia_subset.key_columns = ia.key_columns + AND ia_subset.action = N'DISABLE' + AND ia_subset.target_index_name = ia.index_name + /* This complex check handles cases where the superset doesn't contain all subset columns */ + AND CHARINDEX(ISNULL(ia_subset.included_columns, N''), ISNULL(ia.included_columns, N'')) = 0 + AND ISNULL(ia_subset.included_columns, N'') <> N'' + ) + OPTION(RECOMPILE); + + /* Insert merge scripts for indexes */ + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after all updates', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + + RAISERROR('Generating #index_cleanup_results insert, MERGE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + target_index_name, + script, + additional_info, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'MERGE', + /* Put merge target indexes higher in sort order (5) so they appear before + indexes that will be disabled (20) */ + sort_order = 5, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = N'MERGE SCRIPT', + ia.consolidation_rule, + ia.target_index_name, + script = + CASE + WHEN ia.action = N'MAKE UNIQUE' + THEN N'CREATE UNIQUE ' + WHEN ia.action = N'MERGE INCLUDES' + THEN N'CREATE ' + ELSE N'CREATE ' + END + + N'INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + N' (' + + ia.key_columns + + N')' + + CASE + WHEN ia.included_columns IS NOT NULL + AND LEN(ia.included_columns) > 0 + AND ia.action = N'MERGE INCLUDES' + THEN N' INCLUDE (' + + ia.included_columns + + N')' + WHEN ia.included_columns IS NOT NULL + AND LEN(ia.included_columns) > 0 + THEN N' INCLUDE (' + + ia.included_columns + + N')' + ELSE N'' + END + + CASE + WHEN ia.filter_definition IS NOT NULL + THEN N' WHERE ' + + ia.filter_definition + ELSE N'' + END + + N' WITH (DROP_EXISTING = ON, FILLFACTOR = 100, SORT_IN_TEMPDB = ON, ONLINE = ' + + CASE + WHEN @online = 1 + THEN N'ON' + ELSE N'OFF' + END + + CASE + WHEN ce.can_compress = 1 + THEN ', DATA_COMPRESSION = PAGE' + ELSE N'' + END + + N')' + + CASE + WHEN ps.partition_function_name IS NOT NULL + THEN N' ON ' + + QUOTENAME(ps.partition_function_name) + + N'(' + + ISNULL(ps.partition_columns, N'') + + N')' + WHEN ps.built_on IS NOT NULL + THEN N' ON ' + + QUOTENAME(ps.built_on) + ELSE N'' + END + N';', + /* Additional info about what this script does */ + additional_info = + CASE + WHEN ia.action = N'MERGE INCLUDES' + THEN N'This index will absorb includes from duplicate indexes' + WHEN ia.action = N'MAKE UNIQUE' + THEN N'This index will replace a unique constraint' + ELSE NULL + END, + /* Add superseded_by information if available */ + ia.superseded_by, + /* Original index definition for validation */ + ia.original_index_definition, + NULL, + NULL, + NULL, + NULL + FROM #index_analysis AS ia + LEFT JOIN + ( + /* Get the partition info for each index */ + SELECT + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + FROM #partition_stats ps + GROUP BY + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + ) AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ce.can_compress = 1 + /* Only create merge scripts for the indexes that should remain after merging */ + AND ia.target_index_name IS NULL + OPTION(RECOMPILE); + + /* Debug which indexes are getting MERGE scripts */ + IF @debug = 1 + BEGIN + RAISERROR('Indexes getting MERGE scripts:', 0, 0) WITH NOWAIT; + SELECT + ia.index_name, + ia.action, + ia.consolidation_rule, + ia.target_index_name, + script_type = 'WILL GET MERGE SCRIPT', + ia.included_columns + FROM #index_analysis AS ia + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ce.can_compress = 1 + AND ia.target_index_name IS NULL + ORDER BY + ia.index_name + OPTION(RECOMPILE); + END; + + /* Insert disable scripts for unneeded indexes */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, DISABLE', 0, 0) WITH NOWAIT; + + /* Debug for indexes that should get DISABLE scripts */ + RAISERROR('Indexes that should get DISABLE scripts:', 0, 0) WITH NOWAIT; + SELECT + ia.index_name, + ia.consolidation_rule, + ia.action, + ia.target_index_name, + ia.is_unique, + ia.index_priority, + is_unique_constraint = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id + WHERE id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_unique_constraint = 1 + ) + THEN 'YES' + ELSE 'NO' + END, + make_unique_target = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_analysis AS ia_make + WHERE ia_make.database_id = ia.database_id + AND ia_make.object_id = ia.object_id + AND ia_make.action = N'MAKE UNIQUE' + AND ia_make.target_index_name = ia.index_name + ) + THEN 'YES' + ELSE 'NO' + END, + will_get_script = + CASE + WHEN ia.action = N'DISABLE' + AND NOT EXISTS + ( + SELECT 1 + FROM #index_details AS id_uc + WHERE id_uc.database_id = ia.database_id + AND id_uc.object_id = ia.object_id + AND id_uc.index_id = ia.index_id + AND id_uc.is_unique_constraint = 1 + ) + THEN 'YES' + ELSE 'NO' + END + FROM #index_analysis AS ia + WHERE ia.index_name LIKE 'ix_filtered_%' + OR ia.index_name LIKE 'ix_desc_%' + ORDER BY + ia.index_name + OPTION(RECOMPILE); + + /* Debug for all indexes marked with action = DISABLE */ + RAISERROR('All indexes with action = DISABLE:', 0, 0) WITH NOWAIT; + SELECT + ia.index_name, + ia.consolidation_rule, + ia.action, + ia.target_index_name + FROM #index_analysis AS ia + WHERE ia.action = N'DISABLE' + ORDER BY + ia.index_name + OPTION(RECOMPILE); + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + script, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'DISABLE', + /* Sort duplicate/subset indexes first (20), then unused indexes last (25) */ + sort_order = + CASE + WHEN ia.consolidation_rule LIKE 'Unused Index%' THEN 25 + ELSE 20 + END, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'DISABLE SCRIPT', + ia.consolidation_rule, + script = + /* Use regular DISABLE syntax for indexes */ + N'ALTER INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + N' DISABLE;', + CASE + WHEN ia.consolidation_rule = N'Key Subset' + THEN N'This index is superseded by a wider index: ' + + ISNULL(ia.target_index_name, N'(unknown)') + WHEN ia.consolidation_rule = N'Exact Duplicate' + THEN N'This index is an exact duplicate of: ' + + ISNULL(ia.target_index_name, N'(unknown)') + WHEN ia.consolidation_rule = N'Key Duplicate' + THEN N'This index has the same keys as: ' + + ISNULL(ia.target_index_name, N'(unknown)') + WHEN ia.consolidation_rule LIKE 'Unused Index%' + THEN ia.consolidation_rule + WHEN ia.action = N'DISABLE' + THEN N'This index is redundant and will be disabled' + ELSE N'This index is redundant' + END, + ia.target_index_name, /* Include the target index name */ + superseded_info = NULL, /* Don't need superseded_by info for disabled indexes */ + /* Original index definition for validation */ + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + WHERE ia.action = N'DISABLE' + /* Exclude unique constraints - they are handled by DISABLE CONSTRAINT scripts */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id_uc + WHERE id_uc.database_id = ia.database_id + AND id_uc.object_id = ia.object_id + AND id_uc.index_id = ia.index_id + AND id_uc.is_unique_constraint = 1 + ) + /* Also exclude any index that is also going to be made unique in rule 7.5 */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_analysis AS ia_unique + WHERE ia_unique.database_id = ia.database_id + AND ia_unique.object_id = ia.object_id + AND ia_unique.index_name = ia.index_name + AND ia_unique.action = N'MAKE UNIQUE' + ) + OPTION(RECOMPILE); + + /* Add clustered indexes to #index_analysis specifically for compression purposes */ + IF @debug = 1 + BEGIN + RAISERROR('Adding clustered indexes to #index_analysis for compression', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_analysis + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + table_name, + object_id, + index_id, + index_name, + is_unique, + key_columns, + included_columns, + filter_definition, + original_index_definition + ) + SELECT + fo.database_id, + fo.database_name, + fo.schema_id, + fo.schema_name, + fo.table_name, + fo.object_id, + fo.index_id, + fo.index_name, + is_unique = + CASE + WHEN ce.can_compress = 1 + THEN id.is_unique + ELSE NULL + END, + key_columns = + STUFF + ( + ( + SELECT + N', ' + + QUOTENAME(id2.column_name) + + CASE + WHEN id2.is_descending_key = 1 + THEN N' DESC' + ELSE N'' + END + FROM #index_details id2 + WHERE id2.object_id = fo.object_id + AND id2.index_id = fo.index_id + AND id2.is_included_column = 0 + GROUP BY + id2.column_name, + id2.is_descending_key, + id2.key_ordinal + ORDER BY + id2.key_ordinal + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ), + included_columns = NULL, /* Clustered indexes cannot have included columns */ + filter_definition = NULL, /* Clustered indexes cannot have filters */ + original_index_definition = + CASE + WHEN id.is_primary_key = 1 + THEN + N'ALTER TABLE ' + + QUOTENAME(fo.database_name) + + N'.' + + QUOTENAME(fo.schema_name) + + N'.' + + QUOTENAME(fo.table_name) + + N' ADD CONSTRAINT ' + + QUOTENAME(fo.index_name) + + N' PRIMARY KEY ' + + CASE + WHEN ce.index_id = 1 + THEN N'CLUSTERED' + ELSE N'NONCLUSTERED' + END + + + N' (' + + STUFF + ( + ( + SELECT + N', ' + + QUOTENAME(id2.column_name) + + CASE + WHEN id2.is_descending_key = 1 + THEN N' DESC' + ELSE N'' + END + FROM #index_details id2 + WHERE id2.object_id = fo.object_id + AND id2.index_id = fo.index_id + AND id2.is_included_column = 0 + GROUP BY + id2.column_name, + id2.is_descending_key, + id2.key_ordinal + ORDER BY + id2.key_ordinal + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ) + + N');' + WHEN id.is_primary_key = 0 + THEN N'CREATE ' + + CASE + WHEN id.is_unique = 1 + THEN N'UNIQUE ' + ELSE N'' + END + + N'CLUSTERED INDEX' + + QUOTENAME(fo.index_name) + + N' ON ' + + QUOTENAME(fo.database_name) + + N'.' + + QUOTENAME(fo.schema_name) + + N'.' + + QUOTENAME(fo.table_name) + + N' (' + + STUFF + ( + ( + SELECT + N', ' + + QUOTENAME(id2.column_name) + + CASE + WHEN id2.is_descending_key = 1 + THEN N' DESC' + ELSE N'' + END + FROM #index_details id2 + WHERE id2.object_id = fo.object_id + AND id2.index_id = fo.index_id + AND id2.is_included_column = 0 + GROUP BY + id2.column_name, + id2.is_descending_key, + id2.key_ordinal + ORDER BY + id2.key_ordinal + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ) + + N');' + ELSE N'' + END + FROM #filtered_objects AS fo + JOIN #index_details AS id + ON id.database_id = fo.database_id + AND id.object_id = fo.object_id + AND id.index_id = fo.index_id + AND id.key_ordinal = 1 /* Only need one row per index */ + JOIN #compression_eligibility AS ce + ON ce.database_id = fo.database_id + AND ce.object_id = fo.object_id + AND ce.index_id = fo.index_id + WHERE + ( + fo.index_id = 1 /* Clustered indexes only */ + OR id.is_primary_key = 1 + ) + AND ce.can_compress = 1 /* Only those eligible for compression */ + /* Only add if not already in #index_analysis */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_analysis AS ia + WHERE ia.database_id = fo.database_id + AND ia.object_id = fo.object_id + AND ia.index_id = fo.index_id + ) + OPTION(RECOMPILE); + + /* If any clustered indexes were added, mark them as KEEP */ + UPDATE #index_analysis + SET action = N'KEEP' + WHERE index_id = 1 /* Clustered indexes */ + AND action IS NULL; + + /* Update index priority for clustered indexes to ensure they're not chosen for deduplication */ + UPDATE #index_analysis + SET index_priority = 1000 /* Maximum priority */ + WHERE index_id = 1 /* Clustered indexes */ + AND index_priority IS NULL; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after adding clustered indexes', + * + FROM #index_analysis AS ia + WHERE ia.index_id = 1 + OPTION(RECOMPILE); + END; + + /* Insert compression scripts for remaining indexes */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, COMPRESS', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + script, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'COMPRESS', + sort_order = 40, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'COMPRESSION SCRIPT', + script = + N'ALTER INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + CASE + WHEN ps.partition_function_name IS NOT NULL + THEN N' REBUILD PARTITION = ALL' + ELSE N' REBUILD' + END + + N' WITH (FILLFACTOR = 100, SORT_IN_TEMPDB = ON, ONLINE = ' + + CASE + WHEN @online = 1 + THEN N'ON' + ELSE N'OFF' + END + + CASE + WHEN ce.can_compress = 1 + THEN ', DATA_COMPRESSION = PAGE' + ELSE N'' + END + + N')', + additional_info = N'Compression type: All Partitions', + superseded_info = NULL, /* No target index for compression scripts */ + ia.superseded_by, /* Include superseded_by info for compression scripts */ + /* Original index definition for validation */ + ia.original_index_definition, + ps_full.total_space_gb, + ps_full.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN + ( + /* Get the partition info for each index */ + SELECT + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + FROM #partition_stats ps + GROUP BY + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + ) + AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #partition_stats AS ps_full + ON ia.database_id = ps_full.database_id + AND ia.object_id = ps_full.object_id + AND ia.index_id = ps_full.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE + /* Indexes that are not being disabled or merged */ + (ia.action IS NULL OR ia.action = N'KEEP') + /* Only indexes eligible for compression */ + AND ce.can_compress = 1 + OPTION(RECOMPILE); + + /* Insert disable scripts for unique constraints */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, CONSTRAINT', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + additional_info, + script, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'CONSTRAINT', + sort_order = 30, + ia_uc.database_name, + ia_uc.schema_name, + ia_uc.table_name, + ia_uc.index_name, + script_type = 'DISABLE CONSTRAINT SCRIPT', + additional_info = + N'This constraint is being replaced by: ' + + ISNULL(ia_uc.target_index_name, N'(unknown)'), + script = + N'ALTER TABLE ' + + QUOTENAME(ia_uc.database_name) + + N'.' + + QUOTENAME(ia_uc.schema_name) + + N'.' + + QUOTENAME(ia_uc.table_name) + + N' NOCHECK CONSTRAINT ' + + QUOTENAME(ia_uc.index_name) + + N';', + /* Original index definition for validation */ + original_index_definition = ia_uc.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id2.user_seeks + id2.user_scans + id2.user_lookups), + id2.user_updates + FROM #index_analysis AS ia_uc + JOIN #index_details AS id + ON id.database_id = ia_uc.database_id + AND id.object_id = ia_uc.object_id + AND id.index_id = ia_uc.index_id + AND id.is_unique_constraint = 1 + LEFT JOIN #index_details AS id2 + ON id2.database_id = ia_uc.database_id + AND id2.object_id = ia_uc.object_id + AND id2.index_id = ia_uc.index_id + AND id2.is_included_column = 0 /* Get only one row per index */ + AND id2.key_ordinal > 0 + LEFT JOIN #partition_stats AS ps + ON ia_uc.database_id = ps.database_id + AND ia_uc.object_id = ps.object_id + AND ia_uc.index_id = ps.index_id + WHERE + /* Only constraints that are marked for disabling */ + ia_uc.action = N'DISABLE' + /* That have consolidation_rule of 'Unique Constraint Replacement' */ + AND ia_uc.consolidation_rule = N'Unique Constraint Replacement' + OPTION(RECOMPILE); + + /* Insert per-partition compression scripts */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, COMPRESS_PARTITION', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + script, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'COMPRESS_PARTITION', + sort_order = 50, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'COMPRESSION SCRIPT - PARTITION', + script = + N'ALTER INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + N' REBUILD PARTITION = ' + + CONVERT + ( + nvarchar(20), + ps.partition_number + ) + + N' WITH (FILLFACTOR = 100, SORT_IN_TEMPDB = ON, ONLINE = ' + + CASE + WHEN @online = 1 + THEN N'ON' + ELSE N'OFF' + END + + CASE + WHEN ce.can_compress = 1 + THEN ', DATA_COMPRESSION = PAGE' + ELSE N'' + END + + N')', + N'Compression type: Per Partition | Partition: ' + + CONVERT + ( + nvarchar(20), + ps.partition_number + ) + + N' | Rows: ' + + CONVERT + ( + nvarchar(20), + ps.total_rows + ) + + N' | Size: ' + + CONVERT + ( + nvarchar(20), + CONVERT + ( + decimal(10,4), + ps.total_space_gb + ) + ) + + N' GB', + target_index_name = NULL, + superseded_info = NULL, + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE + /* Only partitioned indexes */ + ps.partition_function_name IS NOT NULL + /* Indexes that are not being disabled or merged */ + AND (ia.action IS NULL OR ia.action = N'KEEP') + /* Only indexes eligible for compression */ + AND ce.can_compress = 1 + OPTION(RECOMPILE); + + /* Insert compression ineligible info */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, INELIGIBLE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + additional_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'INELIGIBLE', + sort_order = 90, + ce.database_name, + ce.schema_name, + ce.table_name, + ce.index_name, + script_type = 'COMPRESSION INELIGIBLE', + ce.reason, + /* Original index definition for validation */ + original_index_definition = + ( + SELECT TOP (1) + ia.original_index_definition + FROM #index_analysis AS ia + WHERE ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + ), + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #compression_eligibility AS ce + LEFT JOIN #partition_stats AS ps + ON ce.database_id = ps.database_id + AND ce.object_id = ps.object_id + AND ce.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ce.database_id + AND id.object_id = ce.object_id + AND id.index_id = ce.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + WHERE ce.can_compress = 0 + OPTION(RECOMPILE); + + + /* Insert indexes identified for manual review */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, REVIEW', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + target_index_name, + additional_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'REVIEW', + sort_order = 93, /* Just before KEPT indexes */ + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'NEEDS REVIEW', + ia.consolidation_rule, + ia.target_index_name, + additional_info = + CASE + WHEN ia.consolidation_rule = N'Same Keys Different Order' + THEN N'This index has the same key columns as ' + + ISNULL(ia.target_index_name, N'(unknown)') + + N' but in a different order. May be redundant depending on query patterns.' + ELSE N'This index needs manual review' + END, + /* Original index definition for validation */ + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + WHERE ia.action = N'REVIEW' + OPTION(RECOMPILE); + + + /* Insert kept indexes into results - Consolidated all kept indexes logic in one place */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, KEPT INDEXES', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + superseded_info, + additional_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes, + script + ) + SELECT DISTINCT + result_type = 'KEPT', + sort_order = 95, /* Put kept indexes at the end */ + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = + CASE + /* Add compression status to script_type */ + WHEN ce.can_compress = 1 THEN 'KEPT - NEEDS COMPRESSION' + ELSE 'KEPT' + END, + ia.consolidation_rule, + ia.superseded_by, + additional_info = + CASE + WHEN ia.superseded_by IS NOT NULL + THEN 'This index supersedes other indexes and already has all needed columns' + WHEN ia.action = N'KEEP' + THEN 'This index is being kept' + ELSE NULL + END, + /* Original index definition for validation */ + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates, + /* Include compression script directly on KEPT records when needed */ + script = + CASE + WHEN ce.can_compress = 1 + THEN N'ALTER INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + CASE + WHEN ps_part.partition_function_name IS NOT NULL + THEN N' REBUILD PARTITION = ALL' + ELSE N' REBUILD' + END + + N' WITH (FILLFACTOR = 100, SORT_IN_TEMPDB = ON, ONLINE = ' + + CASE + WHEN @online = 1 + THEN N'ON' + ELSE N'OFF' + END + + N', DATA_COMPRESSION = PAGE)' + ELSE NULL + END + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN + ( + /* Get the partition info for each index */ + SELECT + ps.database_id, + ps.object_id, + ps.index_id, + ps.partition_function_name + FROM #partition_stats ps + GROUP BY + ps.database_id, + ps.object_id, + ps.index_id, + ps.partition_function_name + ) + AS ps_part + ON ia.database_id = ps_part.database_id + AND ia.object_id = ps_part.object_id + AND ia.index_id = ps_part.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + LEFT JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + /* Check that this index is not already in the results */ + WHERE NOT EXISTS + ( + SELECT + 1/0 + FROM #index_cleanup_results AS ir + WHERE ir.database_name = ia.database_name + AND ir.schema_name = ia.schema_name + AND ir.table_name = ia.table_name + AND ir.index_name = ia.index_name + AND ir.script_type NOT LIKE N'COMPRESSION%' + ) + /* Include only indexes that should be kept */ + AND + ( + ia.action = N'KEEP' + OR + ( + ia.action IS NULL + AND ia.index_id > 0 + ) + ) + OPTION(RECOMPILE); + + /* Insert overall summary information */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats insert, SUMMARY', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_reporting_stats + WITH + (TABLOCK) + ( + summary_level, + server_uptime_days, + uptime_warning, + tables_analyzed, + index_count, + total_size_gb, + indexes_to_disable, + indexes_to_merge, + avg_indexes_per_table, + space_saved_gb, + compression_min_savings_gb, + compression_max_savings_gb, + total_min_savings_gb, + total_max_savings_gb, + total_rows + ) + SELECT + summary_level = 'SUMMARY', + server_uptime_days = @uptime_days, + uptime_warning = @uptime_warning, + tables_analyzed = + COUNT_BIG(DISTINCT CONCAT(ia.database_id, N'.', ia.schema_id, N'.', ia.object_id)), + index_count = COUNT_BIG(*), + total_size_gb = SUM(ps.total_space_gb), + indexes_to_disable = + SUM + ( + CASE + WHEN ia.action = N'DISABLE' + THEN 1 + ELSE 0 + END + ), + indexes_to_merge = + SUM + ( + CASE + WHEN ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + THEN 1 + ELSE 0 + END + ), + avg_indexes_per_table = + COUNT_BIG(*) * 1.0 / + NULLIF + ( + COUNT_BIG(DISTINCT CONCAT(ia.database_id, N'.', ia.schema_id, N'.', ia.object_id)), + 0 + ), + /* Space savings from cleanup - only count DISABLE actions */ + space_saved_gb = + SUM + ( + CASE + WHEN ia.action = N'DISABLE' + THEN ps.total_space_gb + ELSE 0 + END + ), + /* Conservative compression savings estimate (20%) */ + compression_min_savings_gb = + SUM + ( + CASE + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.20 + ELSE 0 + END + ), + /* Optimistic compression savings estimate (60%) */ + compression_max_savings_gb = + SUM + ( + CASE + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.60 + ELSE 0 + END + ), + /* Total conservative savings - only count DISABLE actions for space savings */ + total_min_savings_gb = + SUM + ( + CASE + WHEN ia.action = N'DISABLE' + THEN ps.total_space_gb + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.20 + ELSE 0 + END + ), + /* Total optimistic savings - only count DISABLE actions for space savings */ + total_max_savings_gb = + SUM + ( + CASE + WHEN ia.action = N'DISABLE' + THEN ps.total_space_gb + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.60 + ELSE 0 + END + ), + /* Get total rows from database unique tables */ + total_rows = + ( + SELECT + SUM(t.row_count) + FROM + ( + SELECT + ps_distinct.object_id, + row_count = + MAX + ( + CASE + WHEN ps_distinct.index_id IN (0, 1) + THEN ps_distinct.total_rows + ELSE 0 + END + ) + FROM #partition_stats AS ps_distinct + WHERE ps_distinct.index_id IN (0, 1) + GROUP BY + ps_distinct.object_id + ) AS t + ) + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE ia.index_id > 1 + OPTION(RECOMPILE); + + /* Return enhanced database impact summaries */ + IF @debug = 1 + BEGIN + RAISERROR('Generating enhanced summary reports', 0, 0) WITH NOWAIT; + END; + + /* Insert database-level summaries */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats insert, DATABASE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_reporting_stats + WITH + (TABLOCK) + ( + summary_level, + database_name, + index_count, + total_size_gb, + total_rows, + indexes_to_merge, + unused_indexes, + unused_size_gb, + compression_min_savings_gb, + compression_max_savings_gb, + total_min_savings_gb, + total_max_savings_gb, + total_reads, + total_writes, + user_seeks, + user_scans, + user_lookups, + user_updates, + range_scan_count, + singleton_lookup_count, + row_lock_count, + row_lock_wait_count, + row_lock_wait_in_ms, + page_lock_count, + page_lock_wait_count, + page_lock_wait_in_ms, + page_latch_wait_count, + page_latch_wait_in_ms, + page_io_latch_wait_count, + page_io_latch_wait_in_ms, + forwarded_fetch_count, + leaf_insert_count, + leaf_update_count, + leaf_delete_count + ) + SELECT + summary_level = 'DATABASE', + ps.database_name, + index_count = + COUNT_BIG(DISTINCT CONCAT(ps.object_id, N'.', ps.index_id)), + total_size_gb = SUM(DISTINCT ps.total_space_gb), + /* Use a simple aggregation to avoid double-counting */ + /* Get actual row count by grabbing the real row count from clustered index/heap per table */ + total_rows = SUM(DISTINCT d.actual_rows), + indexes_to_merge = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ia.database_id = ps.database_id + ), + /* Use count from analysis to keep consistent with SUMMARY level */ + unused_indexes = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action = N'DISABLE' + AND ia.database_id = ps.database_id + ), + unused_size_gb = + ( + SELECT + SUM(subps.total_space_gb) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + WHERE subia.action = N'DISABLE' + AND subia.database_id = ps.database_id + ), + /* Conservative compression savings estimate (20%) */ + compression_min_savings_gb = + ( + SELECT + SUM(subps.total_space_gb * 0.20) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + AND subia.database_id = ps.database_id + ), + /* Optimistic compression savings estimate (60%) */ + compression_max_savings_gb = + ( + SELECT + SUM(subps.total_space_gb * 0.60) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + AND subia.database_id = ps.database_id + ), + /* Total conservative savings */ + total_min_savings_gb = + ( + SELECT + SUM( + CASE + WHEN subia.action = N'DISABLE' + THEN subps.total_space_gb + WHEN (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + THEN subps.total_space_gb * 0.20 + ELSE 0 + END + ) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + LEFT JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE subia.database_id = ps.database_id + ), + /* Total optimistic savings */ + total_max_savings_gb = + ( + SELECT + SUM( + CASE + WHEN subia.action = N'DISABLE' + THEN subps.total_space_gb + WHEN (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + THEN subps.total_space_gb * 0.60 + ELSE 0 + END + ) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + LEFT JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE subia.database_id = ps.database_id + ), + total_reads = SUM(id.user_seeks + id.user_scans + id.user_lookups), + total_writes = SUM(id.user_updates), + user_seeks = SUM(id.user_seeks), + user_scans = SUM(id.user_scans), + user_lookups = SUM(id.user_lookups), + user_updates = SUM(id.user_updates), + range_scan_count = SUM(os.range_scan_count), + singleton_lookup_count = SUM(os.singleton_lookup_count), + row_lock_count = SUM(os.row_lock_count), + row_lock_wait_count = SUM(os.row_lock_wait_count), + row_lock_wait_in_ms = SUM(os.row_lock_wait_in_ms), + page_lock_count = SUM(os.page_lock_count), + page_lock_wait_count = SUM(os.page_lock_wait_count), + page_lock_wait_in_ms = SUM(os.page_lock_wait_in_ms), + page_latch_wait_count = SUM(os.page_latch_wait_count), + page_latch_wait_in_ms = SUM(os.page_latch_wait_in_ms), + page_io_latch_wait_count = SUM(os.page_io_latch_wait_count), + page_io_latch_wait_in_ms = SUM(os.page_io_latch_wait_in_ms), + forwarded_fetch_count = SUM(os.forwarded_fetch_count), + leaf_insert_count = SUM(os.leaf_insert_count), + leaf_update_count = SUM(os.leaf_update_count), + leaf_delete_count = SUM(os.leaf_delete_count) + FROM #partition_stats AS ps + LEFT JOIN #index_details AS id + ON id.database_id = ps.database_id + AND id.object_id = ps.object_id + AND id.index_id = ps.index_id + AND id.is_included_column = 0 + AND id.key_ordinal > 0 + LEFT JOIN #operational_stats AS os + ON os.database_id = ps.database_id + AND os.object_id = ps.object_id + AND os.index_id = ps.index_id + OUTER APPLY + ( + /* Get actual row count per table using MAX from clustered index/heap */ + SELECT + actual_rows = + MAX + ( + CASE + WHEN ps2.index_id IN (0, 1) + THEN ps2.total_rows + ELSE 0 + END + ) + FROM #partition_stats AS ps2 + WHERE ps2.database_id = ps.database_id + AND ps2.object_id = ps.object_id + AND ps2.index_id IN (0, 1) + GROUP BY + ps2.object_id + ) AS d + GROUP BY + ps.database_name, + ps.database_id + OPTION(RECOMPILE); + + /* Insert table-level summaries */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats insert, TABLE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_reporting_stats + WITH + (TABLOCK) + ( + summary_level, + database_name, + schema_name, + table_name, + index_count, + total_size_gb, + total_rows, + indexes_to_merge, + unused_indexes, + unused_size_gb, + compression_min_savings_gb, + compression_max_savings_gb, + total_min_savings_gb, + total_max_savings_gb, + total_reads, + total_writes, + user_seeks, + user_scans, + user_lookups, + user_updates, + range_scan_count, + singleton_lookup_count, + row_lock_count, + row_lock_wait_count, + row_lock_wait_in_ms, + page_lock_count, + page_lock_wait_count, + page_lock_wait_in_ms, + page_latch_wait_count, + page_latch_wait_in_ms, + page_io_latch_wait_count, + page_io_latch_wait_in_ms, + forwarded_fetch_count, + leaf_insert_count, + leaf_update_count, + leaf_delete_count + ) + SELECT + summary_level = 'TABLE', + ps.database_name, + ps.schema_name, + ps.table_name, + index_count = COUNT_BIG(DISTINCT ps.index_id), + total_size_gb = SUM(DISTINCT ps.total_space_gb), + /* Use MAX to get the row count from the clustered index or heap */ + total_rows = + MAX + ( + CASE + WHEN ps.index_id IN (0, 1) + THEN ps.total_rows + ELSE 0 + END + ), + indexes_to_merge = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ia.database_id = ps.database_id + AND ia.schema_id = ps.schema_id + AND ia.object_id = ps.object_id + ), + /* Use count from analysis to keep consistent with SUMMARY level */ + unused_indexes = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action = N'DISABLE' + AND ia.database_id = ps.database_id + AND ia.schema_id = ps.schema_id + AND ia.object_id = ps.object_id + ), + unused_size_gb = + ( + SELECT + SUM(subps.total_space_gb) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + WHERE subia.action = N'DISABLE' + AND subia.database_id = ps.database_id + AND subia.schema_id = ps.schema_id + AND subia.object_id = ps.object_id + ), + /* Conservative compression savings estimate (20%) */ + compression_min_savings_gb = + ( + SELECT + SUM(subps.total_space_gb * 0.20) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + AND subia.database_id = ps.database_id + AND subia.schema_id = ps.schema_id + AND subia.object_id = ps.object_id + ), + /* Optimistic compression savings estimate (60%) */ + compression_max_savings_gb = + ( + SELECT + SUM(subps.total_space_gb * 0.60) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + AND subia.database_id = ps.database_id + AND subia.schema_id = ps.schema_id + AND subia.object_id = ps.object_id + ), + /* Total conservative savings */ + total_min_savings_gb = + ( + SELECT + SUM( + CASE + WHEN subia.action = N'DISABLE' + THEN subps.total_space_gb + WHEN (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + THEN subps.total_space_gb * 0.20 + ELSE 0 + END + ) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + LEFT JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE subia.database_id = ps.database_id + AND subia.schema_id = ps.schema_id + AND subia.object_id = ps.object_id + ), + /* Total optimistic savings */ + total_max_savings_gb = + ( + SELECT + SUM( + CASE + WHEN subia.action = N'DISABLE' + THEN subps.total_space_gb + WHEN (subia.action IS NULL OR subia.action = N'KEEP') + AND subce.can_compress = 1 + THEN subps.total_space_gb * 0.60 + ELSE 0 + END + ) + FROM #partition_stats AS subps + JOIN #index_analysis AS subia + ON subps.database_id = subia.database_id + AND subps.object_id = subia.object_id + AND subps.index_id = subia.index_id + LEFT JOIN #compression_eligibility AS subce + ON subce.database_id = subia.database_id + AND subce.object_id = subia.object_id + AND subce.index_id = subia.index_id + WHERE subia.database_id = ps.database_id + AND subia.schema_id = ps.schema_id + AND subia.object_id = ps.object_id + ), + total_reads = SUM(id.user_seeks + id.user_scans + id.user_lookups), + total_writes = SUM(id.user_updates), + user_seeks = SUM(id.user_seeks), + user_scans = SUM(id.user_scans), + user_lookups = SUM(id.user_lookups), + user_updates = SUM(id.user_updates), + range_scan_count = SUM(os.range_scan_count), + singleton_lookup_count = SUM(os.singleton_lookup_count), + row_lock_count = SUM(os.row_lock_count), + row_lock_wait_count = SUM(os.row_lock_wait_count), + row_lock_wait_in_ms = SUM(os.row_lock_wait_in_ms), + page_lock_count = SUM(os.page_lock_count), + page_lock_wait_count = SUM(os.page_lock_wait_count), + page_lock_wait_in_ms = SUM(os.page_lock_wait_in_ms), + page_latch_wait_count = SUM(os.page_latch_wait_count), + page_latch_wait_in_ms = SUM(os.page_latch_wait_in_ms), + page_io_latch_wait_count = SUM(os.page_io_latch_wait_count), + page_io_latch_wait_in_ms = SUM(os.page_io_latch_wait_in_ms), + forwarded_fetch_count = SUM(os.forwarded_fetch_count), + leaf_insert_count = SUM(os.leaf_insert_count), + leaf_update_count = SUM(os.leaf_update_count), + leaf_delete_count = SUM(os.leaf_delete_count) + FROM #partition_stats AS ps + LEFT JOIN #index_details AS id + ON id.database_id = ps.database_id + AND id.object_id = ps.object_id + AND id.index_id = ps.index_id + AND id.is_included_column = 0 + AND id.key_ordinal > 0 + LEFT JOIN #operational_stats AS os + ON os.database_id = ps.database_id + AND os.object_id = ps.object_id + AND os.index_id = ps.index_id + GROUP BY + ps.database_name, + ps.database_id, + ps.schema_name, + ps.schema_id, + ps.table_name, + ps.object_id + OPTION(RECOMPILE); + + /* We're not doing index-level summaries - focusing on database and table level reports */ + + /* + Return the consolidated results in a single result set + Results are ordered by: + 1. Summary information (overall stats, savings estimates) + 2. Merge scripts (includes merges and unique conversions) - sort_order 5 + 3. Disable scripts (for redundant indexes) - sort_order 20 + 4. Constraint scripts (for unique constraints to disable) + 5. Compression scripts (for tables eligible for compression) + 6. Partition-specific compression scripts + 7. Ineligible objects (tables that can't be compressed) + 8. Kept indexes - sort_order 95 + + Note: Merge target scripts are sorted higher in the results (sort_order 5) + so that new merged indexes are created before subset indexes are disabled. + + Within each category, indexes are sorted by size and impact for better prioritization. + */ + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_reporting_stats', + irs.* + FROM #index_reporting_stats AS irs + OPTION(RECOMPILE); + + SELECT + table_name = '#index_cleanup_results', + icr.* + FROM #index_cleanup_results AS icr + OPTION(RECOMPILE); + END; + + /* Get the next database */ + FETCH NEXT + FROM @database_cursor + INTO + @current_database_name, + @current_database_id; + END; + + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results, RESULTS', 0, 0) WITH NOWAIT; + END; + + SELECT + /* First, show the information needed to understand the script */ + script_type = + CASE + WHEN ir.result_type = 'KEPT' + AND ir.script_type IS NULL + THEN 'KEPT' + ELSE ir.script_type + END, + ir.additional_info, + /* Then show identifying information for the index */ + ir.database_name, + ir.schema_name, + ir.table_name, + ir.index_name, + /* Then show relationship information */ + consolidation_rule = ISNULL(ir.consolidation_rule, N'N/A'), + target_index_name = ISNULL(ir.target_index_name, N'N/A'), + /* Include superseded_by info for winning indexes */ + superseded_info = + CASE + WHEN ia.superseded_by IS NOT NULL + THEN ia.superseded_by + ELSE ISNULL(ir.superseded_info, N'N/A') + END, + /* Add size and usage metrics */ + index_size_gb = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0.0000' + ELSE FORMAT(ISNULL(ir.index_size_gb, 0), 'N4') + END, + index_rows = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0' + ELSE FORMAT(ISNULL(ir.index_rows, 0), 'N0') + END, + index_reads = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0' + ELSE FORMAT(ISNULL(ir.index_reads, 0), 'N0') + END, + index_writes = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0' + ELSE FORMAT(ISNULL(ir.index_writes, 0), 'N0') + END, + original_index_definition = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN N'please enjoy responsibly!' + ELSE ia.original_index_definition + END, + /* Finally show the actual script */ + ir.script + FROM + ( + /* Use a subquery with ROW_NUMBER to ensure we only get one row per index */ + SELECT + irs.*, + ROW_NUMBER() OVER + ( + PARTITION BY + database_name, + schema_name, + table_name, + index_name, + irs.script_type + ORDER BY + result_type DESC /* Prefer non-NULL result types */ + ) AS rn + FROM #index_cleanup_results AS irs + ) AS ir + LEFT JOIN #index_analysis AS ia + ON ir.database_name = ia.database_name + AND ir.schema_name = ia.schema_name + AND ir.table_name = ia.table_name + AND ir.index_name = ia.index_name + WHERE ir.rn = 1 /* Take only the first row for each index */ + ORDER BY + ir.database_name, + ir.sort_order, + /* Within each sort_order group, prioritize by size and usage */ + CASE + /* For SUMMARY, keep the original order */ + WHEN ir.result_type = 'SUMMARY' + THEN 0 + /* For script categories, order by size and impact */ + ELSE ISNULL(ir.index_size_gb, 0) + END DESC, + CASE + /* For SUMMARY, keep the original order */ + WHEN ir.result_type = 'SUMMARY' + THEN 0 + /* For script categories, consider rows as secondary sort */ + ELSE ISNULL(ir.index_rows, 0) + END DESC, + /* Then by database, schema, table, index name for consistent ordering */ + ir.schema_name, + ir.table_name, + ir.index_name + OPTION(RECOMPILE); + + /* + This section now REPLACES the existing summary view rather than supplementing it + We'll modify the existing query below rather than creating new output panes + */ + + /* Return streamlined reporting statistics focused on key metrics */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats, REPORT', 0, 0) WITH NOWAIT; + END; + + SELECT + /* Basic identification with enhanced naming */ + level = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'ANALYZED OBJECT DETAILS' + ELSE irs.summary_level + END, + + /* Server info (for summary) or database name */ + database_info = + CASE + WHEN irs.summary_level = 'SUMMARY' + AND irs.uptime_warning = 1 + THEN 'WARNING: Server uptime only ' + + CONVERT(varchar(10), irs.server_uptime_days) + + ' days - usage data may be incomplete!' + WHEN irs.summary_level = 'SUMMARY' + THEN 'Server uptime: ' + + CONVERT(varchar(10), irs.server_uptime_days) + + ' days' + ELSE irs.database_name + END, + + /* Schema and table names (except for summary) */ + schema_name = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN ISNULL(irs.schema_name, 'ALWAYS TEST THESE RECOMMENDATIONS') + WHEN irs.summary_level = 'DATABASE' + THEN N'N/A' + ELSE irs.schema_name + END, + table_name = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN ISNULL(irs.table_name, 'IN A NON-PRODUCTION ENVIRONMENT FIRST!') + WHEN irs.summary_level = 'DATABASE' + THEN N'N/A' + ELSE irs.table_name + END, + + /* ===== Section 1: Index Counts ===== */ + /* Tables analyzed (summary only) */ + tables_analyzed = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT(irs.tables_analyzed, 'N0') + WHEN irs.summary_level = 'DATABASE' + THEN FORMAT + ( + ( + SELECT + COUNT_BIG(DISTINCT CONCAT(ia.schema_id, N'.', ia.object_id)) + FROM #index_analysis AS ia + WHERE ia.database_name = irs.database_name + ), + 'N0' + ) + WHEN irs.summary_level = 'TABLE' + THEN FORMAT(1, 'N0') /* Each table row represents 1 analyzed table */ + ELSE FORMAT(0, 'N0') /* Show 0 instead of NULL */ + END, + + /* Total indexes */ + total_indexes = FORMAT(ISNULL(irs.index_count, 0), 'N0'), + + /* Removable indexes - report consistent values across levels */ + removable_indexes = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT(ISNULL(irs.indexes_to_disable, 0), 'N0') /* Indexes that will be disabled based on analysis */ + ELSE FORMAT(ISNULL(irs.unused_indexes, 0), 'N0') /* Unused indexes at database/table level */ + END, + + /* Show mergeable indexes across all levels */ + mergeable_indexes = FORMAT(ISNULL(irs.indexes_to_merge, 0), 'N0'), + + /* Percent of indexes that can be removed */ + percent_removable = + CASE + WHEN irs.summary_level = 'SUMMARY' + AND irs.index_count > 0 + THEN FORMAT(100.0 * ISNULL(irs.indexes_to_disable, 0) + / NULLIF(irs.index_count, 0), 'N1') + '%' + WHEN irs.index_count > 0 + THEN FORMAT(100.0 * ISNULL(irs.unused_indexes, 0) + / NULLIF(irs.index_count, 0), 'N1') + '%' + ELSE '0.0%' + END, + + /* ===== Section 2: Size and Space Savings with Before/After comparison ===== */ + /* Current size in GB */ + current_size_gb = FORMAT(ISNULL(irs.total_size_gb, 0), 'N2'), + + /* Size after cleanup - added this as new metric */ + size_after_cleanup_gb = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT(ISNULL(irs.total_size_gb, 0) - ISNULL(irs.space_saved_gb, 0), 'N2') + ELSE FORMAT(ISNULL(irs.total_size_gb, 0) - ISNULL(irs.unused_size_gb, 0), 'N2') + END, + + /* Size that can be saved through cleanup */ + space_saved_gb = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT(ISNULL(irs.space_saved_gb, 0), 'N2') + ELSE FORMAT(ISNULL(irs.unused_size_gb, 0), 'N2') + END, + + /* Space reduction percentage - added this as new metric */ + space_reduction_percent = + CASE + WHEN ISNULL(irs.total_size_gb, 0) > 0 + THEN + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT((ISNULL(irs.space_saved_gb, 0) / + NULLIF(irs.total_size_gb, 0)) * 100, 'N1') + '%' + ELSE FORMAT((ISNULL(irs.unused_size_gb, 0) / + NULLIF(irs.total_size_gb, 0)) * 100, 'N1') + '%' + END + ELSE '0.0%' + END, + + /* ===== Additional Space Savings from Compression ===== */ + /* Conservative compression estimate (20%) */ + compression_savings_potential = + N'minimum: ' + + FORMAT(ISNULL(irs.compression_min_savings_gb, 0), 'N2') + + N' GB maximum ' + + FORMAT(ISNULL(irs.compression_max_savings_gb, 0), 'N2') + + N'GB', + compression_savings_potential_total = + N'total minimum: ' + + FORMAT(ISNULL(irs.total_min_savings_gb, 0), 'N2') + + N' GB total maximum: ' + + FORMAT(ISNULL(irs.total_max_savings_gb, 0), 'N2') + + N'GB', + + /* ===== Section 3: Table and Usage Statistics ===== */ + /* Row count */ + total_rows = FORMAT(ISNULL(irs.total_rows, 0), 'N0'), + + /* Total reads - combined total and breakdown */ + reads_breakdown = + CASE + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.total_reads, 0), 'N0') + + ' (' + + FORMAT(ISNULL(irs.user_seeks, 0), 'N0') + + ' seeks, ' + + FORMAT(ISNULL(irs.user_scans, 0), 'N0') + + ' scans, ' + + FORMAT(ISNULL(irs.user_lookups, 0), 'N0') + + ' lookups)' + ELSE 'N/A' + END, + + /* Total writes */ + writes = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.total_writes, 0), 'N0') + ELSE '0' + END, + + /* Write operations saved - added as new metric */ + daily_write_ops_saved = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' /* For SUMMARY row, use N/A to be consistent with other metrics */ + WHEN irs.summary_level = 'DATABASE' + THEN 'N/A' + WHEN irs.summary_level = 'TABLE' + THEN + /* For TABLE rows, calculate estimated savings */ + CASE + WHEN ISNULL(irs.unused_indexes, 0) > 0 + THEN FORMAT + ( + CONVERT(decimal(38,2), + ISNULL + ( + irs.user_updates / + NULLIF + ( + CONVERT + ( + decimal(38,2), + ( + SELECT TOP (1) + irs2.server_uptime_days + FROM #index_reporting_stats AS irs2 + WHERE irs2.summary_level = 'SUMMARY' + ) + ), + 0 + ) * + ( + ISNULL + ( + irs.unused_indexes, + 0 + ) / + NULLIF + ( + CONVERT + ( + decimal(38,2), + irs.index_count + ), + 0 + ) + ), + 0 + ) + ), + 'N0' + ) + /* Rows without unused indexes have no savings */ + ELSE '0' + END + ELSE '0' + END, + + /* ===== Section 4: Consolidated Performance Metrics ===== */ + /* Total count of lock waits (row + page) */ + lock_wait_count = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.row_lock_wait_count, 0) + + ISNULL(irs.page_lock_wait_count, 0), 'N0') + ELSE '0' + END, + + /* Lock waits saved - new column */ + daily_lock_waits_saved = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' /* For SUMMARY row, use N/A to be consistent with other metrics */ + WHEN irs.summary_level = 'DATABASE' + THEN 'N/A' + WHEN irs.summary_level = 'TABLE' + THEN + /* For TABLE rows, calculate estimated savings */ + CASE + WHEN ISNULL(irs.unused_indexes, 0) > 0 + THEN + FORMAT + ( + CONVERT(decimal(38,2), + ISNULL + ( + (irs.row_lock_wait_count + irs.page_lock_wait_count) / + NULLIF + ( + CONVERT + ( + decimal(38,2), + ( + SELECT TOP (1) + irs2.server_uptime_days + FROM #index_reporting_stats AS irs2 + WHERE irs2.summary_level = 'SUMMARY' + ) + ), + 0 + ) * + ( + ISNULL + ( + irs.unused_indexes, + 0 + ) / + NULLIF + ( + CONVERT + ( + decimal(38,2), + irs.index_count + ), + 0 + ) + ), + 0 + ) + ), + 'N0' + ) + /* Rows without unused indexes have no savings */ + ELSE '0' + END + ELSE '0' + END, + + /* Average lock wait time in ms */ + avg_lock_wait_ms = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' + WHEN irs.summary_level <> 'SUMMARY' + AND (ISNULL(irs.row_lock_wait_count, 0) + + ISNULL(irs.page_lock_wait_count, 0)) > 0 + THEN FORMAT(1.0 * (ISNULL(irs.row_lock_wait_in_ms, 0) + + ISNULL(irs.page_lock_wait_in_ms, 0)) / + NULLIF(ISNULL(irs.row_lock_wait_count, 0) + + ISNULL(irs.page_lock_wait_count, 0), 0), 'N2') + ELSE '0' + END, + + /* Total count of latch waits (page + io) - new column */ + latch_wait_count = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.page_latch_wait_count, 0) + + ISNULL(irs.page_io_latch_wait_count, 0), 'N0') + ELSE '0' + END, + + /* Latch waits saved - new column */ + daily_latch_waits_saved = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' /* For SUMMARY row, use N/A to be consistent with other metrics */ + WHEN irs.summary_level = 'DATABASE' + THEN 'N/A' + WHEN irs.summary_level = 'TABLE' + THEN + /* For TABLE rows, calculate estimated savings */ + CASE + WHEN ISNULL(irs.unused_indexes, 0) > 0 + THEN + FORMAT + ( + CONVERT(decimal(38,2), + ISNULL + ( + (irs.page_latch_wait_count + irs.page_io_latch_wait_count) / + NULLIF + ( + CONVERT + ( + decimal(38,2), + ( + SELECT TOP (1) + irs2.server_uptime_days + FROM #index_reporting_stats AS irs2 + WHERE irs2.summary_level = 'SUMMARY' + ) + ), + 0 + ) * + ( + ISNULL + ( + irs.unused_indexes, + 0 + ) / + NULLIF + ( + CONVERT + ( + decimal(38,2), + irs.index_count + ), + 0 + ) + ), + 0 + ) + ), + 'N0' + ) + /* Rows without unused indexes have no savings */ + ELSE '0' + END + ELSE '0' + END, + + /* Combined latch wait time in ms */ + avg_latch_wait_ms = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN 'N/A' + WHEN irs.summary_level <> 'SUMMARY' + AND (ISNULL(irs.page_latch_wait_count, 0) + + ISNULL(irs.page_io_latch_wait_count, 0)) > 0 + THEN FORMAT(1.0 * (ISNULL(irs.page_latch_wait_in_ms, 0) + + ISNULL(irs.page_io_latch_wait_in_ms, 0)) / + NULLIF(ISNULL(irs.page_latch_wait_count, 0) + + ISNULL(irs.page_io_latch_wait_count, 0), 0), 'N2') + ELSE '0' + END + FROM #index_reporting_stats AS irs + ORDER BY + /* Order by database name */ + irs.database_name, + /* Then order by level - summary first */ + CASE + WHEN irs.summary_level = 'SUMMARY' THEN 0 + WHEN irs.summary_level = 'DATABASE' THEN 1 + WHEN irs.summary_level = 'TABLE' THEN 2 + ELSE 3 + END, + /* For tables, sort by potential savings and size */ + CASE + WHEN irs.summary_level = 'TABLE' THEN irs.unused_size_gb + ELSE 0 + END DESC, + CASE + WHEN irs.summary_level = 'TABLE' THEN irs.total_size_gb + ELSE 0 + END DESC, + /* Then by schema, table */ + irs.schema_name, + irs.table_name + OPTION(RECOMPILE); + +END TRY +BEGIN CATCH + THROW; +END CATCH; +END; /*Final End*/ +GO diff --git a/sp_IndexCleanup/sp_IndexCleanup_Old.sql b/sp_IndexCleanup/sp_IndexCleanup_Old.sql new file mode 100644 index 00000000..acaa3d27 --- /dev/null +++ b/sp_IndexCleanup/sp_IndexCleanup_Old.sql @@ -0,0 +1,4586 @@ +SET ANSI_WARNINGS ON; +SET ARITHABORT ON; +SET CONCAT_NULL_YIELDS_NULL ON; +SET QUOTED_IDENTIFIER ON; +SET NUMERIC_ROUNDABORT OFF; +SET IMPLICIT_TRANSACTIONS OFF; +SET STATISTICS TIME, IO OFF; +GO + +IF OBJECT_ID('dbo.sp_IndexCleanup', 'P') IS NULL +BEGIN + EXECUTE ('CREATE PROCEDURE dbo.sp_IndexCleanup AS RETURN 138;'); +END; +GO + +ALTER PROCEDURE + dbo.sp_IndexCleanup +( + @database_name sysname = NULL, + @schema_name sysname = NULL, + @table_name sysname = NULL, + @min_reads bigint = 0, + @min_writes bigint = 0, + @min_size_gb decimal(10,2) = 0, + @min_rows bigint = 0, + @help bit = 'false', + @debug bit = 'false', + @version varchar(20) = NULL OUTPUT, + @version_date datetime = NULL OUTPUT +) +WITH RECOMPILE +AS +BEGIN +SET NOCOUNT ON; + +BEGIN TRY + /* Check for SQL Server 2012 (11.0) or later for FORMAT and CONCAT functions*/ + + IF + /* Check SQL Server 2012+ for FORMAT and CONCAT functions */ + ( + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) NOT IN (5, 8) /* Not Azure SQL DB or Managed Instance */ + AND CONVERT + ( + integer, + SUBSTRING + ( + CONVERT + ( + varchar(20), + SERVERPROPERTY('ProductVersion') + ), + 1, + 2 + ) + ) < 11) /* Pre-2012 */ + BEGIN + RAISERROR('This procedure requires SQL Server 2012 (11.0) or later due to the use of FORMAT and CONCAT functions.', 11, 1); + RETURN; + END; + + SELECT + @version = '1.4', + @version_date = '20250401'; + + SELECT + for_insurance_purposes = + N'ALWAYS TEST THESE RECOMMENDATIONS IN A NON-PRODUCTION ENVIRONMENT FIRST!'; + + /* + Help section, for help. + Will become more helpful when out of beta. + */ + IF @help = 1 + BEGIN + SELECT + help = N'hello, i am sp_IndexCleanup' + UNION ALL + SELECT + help = N'this is a script to help clean up unused and duplicate indexes.' + UNION ALL + SELECT + help = N'it will also give you scripted out statements to add page compression to uncompressed indexes.' + UNION ALL + SELECT + help = N'always validate all changes against a non-production environment!' + UNION ALL + SELECT + help = N'without careful analysis and consideration, index changes can negative impacts on performance.'; + + /* + Parameters + */ + SELECT + parameter_name = + ap.name, + data_type = + t.name, + description = + CASE + ap.name + WHEN N'@database_name' THEN 'the name of the database you wish to analyze' + WHEN N'@schema_name' THEN 'the schema name to filter indexes by' + WHEN N'@table_name' THEN 'the table name to filter indexes by' + WHEN N'@min_reads' THEN 'minimum number of reads for an index to be considered used' + WHEN N'@min_writes' THEN 'minimum number of writes for an index to be considered used' + WHEN N'@min_size_gb' THEN 'minimum size in GB for an index to be analyzed' + WHEN N'@min_rows' THEN 'minimum number of rows for a table to be analyzed' + WHEN N'@help' THEN 'displays this help information' + WHEN N'@debug' THEN 'prints debug information during execution' + WHEN N'@version' THEN 'returns the version number of the procedure' + WHEN N'@version_date' THEN 'returns the date this version was released' + ELSE NULL + END, + valid_inputs = + CASE + ap.name + WHEN N'@database_name' THEN 'the name of a database you care about indexes in' + WHEN N'@schema_name' THEN 'schema name or NULL for all schemas' + WHEN N'@table_name' THEN 'table name or NULL for all tables' + WHEN N'@min_reads' THEN 'any positive integer or 0' + WHEN N'@min_writes' THEN 'any positive integer or 0' + WHEN N'@min_size_gb' THEN 'any positive decimal or 0' + WHEN N'@min_rows' THEN 'any positive integer or 0' + WHEN N'@help' THEN '0 or 1' + WHEN N'@debug' THEN '0 or 1' + WHEN N'@version' THEN 'OUTPUT parameter' + WHEN N'@version_date' THEN 'OUTPUT parameter' + ELSE NULL + END, + defaults = + CASE + ap.name + WHEN N'@database_name' THEN 'NULL' + WHEN N'@schema_name' THEN 'NULL' + WHEN N'@table_name' THEN 'NULL' + WHEN N'@min_reads' THEN '0' + WHEN N'@min_writes' THEN '0' + WHEN N'@min_size_gb' THEN '0' + WHEN N'@min_rows' THEN '0' + WHEN N'@help' THEN 'false' + WHEN N'@debug' THEN 'true' + WHEN N'@version' THEN 'NULL' + WHEN N'@version_date' THEN 'NULL' + ELSE NULL + END + FROM sys.all_parameters AS ap + JOIN sys.all_objects AS o + ON ap.object_id = o.object_id + JOIN sys.types AS t + ON ap.system_type_id = t.system_type_id + AND ap.user_type_id = t.user_type_id + WHERE o.name = N'sp_IndexCleanup' + OPTION(MAXDOP 1, RECOMPILE); + + SELECT + mit_license_yo = 'i am MIT licensed, so like, do whatever' + + UNION ALL + + SELECT + mit_license_yo = 'see printed messages for full license'; + + RAISERROR(' +MIT License + +Copyright 2025 Darling Data, LLC + +https://www.erikdarling.com/ + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, +sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the +following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE +FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +', 0, 1) WITH NOWAIT; + + RETURN; + END; + + IF @debug = 1 + BEGIN + RAISERROR('Declaring variables', 0, 0) WITH NOWAIT; + END; + + DECLARE + /*general script variables*/ + @sql nvarchar(max) = N'', + @database_id integer = NULL, + @object_id integer = NULL, + @full_object_name nvarchar(768) = NULL, + @uptime_warning bit = 0, /* Will set after @uptime_days is calculated */ + /*print variables*/ + @online bit = + CASE + WHEN + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) IN (3, 5, 8) + THEN 'true' /* Enterprise, Azure SQL DB, Managed Instance */ + ELSE 'false' + END, + /* Compression variables */ + @can_compress bit = + CASE + WHEN + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) IN (3, 5, 8) + OR + ( + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) = 2 + AND CONVERT + ( + integer, + SUBSTRING + ( + CONVERT + ( + varchar(20), + SERVERPROPERTY('ProductVersion') + ), + 1, + 2 + ) + ) >= 13 + ) + THEN 1 + ELSE 0 + END, + @uptime_days nvarchar(10) = + ( + SELECT + DATEDIFF + ( + DAY, + osi.sqlserver_start_time, + SYSDATETIME() + ) + FROM sys.dm_os_sys_info AS osi + ); + + /* Set uptime warning flag after @uptime_days is calculated */ + SELECT + @uptime_warning = + CASE + WHEN CONVERT(integer, @uptime_days) < 14 + THEN 1 + ELSE 0 + END; + + /* + Initial checks for object validity + */ + IF @debug = 1 + BEGIN + RAISERROR('Checking paramaters...', 0, 0) WITH NOWAIT; + END; + + IF @database_name IS NULL + AND DB_NAME() NOT IN + ( + N'master', + N'model', + N'msdb', + N'tempdb', + N'rdsadmin' + ) + BEGIN + SELECT + @database_name = DB_NAME(); + END; + + IF @database_name IS NOT NULL + BEGIN + SELECT + @database_id = d.database_id + FROM sys.databases AS d + WHERE d.name NOT IN (N'master', N'model', N'msdb', N'tempdb', 'rdsadmin') + AND d.state = 0 + AND d.is_in_standby = 0 + AND d.is_read_only = 0 + OPTION(RECOMPILE); + END; + + IF @schema_name IS NULL + AND @table_name IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @schema_name cannot be NULL when specifying a table, defaulting to dbo', 10, 1) WITH NOWAIT; + END; + + SELECT + @schema_name = N'dbo'; + END; + + IF @schema_name IS NOT NULL + AND @table_name IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('validating object existence for %s.%s.&s.', 0, 0, @database_name, @schema_name, @table_name) WITH NOWAIT; + END; + + SELECT + @full_object_name = + QUOTENAME(@database_name) + + N'.' + + QUOTENAME(@schema_name) + + N'.' + + QUOTENAME(@table_name); + + SELECT + @object_id = + OBJECT_ID(@full_object_name); + + IF @object_id IS NULL + BEGIN + RAISERROR('The object %s doesn''t seem to exist', 16, 1, @full_object_name) WITH NOWAIT; + RETURN; + END; + END; + + /* Parameter validation */ + IF @min_reads < 0 + OR @min_reads IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_reads cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_reads = 0; + END; + + IF @min_writes < 0 + OR @min_writes IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_writes cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_writes = 0; + END; + + IF @min_size_gb < 0 + OR @min_size_gb IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_size_gb cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_size_gb = 0; + END; + + IF @min_rows < 0 + OR @min_rows IS NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Parameter @min_rows cannot be NULL or negative. Setting to 0.', 10, 1) WITH NOWAIT; + END; + + SET @min_rows = 0; + END; + + /* + Temp tables! + */ + + IF @debug = 1 + BEGIN + RAISERROR('Creating temp tables', 0, 0) WITH NOWAIT; + END; + + CREATE TABLE + #filtered_objects + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + can_compress bit NOT NULL + PRIMARY KEY CLUSTERED(database_id, schema_id, object_id, index_id) + ); + + CREATE TABLE + #operational_stats + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + range_scan_count bigint NULL, + singleton_lookup_count bigint NULL, + forwarded_fetch_count bigint NULL, + lob_fetch_in_pages bigint NULL, + row_overflow_fetch_in_pages bigint NULL, + leaf_insert_count bigint NULL, + leaf_update_count bigint NULL, + leaf_delete_count bigint NULL, + leaf_ghost_count bigint NULL, + nonleaf_insert_count bigint NULL, + nonleaf_update_count bigint NULL, + nonleaf_delete_count bigint NULL, + leaf_allocation_count bigint NULL, + nonleaf_allocation_count bigint NULL, + row_lock_count bigint NULL, + row_lock_wait_count bigint NULL, + row_lock_wait_in_ms bigint NULL, + page_lock_count bigint NULL, + page_lock_wait_count bigint NULL, + page_lock_wait_in_ms bigint NULL, + index_lock_promotion_attempt_count bigint NULL, + index_lock_promotion_count bigint NULL, + page_latch_wait_count bigint NULL, + page_latch_wait_in_ms bigint NULL, + tree_page_latch_wait_count bigint NULL, + tree_page_latch_wait_in_ms bigint NULL, + page_io_latch_wait_count bigint NULL, + page_io_latch_wait_in_ms bigint NULL, + page_compression_attempt_count bigint NULL, + page_compression_success_count bigint NULL, + PRIMARY KEY CLUSTERED (database_id, schema_id, object_id, index_id) + ); + + CREATE TABLE + #partition_stats + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NULL, + partition_id bigint NOT NULL, + partition_number integer NOT NULL, + total_rows bigint NULL, + total_space_gb decimal(38, 4) NULL, /* Using 4 decimal places for GB to maintain precision */ + reserved_lob_gb decimal(38, 4) NULL, /* Using 4 decimal places for GB to maintain precision */ + reserved_row_overflow_gb decimal(38, 4) NULL, /* Using 4 decimal places for GB to maintain precision */ + data_compression_desc nvarchar(60) NULL, + built_on sysname NULL, + partition_function_name sysname NULL, + partition_columns nvarchar(max) + PRIMARY KEY CLUSTERED(database_id, schema_id, object_id, index_id, partition_id) + ); + + CREATE TABLE + #index_details + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NULL, + column_name sysname NOT NULL, + is_primary_key bit NULL, + is_unique bit NULL, + is_unique_constraint bit NULL, + is_indexed_view integer NOT NULL, + is_foreign_key bit NULL, + is_foreign_key_reference bit NULL, + key_ordinal tinyint NOT NULL, + index_column_id integer NOT NULL, + is_descending_key bit NOT NULL, + is_included_column bit NULL, + filter_definition nvarchar(max) NULL, + is_max_length integer NOT NULL, + user_seeks bigint NOT NULL, + user_scans bigint NOT NULL, + user_lookups bigint NOT NULL, + user_updates bigint NOT NULL, + last_user_seek datetime NULL, + last_user_scan datetime NULL, + last_user_lookup datetime NULL, + last_user_update datetime NULL, + is_eligible_for_dedupe bit NOT NULL + PRIMARY KEY CLUSTERED(database_id, object_id, index_id, column_name) + ); + + CREATE TABLE + #index_analysis + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + is_unique bit NULL, + key_columns nvarchar(MAX) NULL, + included_columns nvarchar(MAX) NULL, + filter_definition nvarchar(MAX) NULL, + /* Query plan for original CREATE INDEX statement */ + original_index_definition nvarchar(MAX) NULL, + /* + Consolidation rule that matched (e.g., Key Duplicate, Key Subset, etc) + For exact duplicates, use one of: Exact Duplicate, Reverse Duplicate, or Equal Except For Filter + */ + consolidation_rule nvarchar(256) NULL, + /* + Action to take (e.g., DISABLE, MERGE INCLUDES, KEEP) + If NULL, no action to be taken + */ + action nvarchar(100) NULL, + /* Target index to merge with or use instead of this one */ + target_index_name sysname NULL, + /* When this is a target, the index which points to it as a supersedes in consolidation */ + superseded_by nvarchar(4000) NULL, + /* Priority score from 0-1 to determine which index to keep (higher is better) */ + index_priority decimal(10,6) NULL + PRIMARY KEY CLUSTERED(database_id, object_id, index_id) + ); + + CREATE TABLE + #compression_eligibility + ( + database_id integer NOT NULL, + database_name sysname NOT NULL, + schema_id integer NOT NULL, + schema_name sysname NOT NULL, + object_id integer NOT NULL, + table_name sysname NOT NULL, + index_id integer NOT NULL, + index_name sysname NOT NULL, + can_compress bit NOT NULL, + reason nvarchar(200) NULL, + PRIMARY KEY CLUSTERED(database_id, object_id, index_id) + ); + + CREATE TABLE + #index_cleanup_results + ( + result_type varchar(100) NOT NULL, + sort_order integer NOT NULL, + database_name sysname NULL, + schema_name sysname NULL, + table_name sysname NULL, + index_name sysname NULL, + script_type nvarchar(60) NULL, /* Type of script (e.g., MERGE SCRIPT, DISABLE SCRIPT, etc.) */ + consolidation_rule nvarchar(256) NULL, /* Reason for action (e.g., Exact Duplicate, Key Subset) */ + target_index_name sysname NULL, /* If this index is a duplicate, indicates which index is the preferred one */ + superseded_info nvarchar(4000) NULL, /* If this is a kept index, indicates which indexes it supersedes */ + additional_info nvarchar(max) NULL, /* Additional information about the action */ + original_index_definition nvarchar(max) NULL, /* Original statement to create the index */ + index_size_gb decimal(38, 4) NULL, /* Size of the index in GB */ + index_rows bigint NULL, /* Number of rows in the index */ + index_reads bigint NULL, /* Total reads (seeks + scans + lookups) */ + index_writes bigint NULL, /* Total writes */ + script nvarchar(max) NULL /* Script to execute the action */ + ); + + CREATE TABLE + #key_duplicate_dedupe + ( + database_id integer NOT NULL, + object_id integer NOT NULL, + database_name sysname NOT NULL, + schema_name sysname NOT NULL, + table_name sysname NOT NULL, + base_key_columns nvarchar(max) NULL, + filter_definition nvarchar(max) NULL, + winning_index_name sysname NULL, + index_list nvarchar(max) NULL, + ); + + CREATE TABLE + #include_subset_dedupe + ( + database_id integer NOT NULL, + object_id integer NOT NULL, + subset_index_name sysname NULL, + superset_index_name sysname NULL, + subset_included_columns nvarchar(max) NULL, + superset_included_columns nvarchar(max) NULL + ); + + /* Create a new temp table for detailed reporting statistics */ + CREATE TABLE + #index_reporting_stats + ( + summary_level varchar(20) NOT NULL, /* 'DATABASE', 'TABLE', 'INDEX', 'SUMMARY' */ + database_name sysname NULL, + schema_name sysname NULL, + table_name sysname NULL, + index_name sysname NULL, + server_uptime_days integer NULL, + uptime_warning bit NULL, + tables_analyzed integer NULL, + index_count integer NULL, + total_size_gb decimal(38, 4) NULL, + total_rows bigint NULL, + unused_indexes integer NULL, + unused_size_gb decimal(38, 4) NULL, + indexes_to_disable integer NULL, + indexes_to_merge integer NULL, + avg_indexes_per_table decimal(10, 2) NULL, + space_saved_gb decimal(10, 4) NULL, + compression_min_savings_gb decimal(10, 4) NULL, + compression_max_savings_gb decimal(10, 4) NULL, + total_min_savings_gb decimal(10, 4) NULL, + total_max_savings_gb decimal(10, 4) NULL, + /* Index usage metrics */ + total_reads bigint NULL, + total_writes bigint NULL, + user_seeks bigint NULL, + user_scans bigint NULL, + user_lookups bigint NULL, + user_updates bigint NULL, + /* Operational stats */ + range_scan_count bigint NULL, + singleton_lookup_count bigint NULL, + /* Lock stats */ + row_lock_count bigint NULL, + row_lock_wait_count bigint NULL, + row_lock_wait_in_ms bigint NULL, + page_lock_count bigint NULL, + page_lock_wait_count bigint NULL, + page_lock_wait_in_ms bigint NULL, + /* Latch stats */ + page_latch_wait_count bigint NULL, + page_latch_wait_in_ms bigint NULL, + page_io_latch_wait_count bigint NULL, + page_io_latch_wait_in_ms bigint NULL, + /* Misc stats */ + forwarded_fetch_count bigint NULL, + leaf_insert_count bigint NULL, + leaf_update_count bigint NULL, + leaf_delete_count bigint NULL + ); + + /* + Start insert queries + */ + + IF @debug = 1 + BEGIN + RAISERROR('Generating #filtered_object insert', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql = N' + SELECT DISTINCT + @database_id, + database_name = DB_NAME(@database_id), + schema_id = t.schema_id, + schema_name = s.name, + object_id = t.object_id, + table_name = t.name, + index_id = i.index_id, + index_name = ISNULL(i.name, t.name + N''.Heap''), + can_compress = + CASE + WHEN p.index_id > 0 + AND p.data_compression = 0 + THEN 1 + ELSE 0 + END + FROM ' + QUOTENAME(@database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.indexes AS i + ON t.object_id = i.object_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.partitions AS p + ON i.object_id = p.object_id + AND i.index_id = p.index_id + LEFT JOIN ' + QUOTENAME(@database_name) + N'.sys.dm_db_index_usage_stats AS us + ON t.object_id = us.object_id + AND us.database_id = @database_id + WHERE t.is_ms_shipped = 0 + AND t.type <> N''TF'' + AND NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.views AS v + WHERE v.object_id = i.object_id + )'; + + IF /* Check for temporal tables support */ + ( + CONVERT + ( + integer, + SERVERPROPERTY('EngineEdition') + ) IN (5, 8) /* Azure SQL DB or Managed Instance */ + OR CONVERT + ( + integer, + SUBSTRING + ( + CONVERT + ( + varchar(20), + SERVERPROPERTY('ProductVersion') + ), + 1, + 2 + ) + ) >= 13 + ) /* SQL 2016+ */ + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding temporal table screening', 0, 0) WITH NOWAIT; + END; + + SET @sql += N' + AND NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.tables AS t + WHERE t.object_id = i.object_id + AND t.temporal_type > 0 + )'; + END; + + + IF @object_id IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding object_id filter', 0, 0) WITH NOWAIT; + END; + + SELECT @sql += N' + AND t.object_id = @object_id'; + END; + + SET @sql += N' + AND EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.dm_db_partition_stats AS ps + JOIN ' + QUOTENAME(@database_name) + N'.sys.allocation_units AS au + ON ps.partition_id = au.container_id + WHERE ps.object_id = t.object_id + GROUP BY + ps.object_id + HAVING + SUM(au.total_pages) * 8.0 / 1048576.0 >= @min_size_gb + ) + AND EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.dm_db_partition_stats AS ps + WHERE ps.object_id = t.object_id + AND ps.index_id IN (0, 1) + GROUP BY + ps.object_id + HAVING + SUM(ps.row_count) >= @min_rows + ) + AND EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.dm_db_index_usage_stats AS ius + WHERE ius.object_id = t.object_id + AND ius.database_id = @database_id + GROUP BY + ius.object_id + HAVING + SUM(ius.user_seeks + ius.user_scans + ius.user_lookups) >= @min_reads + OR + SUM(ius.user_updates) >= @min_writes + ) + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT @sql; + END; + + INSERT + #filtered_objects + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + object_id, + table_name, + index_id, + index_name, + can_compress + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id int, + @min_reads bigint, + @min_writes bigint, + @min_size_gb decimal(10,2), + @min_rows bigint, + @object_id integer', + @database_id, + @min_reads, + @min_writes, + @min_size_gb, + @min_rows, + @object_id; + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #filtered_objects', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#filtered_objects', + fo.* + FROM #filtered_objects AS fo + OPTION(RECOMPILE); + + RAISERROR('Generating #compression_eligibility insert', 0, 0) WITH NOWAIT; + END; + + /* Populate compression eligibility table */ + INSERT INTO + #compression_eligibility + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + object_id, + table_name, + index_id, + index_name, + can_compress, + reason + ) + SELECT + fo.database_id, + fo.database_name, + fo.schema_id, + fo.schema_name, + fo.object_id, + fo.table_name, + fo.index_id, + fo.index_name, + 1, /* Default to compressible */ + NULL + FROM #filtered_objects AS fo + WHERE fo.can_compress = 1 + OPTION(RECOMPILE); + + /* If SQL Server edition doesn't support compression, mark all as ineligible */ + IF @can_compress = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('updating compression eligibility', 0, 0) WITH NOWAIT; + END; + + UPDATE + #compression_eligibility + SET + #compression_eligibility.can_compress = 0, + #compression_eligibility.reason = N'SQL Server edition or version does not support compression' + WHERE #compression_eligibility.can_compress = 1 + OPTION(RECOMPILE); + END; + + /* Check for sparse columns or incompatible data types */ + IF @can_compress = 1 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Updating #compression_eligibility', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + + UPDATE + ce + SET + ce.can_compress = 0, + ce.reason = ''Table contains sparse columns or incompatible data types'' + FROM #compression_eligibility AS ce + WHERE EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.columns AS c + JOIN ' + QUOTENAME(@database_name) + N'.sys.types AS t + ON c.user_type_id = t.user_type_id + WHERE c.object_id = ce.object_id + AND + ( + c.is_sparse = 1 + OR t.name IN (N''text'', N''ntext'', N''image'') + ) + ) + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#compression_eligibility', + ce.* + FROM #compression_eligibility AS ce + OPTION(RECOMPILE); + + RAISERROR('Generating #operational_stats insert', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql += N' + SELECT + os.database_id, + database_name = DB_NAME(os.database_id), + schema_id = s.schema_id, + schema_name = s.name, + os.object_id, + table_name = t.name, + os.index_id, + index_name = ISNULL(i.name, t.name + N''.Heap''), + range_scan_count = SUM(os.range_scan_count), + singleton_lookup_count = SUM(os.singleton_lookup_count), + forwarded_fetch_count = SUM(os.forwarded_fetch_count), + lob_fetch_in_pages = SUM(os.lob_fetch_in_pages), + row_overflow_fetch_in_pages = SUM(os.row_overflow_fetch_in_pages), + leaf_insert_count = SUM(os.leaf_insert_count), + leaf_update_count = SUM(os.leaf_update_count), + leaf_delete_count = SUM(os.leaf_delete_count), + leaf_ghost_count = SUM(os.leaf_ghost_count), + nonleaf_insert_count = SUM(os.nonleaf_insert_count), + nonleaf_update_count = SUM(os.nonleaf_update_count), + nonleaf_delete_count = SUM(os.nonleaf_delete_count), + leaf_allocation_count = SUM(os.leaf_allocation_count), + nonleaf_allocation_count = SUM(os.nonleaf_allocation_count), + row_lock_count = SUM(os.row_lock_count), + row_lock_wait_count = SUM(os.row_lock_wait_count), + row_lock_wait_in_ms = SUM(os.row_lock_wait_in_ms), + page_lock_count = SUM(os.page_lock_count), + page_lock_wait_count = SUM(os.page_lock_wait_count), + page_lock_wait_in_ms = SUM(os.page_lock_wait_in_ms), + index_lock_promotion_attempt_count = SUM(os.index_lock_promotion_attempt_count), + index_lock_promotion_count = SUM(os.index_lock_promotion_count), + page_latch_wait_count = SUM(os.page_latch_wait_count), + page_latch_wait_in_ms = SUM(os.page_latch_wait_in_ms), + tree_page_latch_wait_count = SUM(os.tree_page_latch_wait_count), + tree_page_latch_wait_in_ms = SUM(os.tree_page_latch_wait_in_ms), + page_io_latch_wait_count = SUM(os.page_io_latch_wait_count), + page_io_latch_wait_in_ms = SUM(os.page_io_latch_wait_in_ms), + page_compression_attempt_count = SUM(os.page_compression_attempt_count), + page_compression_success_count = SUM(os.page_compression_success_count) + FROM ' + QUOTENAME(@database_name) + N'.sys.dm_db_index_operational_stats + ( + @database_id, + @object_id, + NULL, + NULL + ) AS os + JOIN ' + QUOTENAME(@database_name) + N'.sys.tables AS t + ON os.object_id = t.object_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.indexes AS i + ON os.object_id = i.object_id + AND os.index_id = i.index_id + WHERE EXISTS + ( + SELECT + 1/0 + FROM #filtered_objects AS fo + WHERE fo.database_id = os.database_id + AND fo.object_id = os.object_id + ) + GROUP BY + os.database_id, + DB_NAME(os.database_id), + s.schema_id, + s.name, + os.object_id, + t.name, + os.index_id, + i.name + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT @sql; + END; + + INSERT + #operational_stats + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + object_id, + table_name, + index_id, + index_name, + range_scan_count, + singleton_lookup_count, + forwarded_fetch_count, + lob_fetch_in_pages, + row_overflow_fetch_in_pages, + leaf_insert_count, + leaf_update_count, + leaf_delete_count, + leaf_ghost_count, + nonleaf_insert_count, + nonleaf_update_count, + nonleaf_delete_count, + leaf_allocation_count, + nonleaf_allocation_count, + row_lock_count, + row_lock_wait_count, + row_lock_wait_in_ms, + page_lock_count, + page_lock_wait_count, + page_lock_wait_in_ms, + index_lock_promotion_attempt_count, + index_lock_promotion_count, + page_latch_wait_count, + page_latch_wait_in_ms, + tree_page_latch_wait_count, + tree_page_latch_wait_in_ms, + page_io_latch_wait_count, + page_io_latch_wait_in_ms, + page_compression_attempt_count, + page_compression_success_count + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id integer, + @object_id integer', + @database_id, + @object_id; + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #operational_stats', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#operational_stats', + os.* + FROM #operational_stats AS os + OPTION(RECOMPILE); + + RAISERROR('Generating #index_details insert', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql += N' + SELECT + database_id = @database_id, + database_name = DB_NAME(@database_id), + t.object_id, + i.index_id, + s.schema_id, + schema_name = s.name, + table_name = t.name, + index_name = ISNULL(i.name, t.name + N''.Heap''), + column_name = c.name, + i.is_primary_key, + i.is_unique, + i.is_unique_constraint, + is_indexed_view = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.objects AS so + WHERE i.object_id = so.object_id + AND so.is_ms_shipped = 0 + AND so.type = ''V'' + ) + THEN 1 + ELSE 0 + END, + is_foreign_key = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.foreign_key_columns AS f + WHERE f.parent_column_id = c.column_id + AND f.parent_object_id = c.object_id + ) + THEN 1 + ELSE 0 + END, + is_foreign_key_reference = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.foreign_key_columns AS f + WHERE f.referenced_column_id = c.column_id + AND f.referenced_object_id = c.object_id + ) + THEN 1 + ELSE 0 + END, + ic.key_ordinal, + ic.index_column_id, + ic.is_descending_key, + ic.is_included_column, + i.filter_definition, + is_max_length = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.types AS t + WHERE c.system_type_id = t.system_type_id + AND c.user_type_id = t.user_type_id + AND t.name IN (N''varchar'', N''nvarchar'') + AND t.max_length = -1 + ) + THEN 1 + ELSE 0 + END, + user_seeks = ISNULL(us.user_seeks, 0), + user_scans = ISNULL(us.user_scans, 0), + user_lookups = ISNULL(us.user_lookups, 0), + user_updates = ISNULL(us.user_updates, 0), + us.last_user_seek, + us.last_user_scan, + us.last_user_lookup, + us.last_user_update, + is_eligible_for_dedupe = + CASE + WHEN i.type = 2 + THEN 1 + WHEN i.type = 1 + THEN 0 + END + FROM ' + QUOTENAME(@database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.indexes AS i + ON t.object_id = i.object_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.index_columns AS ic + ON i.object_id = ic.object_id + AND i.index_id = ic.index_id + JOIN ' + QUOTENAME(@database_name) + + CONVERT + ( + nvarchar(MAX), + N'.sys.columns AS c + ON ic.object_id = c.object_id + AND ic.column_id = c.column_id + LEFT JOIN sys.dm_db_index_usage_stats AS us + ON i.object_id = us.object_id + AND i.index_id = us.index_id + AND us.database_id = @database_id + WHERE t.is_ms_shipped = 0 + AND i.type IN (1, 2) + AND i.is_disabled = 0 + AND i.is_hypothetical = 0 + AND EXISTS + ( + SELECT + 1/0 + FROM #filtered_objects AS fo + WHERE fo.database_id = @database_id + AND fo.object_id = t.object_id + ) + AND EXISTS + ( + SELECT + 1/0 + FROM ' + ) + QUOTENAME(@database_name) + + CONVERT + ( + nvarchar(MAX), + N'.sys.dm_db_partition_stats ps + WHERE ps.object_id = t.object_id + AND ps.index_id = 1 + AND ps.row_count >= @min_rows + )' + ); + + IF @object_id IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding object+id filter', 0, 0) WITH NOWAIT; + END; + + SELECT @sql += N' + AND t.object_id = @object_id'; + END; + + SELECT + @sql += CONVERT + ( + nvarchar(max), + N' + AND NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@database_name) + N'.sys.objects AS so + WHERE i.object_id = so.object_id + AND so.is_ms_shipped = 0 + AND so.type = N''TF'' + ) + OPTION(RECOMPILE); + ' + ); + + IF @debug = 1 + BEGIN + PRINT SUBSTRING(@sql, 1, 4000); + PRINT SUBSTRING(@sql, 4000, 8000); + END; + + INSERT + #index_details + WITH + (TABLOCK) + ( + database_id, + database_name, + object_id, + index_id, + schema_id, + schema_name, + table_name, + index_name, + column_name, + is_primary_key, + is_unique, + is_unique_constraint, + is_indexed_view, + is_foreign_key, + is_foreign_key_reference, + key_ordinal, + index_column_id, + is_descending_key, + is_included_column, + filter_definition, + is_max_length, + user_seeks, + user_scans, + user_lookups, + user_updates, + last_user_seek, + last_user_scan, + last_user_lookup, + last_user_update, + is_eligible_for_dedupe + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id integer, + @object_id integer, + @min_rows integer', + @database_id, + @object_id, + @min_rows; + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #index_details', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_details', + * + FROM #index_details AS id; + + RAISERROR('Generating #partition_stats insert', 0, 0) WITH NOWAIT; + END; + + SELECT + @sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;'; + + SELECT + @sql += N' + SELECT + database_id = @database_id, + database_name = DB_NAME(@database_id), + x.object_id, + x.index_id, + x.schema_id, + x.schema_name, + x.table_name, + x.index_name, + x.partition_id, + x.partition_number, + x.total_rows, + x.total_space_gb, + x.reserved_lob_gb, + x.reserved_row_overflow_gb, + x.data_compression_desc, + built_on = + ISNULL + ( + psfg.partition_scheme_name, + psfg.filegroup_name + ), + psfg.partition_function_name, + pc.partition_columns + FROM + ( + SELECT DISTINCT + ps.object_id, + ps.index_id, + s.schema_id, + schema_name = s.name, + table_name = t.name, + index_name = ISNULL(i.name, t.name + N''.Heap''), + ps.partition_id, + p.partition_number, + total_rows = ps.row_count, + total_space_gb = SUM(a.total_pages) * 8 / 1024.0 / 1024.0, /* Convert directly to GB */ + reserved_lob_gb = SUM(ps.lob_reserved_page_count) * 8. / 1024. / 1024.0, /* Convert directly to GB */ + reserved_row_overflow_gb = SUM(ps.row_overflow_reserved_page_count) * 8. / 1024. / 1024.0, /* Convert directly to GB */ + p.data_compression_desc, + i.data_space_id + FROM ' + QUOTENAME(@database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@database_name) + N'.sys.indexes AS i + ON t.object_id = i.object_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.partitions AS p + ON i.object_id = p.object_id + AND i.index_id = p.index_id + JOIN ' + QUOTENAME(@database_name) + N'.sys.allocation_units AS a + ON p.partition_id = a.container_id + LEFT HASH JOIN ' + QUOTENAME(@database_name) + N'.sys.dm_db_partition_stats AS ps + ON p.partition_id = ps.partition_id + WHERE t.type <> N''TF'' + AND i.type IN (1, 2) + AND EXISTS + ( + SELECT + 1/0 + FROM #filtered_objects AS fo + WHERE fo.database_id = @database_id + AND fo.object_id = t.object_id + )'; + + IF @object_id IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('adding in object_id filter', 0, 0) WITH NOWAIT; + END; + + SELECT @sql += N' + AND t.object_id = @object_id'; + END; + + SELECT + @sql += N' + GROUP BY + ps.object_id, + ps.index_id, + s.schema_id, + s.name, + t.name, + i.name, + ps.partition_id, + p.partition_number, + ps.row_count, + p.data_compression_desc, + i.data_space_id + ) AS x + OUTER APPLY + ( + SELECT + filegroup_name = + fg.name, + partition_scheme_name = + ps.name, + partition_function_name = + pf.name + FROM ' + QUOTENAME(@database_name) + N'.sys.filegroups AS fg + FULL JOIN ' + QUOTENAME(@database_name) + N'.sys.partition_schemes AS ps + ON ps.data_space_id = fg.data_space_id + LEFT JOIN ' + QUOTENAME(@database_name) + N'.sys.partition_functions AS pf + ON pf.function_id = ps.function_id + WHERE x.data_space_id = fg.data_space_id + OR x.data_space_id = ps.data_space_id + ) AS psfg + OUTER APPLY + ( + SELECT + partition_columns = + STUFF + ( + ( + SELECT + N'', '' + + c.name + FROM ' + QUOTENAME(@database_name) + N'.sys.index_columns AS ic + JOIN ' + QUOTENAME(@database_name) + N'.sys.columns AS c + ON c.object_id = ic.object_id + AND c.column_id = ic.column_id + WHERE ic.object_id = x.object_id + AND ic.index_id = x.index_id + AND ic.partition_ordinal > 0 + ORDER BY + ic.partition_ordinal + FOR + XML + PATH(''''), + TYPE + ).value(''.'', ''nvarchar(max)''), + 1, + 2, + '''' + ) + ) AS pc + OPTION(RECOMPILE); + '; + + IF @debug = 1 + BEGIN + PRINT SUBSTRING(@sql, 1, 4000); + PRINT SUBSTRING(@sql, 4000, 8000); + END; + + INSERT + #partition_stats WITH(TABLOCK) + ( + database_id, + database_name, + object_id, + index_id, + schema_id, + schema_name, + table_name, + index_name, + partition_id, + partition_number, + total_rows, + total_space_gb, + reserved_lob_gb, + reserved_row_overflow_gb, + data_compression_desc, + built_on, + partition_function_name, + partition_columns + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id integer, + @object_id integer', + @database_id, + @object_id; + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #partition_stats', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#partition_stats', + * + FROM #partition_stats AS ps + OPTION(RECOMPILE); + + RAISERROR('Performing #index_analysis insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_analysis + WITH + (TABLOCK) + ( + database_id, + database_name, + schema_id, + schema_name, + table_name, + object_id, + index_id, + index_name, + is_unique, + key_columns, + included_columns, + filter_definition, + original_index_definition + ) + SELECT + @database_id, + database_name = DB_NAME(@database_id), + id1.schema_id, + id1.schema_name, + id1.table_name, + id1.object_id, + id1.index_id, + id1.index_name, + id1.is_unique, + key_columns = + STUFF + ( + ( + SELECT + N', ' + + id2.column_name + + CASE + WHEN id2.is_descending_key = 1 + THEN N' DESC' + ELSE N'' + END + FROM #index_details id2 + WHERE id2.object_id = id1.object_id + AND id2.index_id = id1.index_id + AND id2.is_included_column = 0 + GROUP BY + id2.column_name, + id2.is_descending_key, + id2.key_ordinal + ORDER BY + id2.key_ordinal + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ), + included_columns = + STUFF + ( + ( + SELECT + N', ' + + id2.column_name + FROM #index_details id2 + WHERE id2.object_id = id1.object_id + AND id2.index_id = id1.index_id + AND id2.is_included_column = 1 + GROUP BY + id2.column_name + ORDER BY + id2.column_name + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ), + id1.filter_definition, + /* Store the original index definition for validation */ + original_index_definition = + CASE + /* For unique constraints, use ALTER TABLE ADD CONSTRAINT syntax */ + WHEN id1.is_unique_constraint = 1 + THEN + N'ALTER TABLE ' + + QUOTENAME(DB_NAME(@database_id)) + + N'.' + + QUOTENAME(id1.schema_name) + + N'.' + + QUOTENAME(id1.table_name) + + N' ADD CONSTRAINT ' + + QUOTENAME(id1.index_name) + + N' UNIQUE (' + /* For regular indexes, use CREATE INDEX syntax */ + ELSE + N'CREATE ' + + CASE WHEN id1.is_unique = 1 THEN N'UNIQUE ' ELSE N'' END + + N'INDEX ' + + QUOTENAME(id1.index_name) + + N' ON ' + + QUOTENAME(DB_NAME(@database_id)) + + N'.' + + QUOTENAME(id1.schema_name) + + N'.' + + QUOTENAME(id1.table_name) + + N' (' + END + + STUFF + ( + ( + SELECT + N', ' + + id2.column_name + + CASE + WHEN id2.is_descending_key = 1 + THEN N' DESC' + ELSE N'' + END + FROM #index_details id2 + WHERE id2.object_id = id1.object_id + AND id2.index_id = id1.index_id + AND id2.is_included_column = 0 + GROUP BY + id2.column_name, + id2.is_descending_key, + id2.key_ordinal + ORDER BY + id2.key_ordinal + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ) + + N')' + + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details id3 + WHERE id3.object_id = id1.object_id + AND id3.index_id = id1.index_id + AND id3.is_included_column = 1 + ) + THEN N' INCLUDE (' + + STUFF + ( + ( + SELECT + N', ' + + id4.column_name + FROM #index_details id4 + WHERE id4.object_id = id1.object_id + AND id4.index_id = id1.index_id + AND id4.is_included_column = 1 + GROUP BY + id4.column_name + ORDER BY + id4.column_name + FOR + XML + PATH(''), + TYPE + ).value('text()[1]','nvarchar(max)'), + 1, + 2, + '' + ) + + N')' + ELSE N'' + END + + CASE + WHEN id1.filter_definition IS NOT NULL + THEN N' WHERE ' + id1.filter_definition + ELSE N'' + END + FROM #index_details id1 + WHERE id1.is_eligible_for_dedupe = 1 + GROUP BY + id1.schema_name, + id1.schema_id, + id1.table_name, + id1.index_name, + id1.index_id, + id1.is_unique, + id1.object_id, + id1.index_id, + id1.filter_definition, + id1.is_unique_constraint + OPTION(RECOMPILE); + + IF ROWCOUNT_BIG() = 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('No rows inserted into #index_analysis', 0, 0) WITH NOWAIT; + END; + END; + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + + RAISERROR('Starting updates', 0, 0) WITH NOWAIT; + END; + + /* Calculate index priority scores based on actual columns that exist */ + UPDATE + #index_analysis + SET + #index_analysis.index_priority = + CASE + WHEN #index_analysis.index_id = 1 + THEN 1000 /* Clustered indexes get highest priority */ + ELSE 0 + END + + + CASE + /* Unique indexes get high priority, but reduce priority for unique constraints */ + WHEN #index_analysis.is_unique = 1 AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id_uc + WHERE id_uc.index_id = #index_analysis.index_id + AND id_uc.object_id = #index_analysis.object_id + AND id_uc.is_unique_constraint = 1 + ) THEN 500 + /* Unique constraints get lower priority */ + WHEN #index_analysis.is_unique = 1 AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id_uc + WHERE id_uc.index_id = #index_analysis.index_id + AND id_uc.object_id = #index_analysis.object_id + AND id_uc.is_unique_constraint = 1 + ) THEN 50 + ELSE 0 + END + + + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id + WHERE id.index_id = #index_analysis.index_id + AND id.object_id = #index_analysis.object_id + AND id.user_seeks > 0 + ) THEN 200 + ELSE 0 + END /* Indexes with seeks get priority */ + + + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id + WHERE id.index_id = #index_analysis.index_id + AND id.object_id = #index_analysis.object_id + AND id.user_scans > 0 + ) THEN 100 ELSE 0 + END + OPTION(RECOMPILE); /* Indexes with scans get some priority */ + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after priority score', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 1: Identify unused indexes */ + UPDATE + #index_analysis + SET + #index_analysis.consolidation_rule = + CASE + WHEN @uptime_warning = 1 + THEN 'Unused Index (WARNING: Server uptime < 14 days - usage data may be incomplete)' + ELSE 'Unused Index' + END, + #index_analysis.action = N'DISABLE' + WHERE EXISTS + ( + SELECT + 1/0 + FROM #index_details id + WHERE id.database_id = #index_analysis.database_id + AND id.object_id = #index_analysis.object_id + AND id.index_id = #index_analysis.index_id + AND id.user_seeks = 0 + AND id.user_scans = 0 + AND id.user_lookups = 0 + AND id.is_primary_key = 0 /* Don't disable primary keys */ + AND id.is_unique_constraint = 0 /* Don't disable unique constraints */ + AND id.is_eligible_for_dedupe = 1 /* Only eligible indexes */ + ) + AND #index_analysis.index_id <> 1 + OPTION(RECOMPILE); /* Don't disable clustered indexes */ + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 1', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 2: Exact duplicates - matching key columns and includes */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Exact Duplicate', + ia1.target_index_name = + CASE + WHEN ia1.index_priority > ia2.index_priority + THEN NULL /* This index is the keeper */ + WHEN ia1.index_priority = ia2.index_priority + AND ia1.index_name < ia2.index_name + THEN NULL /* When tied, use alphabetical ordering for consistency */ + ELSE ia2.index_name /* Other index is the keeper */ + END, + ia1.action = + CASE + WHEN ia1.index_priority > ia2.index_priority + THEN N'KEEP' /* This index is the keeper */ + WHEN ia1.index_priority = ia2.index_priority + AND ia1.index_name < ia2.index_name + THEN N'KEEP' /* When tied, use alphabetical ordering for consistency */ + ELSE N'DISABLE' /* Other index gets disabled */ + END + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia1.key_columns = ia2.key_columns /* Exact key match */ + AND ISNULL(ia1.included_columns, '') = ISNULL(ia2.included_columns, '') /* Exact includes match */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + /* Exclude unique constraints - we'll handle those separately in Rule 7 */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1_uc + WHERE id1_uc.database_id = ia1.database_id + AND id1_uc.object_id = ia1.object_id + AND id1_uc.index_id = ia1.index_id + AND id1_uc.is_unique_constraint = 1 + ) + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2_uc + WHERE id2_uc.database_id = ia2.database_id + AND id2_uc.object_id = ia2.object_id + AND id2_uc.index_id = ia2.index_id + AND id2_uc.is_unique_constraint = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_eligible_for_dedupe = 1 + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 2', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + + /* Special debug for exact duplicates */ + RAISERROR('Special debug for exact duplicates after rule 2:', 0, 0) WITH NOWAIT; + SELECT + ia1.index_name AS index1_name, + ia1.action AS index1_action, + ia1.consolidation_rule AS index1_rule, + ia1.index_priority AS index1_priority, + ia1.target_index_name AS index1_target, + ia1.filter_definition AS index1_filter, + ia2.index_name AS index2_name, + ia2.action AS index2_action, + ia2.consolidation_rule AS index2_rule, + ia2.index_priority AS index2_priority, + ia2.target_index_name AS index2_target, + ia2.filter_definition AS index2_filter + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia1.key_columns = ia2.key_columns /* Exact key match */ + AND ISNULL(ia1.included_columns, '') = ISNULL(ia2.included_columns, '') /* Exact includes match */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + WHERE ia1.consolidation_rule = N'Exact Duplicate' + OR ia2.consolidation_rule = N'Exact Duplicate' + ORDER BY ia1.index_name + OPTION(RECOMPILE); + END; + + /* Rule 3: Key duplicates - matching key columns, different includes */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Key Duplicate', + ia1.target_index_name = + CASE + /* If one is unique and the other isn't, prefer the unique one */ + WHEN ia1.is_unique = 1 + AND ia2.is_unique = 0 + THEN NULL + WHEN ia1.is_unique = 0 + AND ia2.is_unique = 1 + THEN ia2.index_name + /* Otherwise use priority */ + WHEN ia1.index_priority >= ia2.index_priority + THEN NULL + ELSE ia2.index_name + END, + ia1.action = + CASE + WHEN (ia1.is_unique = 1 AND ia2.is_unique = 0) + OR + ( + ia1.index_priority >= ia2.index_priority + AND NOT (ia1.is_unique = 0 AND ia2.is_unique = 1) + ) + AND ISNULL(ia1.included_columns, N'') <> ISNULL(ia2.included_columns, N'') + THEN N'MERGE INCLUDES' /* Keep this index but merge includes */ + ELSE N'DISABLE' /* Other index is keeper, disable this one */ + END, + /* For the winning index, set clear superseded_by text for the report */ + ia1.superseded_by = + CASE + WHEN (ia1.is_unique = 1 AND ia2.is_unique = 0) + OR + ( + ia1.index_priority >= ia2.index_priority + AND NOT (ia1.is_unique = 0 AND ia2.is_unique = 1) + ) + THEN N'Supersedes ' + + ia2.index_name + ELSE NULL + END + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia1.key_columns = ia2.key_columns /* Exact key match */ + AND ISNULL(ia1.included_columns, '') <> ISNULL(ia2.included_columns, '') /* Different includes */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + /* Exclude pairs where either one is a unique constraint (we'll handle those separately in Rule 7) */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1_uc + WHERE id1_uc.database_id = ia1.database_id + AND id1_uc.object_id = ia1.object_id + AND id1_uc.index_id = ia1.index_id + AND id1_uc.is_unique_constraint = 1 + ) + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2_uc + WHERE id2_uc.database_id = ia2.database_id + AND id2_uc.object_id = ia2.object_id + AND id2_uc.index_id = ia2.index_id + AND id2_uc.is_unique_constraint = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_eligible_for_dedupe = 1 + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 3', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 4: Superset/subset key columns */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Key Subset', + ia1.target_index_name = ia2.index_name, + ia1.action = N'DISABLE' /* The narrower index gets disabled */ + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia2.key_columns LIKE (ia1.key_columns + '%') /* ia2 has wider key that starts with ia1's key */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + /* Exception: If narrower index is unique and wider is not, they should not be merged */ + AND NOT (ia1.is_unique = 1 AND ia2.is_unique = 0) + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_eligible_for_dedupe = 1 + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 4', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 5: Mark superset indexes for merging with includes from subset */ + UPDATE + ia2 + SET + ia2.consolidation_rule = N'Key Superset', + ia2.action = N'MERGE INCLUDES', /* The wider index gets merged with includes */ + ia2.superseded_by = + ISNULL + ( + ia2.superseded_by + + ', ', + '' + ) + + N'Supersedes ' + + ia1.index_name + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.target_index_name = ia2.index_name /* Link from Rule 4 */ + WHERE ia1.consolidation_rule = N'Key Subset' + AND ia1.action = N'DISABLE' + AND ia2.consolidation_rule IS NULL /* Not already processed */ + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 5', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 6: Merge includes from subset to superset indexes */ + WITH + KeySubsetSuperset AS + ( + SELECT + superset.database_id, + superset.object_id, + superset.index_id, + superset.index_name, + superset.included_columns AS superset_includes, + subset.included_columns AS subset_includes + FROM #index_analysis AS superset + JOIN #index_analysis AS subset + ON superset.database_id = subset.database_id + AND superset.object_id = subset.object_id + AND subset.target_index_name = superset.index_name + WHERE superset.action = N'MERGE INCLUDES' + AND subset.action = N'DISABLE' + AND superset.consolidation_rule = N'Key Superset' + AND subset.consolidation_rule = N'Key Subset' + ) + UPDATE + ia + SET + ia.included_columns = + CASE + /* If both have includes, combine them without duplicates */ + WHEN kss.superset_includes IS NOT NULL + AND kss.subset_includes IS NOT NULL + THEN + /* Create combined includes using XML method that works with all SQL Server versions */ + ( + SELECT + /* Combine both sets of includes */ + combined_cols = + STUFF + ( + ( + SELECT DISTINCT + N', ' + + t.c.value('.', 'sysname') + FROM + ( + /* Create XML from superset includes */ + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE(kss.superset_includes, N', ', N'') + + N'' + ) + + UNION ALL + + /* Create XML from subset includes */ + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE(kss.subset_includes, N', ', N'') + + N'' + ) + ) AS a + /* Split XML into individual columns */ + CROSS APPLY a.x.nodes('/c') AS t(c) + FOR + XML + PATH('') + ), + 1, + 2, + '' + ) + ) + /* If only subset has includes, use those */ + WHEN kss.superset_includes IS NULL + AND kss.subset_includes IS NOT NULL + THEN kss.subset_includes + /* If only superset has includes or neither has includes, keep superset's includes */ + ELSE kss.superset_includes + END + FROM #index_analysis AS ia + JOIN KeySubsetSuperset AS kss + ON ia.database_id = kss.database_id + AND ia.object_id = kss.object_id + AND ia.index_id = kss.index_id + WHERE ia.action = N'MERGE INCLUDES' + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 6', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Update the superseded_by column for the wider index in a separate statement */ + UPDATE + ia2 + SET + ia2.superseded_by = N'Supersedes ' + ia1.index_name + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name <> ia2.index_name + AND ia2.key_columns LIKE (ia1.key_columns + N'%') /* ia2 has wider key that starts with ia1's key */ + AND ISNULL(ia1.filter_definition, '') = ISNULL(ia2.filter_definition, '') /* Matching filters */ + /* Exception: If narrower index is unique and wider is not, they should not be merged */ + AND NOT (ia1.is_unique = 1 AND ia2.is_unique = 0) + WHERE ia1.consolidation_rule = N'Key Subset' /* Use records just processed in previous UPDATE */ + AND ia1.target_index_name = ia2.index_name /* Make sure we're updating the right wider index */ + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after update superseded', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 7: Unique constraint vs. nonclustered index handling */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Unique Constraint Replacement', + ia1.action = + CASE + WHEN ia1.is_unique = 0 + THEN 'MAKE UNIQUE' /* Convert to unique index */ + ELSE 'KEEP' /* Already unique, so just keep it */ + END + FROM #index_analysis AS ia1 + WHERE ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia1.action IS NULL /* Not already processed by earlier rules */ + AND EXISTS + ( + /* Find nonclustered indexes */ + SELECT + 1/0 + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_eligible_for_dedupe = 1 + ) + AND EXISTS + ( + /* Find unique constraints with matching key columns */ + SELECT + 1/0 + FROM #index_details AS id2 + WHERE id2.database_id = ia1.database_id + AND id2.object_id = ia1.object_id + AND id2.is_unique_constraint = 1 + AND NOT EXISTS + ( + /* Verify key columns match between index and unique constraint */ + SELECT + id2_inner.column_name + FROM #index_details AS id2_inner + WHERE id2_inner.database_id = id2.database_id + AND id2_inner.object_id = id2.object_id + AND id2_inner.index_id = id2.index_id + AND id2_inner.is_included_column = 0 + + EXCEPT + + SELECT + id1_inner.column_name + FROM #index_details AS id1_inner + WHERE id1_inner.database_id = ia1.database_id + AND id1_inner.object_id = ia1.object_id + AND id1_inner.index_id = ia1.index_id + AND id1_inner.is_included_column = 0 + ) + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 7', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 7.5: Mark unique constraints that have matching nonclustered indexes for disabling */ + /* First, mark unique constraints for disabling */ + UPDATE + ia_uc + SET + ia_uc.consolidation_rule = N'Unique Constraint Replacement', + ia_uc.action = N'DISABLE', /* Mark unique constraint for disabling */ + ia_uc.target_index_name = ia_nc.index_name /* Point to the nonclustered index that will replace it */ + FROM #index_analysis AS ia_uc /* Unique constraint */ + JOIN #index_details AS id_uc /* Join to get unique constraint details */ + ON id_uc.database_id = ia_uc.database_id + AND id_uc.object_id = ia_uc.object_id + AND id_uc.index_id = ia_uc.index_id + AND id_uc.is_unique_constraint = 1 /* This is a unique constraint */ + JOIN #index_analysis AS ia_nc /* Join to find nonclustered index */ + ON ia_nc.database_id = ia_uc.database_id + AND ia_nc.object_id = ia_uc.object_id + AND ia_nc.index_name <> ia_uc.index_name /* Different index */ + WHERE + /* Verify key columns EXACT match between index and unique constraint */ + ia_uc.key_columns = ia_nc.key_columns + OPTION(RECOMPILE); + + /* Second, mark nonclustered indexes to be made unique */ + UPDATE + ia_nc + SET + ia_nc.consolidation_rule = N'Unique Constraint Replacement', + ia_nc.action = N'MAKE UNIQUE', /* Mark nonclustered index to be made unique */ + /* CRITICAL: Set target_index_name to NULL to ensure it gets a MERGE script */ + ia_nc.target_index_name = NULL + FROM #index_analysis AS ia_nc /* Nonclustered index */ + JOIN #index_details AS id_nc /* Join to get nonclustered index details */ + ON id_nc.database_id = ia_nc.database_id + AND id_nc.object_id = ia_nc.object_id + AND id_nc.index_id = ia_nc.index_id + AND id_nc.is_unique_constraint = 0 /* This is not a unique constraint */ + WHERE + /* Two conditions for matching: + 1. Index key columns exactly match a unique constraint's key columns + 2. A unique constraint is already marked for DISABLE and has this index as target */ + EXISTS + ( + /* Find unique constraint with matching keys that should be disabled */ + SELECT + 1/0 + FROM #index_analysis AS ia_uc + JOIN #index_details AS id_uc + ON id_uc.database_id = ia_uc.database_id + AND id_uc.object_id = ia_uc.object_id + AND id_uc.index_id = ia_uc.index_id + AND id_uc.is_unique_constraint = 1 + WHERE ia_uc.database_id = ia_nc.database_id + AND ia_uc.object_id = ia_nc.object_id + /* Check that both indexes have EXACTLY the same key columns */ + AND ia_uc.key_columns = ia_nc.key_columns + ) + OPTION(RECOMPILE); + + /* CRITICAL: Ensure that only the unique constraints that exactly match get this treatment */ + /* And remove any incorrect MAKE UNIQUE actions */ + UPDATE + ia + SET + action = NULL, + consolidation_rule = NULL, + target_index_name = NULL + FROM #index_analysis AS ia + WHERE ia.action = N'MAKE UNIQUE' + AND NOT EXISTS ( + /* Check if there's a unique constraint with matching keys that points to this index */ + SELECT 1 + FROM #index_analysis AS ia_uc + WHERE ia_uc.database_id = ia.database_id + AND ia_uc.object_id = ia.object_id + AND ia_uc.key_columns = ia.key_columns + AND ia_uc.action = N'DISABLE' + AND ia_uc.target_index_name = ia.index_name + ) + OPTION(RECOMPILE); + + /* Make sure the nonclustered index has the superseded_by field set correctly */ + UPDATE + ia_nc + SET + ia_nc.superseded_by = + CASE + WHEN ia_nc.superseded_by IS NULL + THEN N'Will replace constraint ' + + ia_uc.index_name + ELSE ia_nc.superseded_by + + N', will replace constraint ' + ia_uc.index_name + END + FROM #index_analysis AS ia_nc + JOIN #index_analysis AS ia_uc + ON ia_uc.database_id = ia_nc.database_id + AND ia_uc.object_id = ia_nc.object_id + AND ia_uc.action = N'DISABLE' + AND ia_uc.target_index_name = ia_nc.index_name + WHERE ia_nc.action = N'MAKE UNIQUE' + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 7.5', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Rule 8: Identify indexes with same keys but in different order after first column */ + /* This rule flags indexes that have the same set of key columns but ordered differently */ + /* These need manual review as they may be redundant depending on query patterns */ + UPDATE + ia1 + SET + ia1.consolidation_rule = N'Same Keys Different Order', + ia1.action = N'REVIEW', /* These need manual review */ + ia1.target_index_name = ia2.index_name /* Reference the partner index */ + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.index_name < ia2.index_name /* Only process each pair once */ + AND ia1.consolidation_rule IS NULL /* Not already processed */ + AND ia2.consolidation_rule IS NULL /* Not already processed */ + WHERE + /* Leading columns match */ + EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id1 + JOIN #index_details AS id2 + ON id1.database_id = id2.database_id + AND id1.object_id = id2.object_id + AND id1.column_name = id2.column_name + AND id1.key_ordinal = 1 + AND id2.key_ordinal = 1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id2.index_id = ia2.index_id + ) + /* Same set of key columns but in different order */ + AND NOT EXISTS + ( + /* Make sure the sets of key columns are exactly the same */ + SELECT + id1.column_name + FROM #index_details AS id1 + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id1.is_included_column = 0 + AND id1.key_ordinal > 0 + + EXCEPT + + SELECT + id2.column_name + FROM #index_details AS id2 + WHERE id2.database_id = ia2.database_id + AND id2.object_id = ia2.object_id + AND id2.index_id = ia2.index_id + AND id2.is_included_column = 0 + AND id2.key_ordinal > 0 + ) + /* But the order is different (excluding the first column) */ + AND EXISTS + ( + /* There's at least one column in a different position */ + SELECT + 1/0 + FROM #index_details AS id1 + JOIN #index_details AS id2 + ON id1.database_id = id2.database_id + AND id1.object_id = id2.object_id + AND id1.column_name = id2.column_name + AND id1.key_ordinal <> id2.key_ordinal + AND id1.key_ordinal > 1 /* After the first column */ + AND id2.key_ordinal > 1 /* After the first column */ + WHERE id1.database_id = ia1.database_id + AND id1.object_id = ia1.object_id + AND id1.index_id = ia1.index_id + AND id2.index_id = ia2.index_id + ) + OPTION(RECOMPILE); + + IF @debug = 1 + BEGIN + SELECT + table_name = '#index_analysis after rule 8', + ia.* + FROM #index_analysis AS ia + OPTION(RECOMPILE); + END; + + /* Create a reference to the detailed summary that will appear at the end */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + consolidation_rule, + script_type, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT + result_type = 'SUMMARY', + sort_order = 1, + database_name = '', + schema_name = '', + table_name = '', + index_name = '', + consolidation_rule = N'', + script_type = 'Index Cleanup Scripts', + additional_info = N'A detailed index analysis report appears after these scripts', + target_index_name = '', + superseded_info = '', + original_index_definition = '', + index_size_gb = 0, + index_rows = 0, + index_reads = 0, + index_writes = 0 + OPTION(RECOMPILE); + + + /* Identify key duplicates where both indexes have MERGE INCLUDES action */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #key_duplicate_dedupe insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #key_duplicate_dedupe + WITH + (TABLOCK) + ( + database_id, + object_id, + database_name, + schema_name, + table_name, + base_key_columns, + filter_definition, + winning_index_name, + index_list + ) + SELECT + ia.database_id, + ia.object_id, + database_name = MAX(ia.database_name), + schema_name = MAX(ia.schema_name), + table_name = MAX(ia.table_name), + base_key_columns = ia.key_columns, + filter_definition = ISNULL(ia.filter_definition, N''), + /* Choose the index with most included columns as the winner (or first alphabetically if tied) */ + winning_index_name = + ( + SELECT TOP (1) + candidate.index_name + FROM #index_analysis AS candidate + WHERE candidate.database_id = ia.database_id + AND candidate.object_id = ia.object_id + AND candidate.key_columns = ia.key_columns + AND ISNULL(candidate.filter_definition, '') = ISNULL(ia.filter_definition, '') + AND candidate.action = N'MERGE INCLUDES' + AND candidate.consolidation_rule = N'Key Duplicate' + ORDER BY + /* Then prefer indexes with more included columns (by length as a proxy) */ + LEN(ISNULL(candidate.included_columns, '')) DESC, + /* Then alphabetically for stability */ + candidate.index_name + ), + /* Build a list of other indexes in this group */ + index_list = + STUFF + ( + ( + SELECT + N', ' + + inner_ia.index_name + FROM #index_analysis AS inner_ia + WHERE inner_ia.database_id = ia.database_id + AND inner_ia.object_id = ia.object_id + AND inner_ia.key_columns = ia.key_columns + AND ISNULL(inner_ia.filter_definition, '') = ISNULL(ia.filter_definition, '') + AND inner_ia.action = N'MERGE INCLUDES' + AND inner_ia.consolidation_rule = N'Key Duplicate' + GROUP BY + inner_ia.index_name + ORDER BY + inner_ia.index_name + FOR + XML + PATH(''), + TYPE + ).value('.', 'nvarchar(max)'), + 1, + 2, + '' + ) + FROM #index_analysis AS ia + WHERE ia.action = N'MERGE INCLUDES' + AND ia.consolidation_rule = N'Key Duplicate' + GROUP BY + ia.database_id, + ia.object_id, + ia.key_columns, + ia.filter_definition + HAVING + COUNT_BIG(*) > 1 + OPTION(RECOMPILE); /* Only groups with multiple MERGE INCLUDES */ + + /* Update the index_analysis table to make only one index the winner in each group */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_analysis updates', 0, 0) WITH NOWAIT; + END; + + UPDATE + ia + SET + ia.action = N'DISABLE', + ia.target_index_name = kdd.winning_index_name, + ia.superseded_by = NULL + FROM #index_analysis AS ia + JOIN #key_duplicate_dedupe AS kdd + ON ia.database_id = kdd.database_id + AND ia.object_id = kdd.object_id + AND ia.key_columns = kdd.base_key_columns + AND ISNULL(ia.filter_definition, N'') = kdd.filter_definition + WHERE ia.index_name <> kdd.winning_index_name + AND ia.action = N'MERGE INCLUDES' + AND ia.consolidation_rule = N'Key Duplicate' + OPTION(RECOMPILE); + + /* Update the winning index's superseded_by to list all other indexes */ + UPDATE + ia + SET + ia.superseded_by = N'Supersedes ' + + REPLACE + ( + kdd.index_list, + ia.index_name + N', ', + N'' + ) /* Remove self from list if present */ + FROM #index_analysis AS ia + JOIN #key_duplicate_dedupe AS kdd + ON ia.database_id = kdd.database_id + AND ia.object_id = kdd.object_id + AND ia.key_columns = kdd.base_key_columns + AND ISNULL(ia.filter_definition, '') = kdd.filter_definition + WHERE ia.index_name = kdd.winning_index_name + OPTION(RECOMPILE); + + /* Find indexes with same key columns where one has includes that are a subset of another */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #include_subset_dedupe insert', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #include_subset_dedupe + WITH + (TABLOCK) + ( + database_id, + object_id, + subset_index_name, + superset_index_name, + subset_included_columns, + superset_included_columns + ) + SELECT + ia1.database_id, + ia1.object_id, + ia1.index_name AS subset_index_name, + ia2.index_name AS superset_index_name, + ia1.included_columns AS subset_included_columns, + ia2.included_columns AS superset_included_columns + FROM #index_analysis AS ia1 + JOIN #index_analysis AS ia2 + ON ia1.database_id = ia2.database_id + AND ia1.object_id = ia2.object_id + AND ia1.key_columns = ia2.key_columns + AND ISNULL(ia1.filter_definition, N'') = ISNULL(ia2.filter_definition, N'') + AND ia1.index_name <> ia2.index_name + AND ia1.action = N'MERGE INCLUDES' + AND ia2.action = N'MERGE INCLUDES' + AND ia1.consolidation_rule = N'Key Duplicate' + AND ia2.consolidation_rule = N'Key Duplicate' + /* Find where subset's includes are contained within superset's includes */ + AND + ( + ia1.included_columns IS NULL + OR CHARINDEX(ia1.included_columns, ia2.included_columns) > 0 + ) + /* Don't match if lengths are the same (would be exact duplicates) */ + AND + ( + ia1.included_columns IS NULL + OR ia2.included_columns IS NULL + OR LEN(ia1.included_columns) < LEN(ia2.included_columns) + ) + OPTION(RECOMPILE); + + /* Update the subset indexes to be disabled, since supersets already contain their columns */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_analysis updates', 0, 0) WITH NOWAIT; + END; + + UPDATE + ia + SET + ia.action = N'DISABLE', + ia.target_index_name = isd.superset_index_name, + ia.superseded_by = NULL + FROM #index_analysis AS ia + JOIN #include_subset_dedupe AS isd + ON ia.database_id = isd.database_id + AND ia.object_id = isd.object_id + AND ia.index_name = isd.subset_index_name + OPTION(RECOMPILE); + + /* Update the superset indexes to indicate they supersede the subset indexes */ + UPDATE + ia + SET + ia.superseded_by = + CASE + WHEN ia.superseded_by IS NULL + THEN N'Supersedes ' + + isd.subset_index_name + ELSE ia.superseded_by + + N', ' + + isd.subset_index_name + END + FROM #index_analysis AS ia + JOIN #include_subset_dedupe AS isd + ON ia.database_id = isd.database_id + AND ia.object_id = isd.object_id + AND ia.index_name = isd.superset_index_name + OPTION(RECOMPILE); + + /* Update winning indexes that don't actually need changes to have action = N'KEEP' */ + UPDATE + ia + SET + /* Change action to 'KEEP' for indexes that don't need to be modified */ + ia.action = N'KEEP' + FROM #index_analysis AS ia + WHERE ia.action = N'MERGE INCLUDES' + AND ia.superseded_by IS NOT NULL + /* This should indicate it already has all the needed includes */ + AND NOT EXISTS + ( + /* Find any indexes it supersedes that have includes not in this index */ + SELECT + 1/0 + FROM #index_analysis AS ia_subset + WHERE ia_subset.database_id = ia.database_id + AND ia_subset.object_id = ia.object_id + AND ia_subset.key_columns = ia.key_columns + AND ia_subset.action = N'DISABLE' + AND ia_subset.target_index_name = ia.index_name + /* This complex check handles cases where the superset doesn't contain all subset columns */ + AND CHARINDEX(ISNULL(ia_subset.included_columns, N''), ISNULL(ia.included_columns, N'')) = 0 + AND ISNULL(ia_subset.included_columns, N'') <> N'' + ) + OPTION(RECOMPILE); + + /* Insert merge scripts for indexes */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, MERGE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + target_index_name, + script, + additional_info, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'MERGE', + /* Put merge target indexes higher in sort order (5) so they appear before + indexes that will be disabled (20) */ + sort_order = 5, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = N'MERGE SCRIPT', + ia.consolidation_rule, + ia.target_index_name, + script = + CASE + WHEN ia.action = N'MAKE UNIQUE' + THEN N'CREATE UNIQUE ' + WHEN ia.action = N'MERGE INCLUDES' + THEN N'CREATE ' + ELSE N'CREATE ' + END + + N'INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + N' (' + + ia.key_columns + + N')' + + CASE + WHEN ia.included_columns IS NOT NULL + AND LEN(ia.included_columns) > 0 + AND ia.action = N'MERGE INCLUDES' + THEN N' INCLUDE (' + + ia.included_columns + + N')' + WHEN ia.included_columns IS NOT NULL + AND LEN(ia.included_columns) > 0 + THEN N' INCLUDE (' + + ia.included_columns + + N')' + ELSE N'' + END + + CASE + WHEN ia.filter_definition IS NOT NULL + THEN N' WHERE ' + + ia.filter_definition + ELSE N'' + END + + N' WITH (DROP_EXISTING = ON, FILLFACTOR = 100, SORT_IN_TEMPDB = ON, ONLINE = ' + + CASE + WHEN @online = 1 + THEN N'ON' + ELSE N'OFF' + END + + CASE + WHEN ce.can_compress = 1 + THEN ', DATA_COMPRESSION = PAGE' + ELSE N'' + END + + N')' + + CASE + WHEN ps.partition_function_name IS NOT NULL + THEN N' ON ' + + QUOTENAME(ps.partition_function_name) + + N'(' + + ISNULL(ps.partition_columns, N'') + + N')' + WHEN ps.built_on IS NOT NULL + THEN N' ON ' + + QUOTENAME(ps.built_on) + ELSE N'' + END + N';', + /* Additional info about what this script does */ + additional_info = + CASE + WHEN ia.action = N'MERGE INCLUDES' + THEN N'This index will absorb includes from duplicate indexes' + WHEN ia.action = N'MAKE UNIQUE' + THEN N'This index will replace a unique constraint' + ELSE NULL + END, + /* Add superseded_by information if available */ + ia.superseded_by, + /* Original index definition for validation */ + ia.original_index_definition, + NULL, + NULL, + NULL, + NULL + FROM #index_analysis AS ia + LEFT JOIN + ( + /* Get the partition info for each index */ + SELECT + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + FROM #partition_stats ps + GROUP BY + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + ) AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ce.can_compress = 1 + /* Only create merge scripts for the indexes that should remain after merging */ + AND ia.target_index_name IS NULL + OPTION(RECOMPILE); + + /* Debug which indexes are getting MERGE scripts */ + IF @debug = 1 + BEGIN + RAISERROR('Indexes getting MERGE scripts:', 0, 0) WITH NOWAIT; + SELECT + ia.index_name, + ia.action, + ia.consolidation_rule, + ia.target_index_name, + script_type = 'WILL GET MERGE SCRIPT', + ia.included_columns + FROM #index_analysis AS ia + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ce.can_compress = 1 + AND ia.target_index_name IS NULL + ORDER BY + ia.index_name + OPTION(RECOMPILE); + END; + + /* Insert disable scripts for unneeded indexes */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, DISABLE', 0, 0) WITH NOWAIT; + + /* Debug for indexes that should get DISABLE scripts */ + RAISERROR('Indexes that should get DISABLE scripts:', 0, 0) WITH NOWAIT; + SELECT + ia.index_name, + ia.consolidation_rule, + ia.action, + ia.target_index_name, + ia.is_unique, + ia.index_priority, + is_unique_constraint = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id + WHERE id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_unique_constraint = 1 + ) + THEN 'YES' + ELSE 'NO' + END, + make_unique_target = + CASE + WHEN EXISTS + ( + SELECT + 1/0 + FROM #index_analysis AS ia_make + WHERE ia_make.database_id = ia.database_id + AND ia_make.object_id = ia.object_id + AND ia_make.action = N'MAKE UNIQUE' + AND ia_make.target_index_name = ia.index_name + ) + THEN 'YES' + ELSE 'NO' + END, + will_get_script = + CASE + WHEN ia.action = N'DISABLE' + AND NOT EXISTS + ( + SELECT 1 + FROM #index_details AS id_uc + WHERE id_uc.database_id = ia.database_id + AND id_uc.object_id = ia.object_id + AND id_uc.index_id = ia.index_id + AND id_uc.is_unique_constraint = 1 + ) + THEN 'YES' + ELSE 'NO' + END + FROM #index_analysis AS ia + WHERE ia.index_name LIKE 'ix_filtered_%' + OR ia.index_name LIKE 'ix_desc_%' + ORDER BY + ia.index_name + OPTION(RECOMPILE); + + /* Debug for all indexes marked with action = DISABLE */ + RAISERROR('All indexes with action = DISABLE:', 0, 0) WITH NOWAIT; + SELECT + ia.index_name, + ia.consolidation_rule, + ia.action, + ia.target_index_name + FROM #index_analysis AS ia + WHERE ia.action = N'DISABLE' + ORDER BY + ia.index_name + OPTION(RECOMPILE); + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + script, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'DISABLE', + /* Sort duplicate/subset indexes first (20), then unused indexes last (25) */ + sort_order = + CASE + WHEN ia.consolidation_rule LIKE 'Unused Index%' THEN 25 + ELSE 20 + END, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'DISABLE SCRIPT', + ia.consolidation_rule, + script = + /* Use regular DISABLE syntax for indexes */ + N'ALTER INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + N' DISABLE;', + CASE + WHEN ia.consolidation_rule = N'Key Subset' + THEN N'This index is superseded by a wider index: ' + + ISNULL(ia.target_index_name, N'(unknown)') + WHEN ia.consolidation_rule = N'Exact Duplicate' + THEN N'This index is an exact duplicate of: ' + + ISNULL(ia.target_index_name, N'(unknown)') + WHEN ia.consolidation_rule = N'Key Duplicate' + THEN N'This index has the same keys as: ' + + ISNULL(ia.target_index_name, N'(unknown)') + WHEN ia.consolidation_rule LIKE 'Unused Index%' + THEN ia.consolidation_rule + WHEN ia.action = N'DISABLE' + THEN N'This index is redundant and will be disabled' + ELSE N'This index is redundant' + END, + ia.target_index_name, /* Include the target index name */ + superseded_info = NULL, /* Don't need superseded_by info for disabled indexes */ + /* Original index definition for validation */ + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + WHERE ia.action = N'DISABLE' + /* Exclude unique constraints - they are handled by DISABLE CONSTRAINT scripts */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_details AS id_uc + WHERE id_uc.database_id = ia.database_id + AND id_uc.object_id = ia.object_id + AND id_uc.index_id = ia.index_id + AND id_uc.is_unique_constraint = 1 + ) + /* Also exclude any index that is also going to be made unique in rule 7.5 */ + AND NOT EXISTS + ( + SELECT + 1/0 + FROM #index_analysis AS ia_unique + WHERE ia_unique.database_id = ia.database_id + AND ia_unique.object_id = ia.object_id + AND ia_unique.index_name = ia.index_name + AND ia_unique.action = N'MAKE UNIQUE' + ) + OPTION(RECOMPILE); + + /* Insert compression scripts for remaining indexes */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, COMPRESS', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + script, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'COMPRESS', + sort_order = 40, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'COMPRESSION SCRIPT', + script = + N'ALTER INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + CASE + WHEN ps.partition_function_name IS NOT NULL + THEN N' REBUILD PARTITION = ALL' + ELSE N' REBUILD' + END + + N' WITH (FILLFACTOR = 100, SORT_IN_TEMPDB = ON, ONLINE = ' + + CASE + WHEN @online = 1 + THEN N'ON' + ELSE N'OFF' + END + + CASE + WHEN ce.can_compress = 1 + THEN ', DATA_COMPRESSION = PAGE' + ELSE N'' + END + + N')', + additional_info = N'Compression type: All Partitions', + superseded_info = NULL, /* No target index for compression scripts */ + ia.superseded_by, /* Include superseded_by info for compression scripts */ + /* Original index definition for validation */ + ia.original_index_definition, + ps_full.total_space_gb, + ps_full.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN + ( + /* Get the partition info for each index */ + SELECT + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + FROM #partition_stats ps + GROUP BY + ps.database_id, + ps.object_id, + ps.index_id, + ps.built_on, + ps.partition_function_name, + ps.partition_columns + ) + AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #partition_stats AS ps_full + ON ia.database_id = ps_full.database_id + AND ia.object_id = ps_full.object_id + AND ia.index_id = ps_full.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE + /* Indexes that are not being disabled or merged */ + (ia.action IS NULL OR ia.action = N'KEEP') + /* Only indexes eligible for compression */ + AND ce.can_compress = 1 + OPTION(RECOMPILE); + + /* Insert disable scripts for unique constraints */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, CONSTRAINT', 0, 0) WITH NOWAIT; + END; + + /* Add code to insert KEPT indexes into the results - THESE WERE MISSING! */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, KEPT', 0, 0) WITH NOWAIT; + END; + + /* Insert KEPT indexes into results */ + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + additional_info, + script, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'KEPT', + sort_order = 95, /* Put kept indexes at the end */ + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = NULL, + ia.consolidation_rule, + additional_info = + CASE + WHEN ia.consolidation_rule IS NOT NULL + THEN 'This index is being kept' + ELSE NULL + END, + script = NULL, /* No script for kept indexes */ + /* Original index definition for validation */ + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + /* Check that this index is not already in the results */ + WHERE NOT EXISTS + ( + SELECT + 1/0 + FROM #index_cleanup_results AS ir + WHERE ir.database_name = ia.database_name + AND ir.schema_name = ia.schema_name + AND ir.table_name = ia.table_name + AND ir.index_name = ia.index_name + ) + /* And include only indexes that should be kept */ + AND + ( + /* Include indexes marked KEEP */ + ia.action = N'KEEP' + /* And all indexes we haven't determined an action for (not disable, merge, etc.) */ + OR + ( + ia.action IS NULL + AND ia.index_id > 0 + ) + ) + OPTION(RECOMPILE); + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + additional_info, + script, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'CONSTRAINT', + sort_order = 30, + ia_uc.database_name, + ia_uc.schema_name, + ia_uc.table_name, + ia_uc.index_name, + script_type = 'DISABLE CONSTRAINT SCRIPT', + additional_info = + N'This constraint is being replaced by: ' + + ISNULL(ia_uc.target_index_name, N'(unknown)'), + script = + N'ALTER TABLE ' + + QUOTENAME(ia_uc.database_name) + + N'.' + + QUOTENAME(ia_uc.schema_name) + + N'.' + + QUOTENAME(ia_uc.table_name) + + N' NOCHECK CONSTRAINT ' + + QUOTENAME(ia_uc.index_name) + + N';', + /* Original index definition for validation */ + original_index_definition = ia_uc.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id2.user_seeks + id2.user_scans + id2.user_lookups), + id2.user_updates + FROM #index_analysis AS ia_uc + JOIN #index_details AS id + ON id.database_id = ia_uc.database_id + AND id.object_id = ia_uc.object_id + AND id.index_id = ia_uc.index_id + AND id.is_unique_constraint = 1 + LEFT JOIN #index_details AS id2 + ON id2.database_id = ia_uc.database_id + AND id2.object_id = ia_uc.object_id + AND id2.index_id = ia_uc.index_id + AND id2.is_included_column = 0 /* Get only one row per index */ + AND id2.key_ordinal > 0 + LEFT JOIN #partition_stats AS ps + ON ia_uc.database_id = ps.database_id + AND ia_uc.object_id = ps.object_id + AND ia_uc.index_id = ps.index_id + WHERE + /* Only constraints that are marked for disabling */ + ia_uc.action = N'DISABLE' + /* That have consolidation_rule of 'Unique Constraint Replacement' */ + AND ia_uc.consolidation_rule = N'Unique Constraint Replacement' + OPTION(RECOMPILE); + + /* Insert per-partition compression scripts */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, COMPRESS_PARTITION', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + script, + additional_info, + target_index_name, + superseded_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'COMPRESS_PARTITION', + sort_order = 50, + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'PARTITION COMPRESSION SCRIPT', + script = + N'ALTER INDEX ' + + QUOTENAME(ia.index_name) + + N' ON ' + + QUOTENAME(ia.database_name) + + N'.' + + QUOTENAME(ia.schema_name) + + N'.' + + QUOTENAME(ia.table_name) + + N' REBUILD PARTITION = ' + + CONVERT + ( + nvarchar(20), + ps.partition_number + ) + + N' WITH (FILLFACTOR = 100, SORT_IN_TEMPDB = ON, ONLINE = ' + + CASE + WHEN @online = 1 + THEN N'ON' + ELSE N'OFF' + END + + CASE + WHEN ce.can_compress = 1 + THEN ', DATA_COMPRESSION = PAGE' + ELSE N'' + END + + N')', + N'Compression type: Per Partition | Partition: ' + + CONVERT + ( + nvarchar(20), + ps.partition_number + ) + + N' | Rows: ' + + CONVERT + ( + nvarchar(20), + ps.total_rows + ) + + N' | Size: ' + + CONVERT + ( + nvarchar(20), + CONVERT + ( + decimal(10,4), + ps.total_space_gb + ) + ) + + N' GB', + target_index_name = NULL, + superseded_info = NULL, + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + WHERE + /* Only partitioned indexes */ + ps.partition_function_name IS NOT NULL + /* Indexes that are not being disabled or merged */ + AND (ia.action IS NULL OR ia.action = N'KEEP') + /* Only indexes eligible for compression */ + AND ce.can_compress = 1 + OPTION(RECOMPILE); + + /* Insert compression ineligible info */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, INELIGIBLE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + additional_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'INELIGIBLE', + sort_order = 90, + ce.database_name, + ce.schema_name, + ce.table_name, + ce.index_name, + script_type = 'INELIGIBLE FOR COMPRESSION', + ce.reason, + /* Original index definition for validation */ + original_index_definition = + ( + SELECT TOP (1) + ia.original_index_definition + FROM #index_analysis AS ia + WHERE ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + ), + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #compression_eligibility AS ce + LEFT JOIN #partition_stats AS ps + ON ce.database_id = ps.database_id + AND ce.object_id = ps.object_id + AND ce.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ce.database_id + AND id.object_id = ce.object_id + AND id.index_id = ce.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + WHERE ce.can_compress = 0 + OPTION(RECOMPILE); + + + /* Insert indexes identified for manual review */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, REVIEW', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + target_index_name, + additional_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'REVIEW', + sort_order = 93, /* Just before KEPT indexes */ + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'NEEDS REVIEW', + ia.consolidation_rule, + ia.target_index_name, + additional_info = + CASE + WHEN ia.consolidation_rule = N'Same Keys Different Order' + THEN N'This index has the same key columns as ' + + ISNULL(ia.target_index_name, N'(unknown)') + + N' but in a different order. May be redundant depending on query patterns.' + ELSE N'This index needs manual review' + END, + /* Original index definition for validation */ + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + WHERE ia.action = N'REVIEW' + OPTION(RECOMPILE); + + + /* Insert indexes that are being kept (superset indexes and others) */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results insert, KEEP', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_cleanup_results + WITH + (TABLOCK) + ( + result_type, + sort_order, + database_name, + schema_name, + table_name, + index_name, + script_type, + consolidation_rule, + superseded_info, + additional_info, + original_index_definition, + index_size_gb, + index_rows, + index_reads, + index_writes + ) + SELECT DISTINCT + result_type = 'KEEP', + sort_order = 95, /* Just before END OF REPORT at 99 */ + ia.database_name, + ia.schema_name, + ia.table_name, + ia.index_name, + script_type = 'KEPT', + ia.consolidation_rule, + ia.superseded_by, + additional_info = + CASE + WHEN ia.superseded_by IS NOT NULL + THEN 'This index supersedes other indexes and already has all needed columns' + WHEN ia.action = N'KEEP' + THEN 'This index is being kept' + ELSE NULL + END, + /* Original index definition for validation */ + ia.original_index_definition, + ps.total_space_gb, + ps.total_rows, + index_reads = + (id.user_seeks + id.user_scans + id.user_lookups), + id.user_updates + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #index_details AS id + ON id.database_id = ia.database_id + AND id.object_id = ia.object_id + AND id.index_id = ia.index_id + AND id.is_included_column = 0 /* Get only one row per index */ + AND id.key_ordinal > 0 + WHERE ia.action = N'KEEP' + OR + ( + ia.action IS NULL + AND ia.consolidation_rule IS NULL + ) + OPTION(RECOMPILE); + + /* Insert database-level summaries */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats insert, DATABASE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_reporting_stats + WITH + (TABLOCK) + ( + summary_level, + database_name, + index_count, + total_size_gb, + total_rows, + indexes_to_merge, + unused_indexes, + unused_size_gb, + total_reads, + total_writes, + user_seeks, + user_scans, + user_lookups, + user_updates, + range_scan_count, + singleton_lookup_count, + row_lock_count, + row_lock_wait_count, + row_lock_wait_in_ms, + page_lock_count, + page_lock_wait_count, + page_lock_wait_in_ms, + page_latch_wait_count, + page_latch_wait_in_ms, + page_io_latch_wait_count, + page_io_latch_wait_in_ms, + forwarded_fetch_count, + leaf_insert_count, + leaf_update_count, + leaf_delete_count + ) + SELECT + summary_level = + 'DATABASE', + ps.database_name, + index_count = + COUNT_BIG(DISTINCT CONCAT(ps.object_id, N'.', ps.index_id)), + total_size_gb = SUM(ps.total_space_gb), + /* Use a simple aggregation to avoid double-counting */ + /* Get actual row count by grabbing the real row count from clustered index/heap per table */ + total_rows = SUM(DISTINCT d.actual_rows), + indexes_to_merge = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ia.database_id = ps.database_id + ), + /* Use count from analysis to keep consistent with SUMMARY level */ + unused_indexes = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action = N'DISABLE' + AND ia.database_id = ps.database_id + ), + unused_size_gb = + SUM + ( + CASE + WHEN id.user_seeks + id.user_scans + id.user_lookups = 0 + THEN ps.total_space_gb + ELSE 0 + END + ), + total_reads = SUM(id.user_seeks + id.user_scans + id.user_lookups), + total_writes = SUM(id.user_updates), + user_seeks = SUM(id.user_seeks), + user_scans = SUM(id.user_scans), + user_lookups = SUM(id.user_lookups), + user_updates = SUM(id.user_updates), + range_scan_count = SUM(os.range_scan_count), + singleton_lookup_count = SUM(os.singleton_lookup_count), + row_lock_count = SUM(os.row_lock_count), + row_lock_wait_count = SUM(os.row_lock_wait_count), + row_lock_wait_in_ms = SUM(os.row_lock_wait_in_ms), + page_lock_count = SUM(os.page_lock_count), + page_lock_wait_count = SUM(os.page_lock_wait_count), + page_lock_wait_in_ms = SUM(os.page_lock_wait_in_ms), + page_latch_wait_count = SUM(os.page_latch_wait_count), + page_latch_wait_in_ms = SUM(os.page_latch_wait_in_ms), + page_io_latch_wait_count = SUM(os.page_io_latch_wait_count), + page_io_latch_wait_in_ms = SUM(os.page_io_latch_wait_in_ms), + forwarded_fetch_count = SUM(os.forwarded_fetch_count), + leaf_insert_count = SUM(os.leaf_insert_count), + leaf_update_count = SUM(os.leaf_update_count), + leaf_delete_count = SUM(os.leaf_delete_count) + FROM #partition_stats AS ps + LEFT JOIN #index_details AS id + ON id.database_id = ps.database_id + AND id.object_id = ps.object_id + AND id.index_id = ps.index_id + AND id.is_included_column = 0 + AND id.key_ordinal > 0 + LEFT JOIN #operational_stats AS os + ON os.database_id = ps.database_id + AND os.object_id = ps.object_id + AND os.index_id = ps.index_id + OUTER APPLY + ( + /* Get actual row count per table using MAX from clustered index/heap */ + SELECT + actual_rows = + MAX + ( + CASE + WHEN ps2.index_id IN (0, 1) + THEN ps2.total_rows + ELSE 0 + END + ) + FROM #partition_stats AS ps2 + WHERE ps2.database_id = ps.database_id + AND ps2.object_id = ps.object_id + AND ps2.index_id IN (0, 1) + GROUP BY + ps2.object_id + ) AS d + GROUP BY + ps.database_name, + ps.database_id + OPTION(RECOMPILE); + + /* Insert table-level summaries */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats insert, TABLE', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_reporting_stats + WITH + (TABLOCK) + ( + summary_level, + database_name, + schema_name, + table_name, + index_count, + total_size_gb, + total_rows, + indexes_to_merge, + unused_indexes, + unused_size_gb, + total_reads, + total_writes, + user_seeks, + user_scans, + user_lookups, + user_updates, + range_scan_count, + singleton_lookup_count, + row_lock_count, + row_lock_wait_count, + row_lock_wait_in_ms, + page_lock_count, + page_lock_wait_count, + page_lock_wait_in_ms, + page_latch_wait_count, + page_latch_wait_in_ms, + page_io_latch_wait_count, + page_io_latch_wait_in_ms, + forwarded_fetch_count, + leaf_insert_count, + leaf_update_count, + leaf_delete_count + ) + SELECT + summary_level = 'TABLE', + ps.database_name, + ps.schema_name, + ps.table_name, + index_count = COUNT_BIG(DISTINCT ps.index_id), + total_size_gb = SUM(ps.total_space_gb), + /* Use MAX to get the row count from the clustered index or heap */ + total_rows = + MAX + ( + CASE + WHEN ps.index_id IN (0, 1) + THEN ps.total_rows + ELSE 0 + END + ), + indexes_to_merge = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + AND ia.database_id = ps.database_id + AND ia.schema_id = ps.schema_id + AND ia.object_id = ps.object_id + ), + /* Use count from analysis to keep consistent with SUMMARY level */ + unused_indexes = + ( + SELECT + COUNT_BIG(*) + FROM #index_analysis AS ia + WHERE ia.action = N'DISABLE' + AND ia.database_id = ps.database_id + AND ia.schema_id = ps.schema_id + AND ia.object_id = ps.object_id + ), + unused_size_gb = + SUM + ( + CASE + WHEN id.user_seeks + id.user_scans + id.user_lookups = 0 + THEN ps.total_space_gb + ELSE 0 + END + ), + total_reads = SUM(id.user_seeks + id.user_scans + id.user_lookups), + total_writes = SUM(id.user_updates), + user_seeks = SUM(id.user_seeks), + user_scans = SUM(id.user_scans), + user_lookups = SUM(id.user_lookups), + user_updates = SUM(id.user_updates), + range_scan_count = SUM(os.range_scan_count), + singleton_lookup_count = SUM(os.singleton_lookup_count), + row_lock_count = SUM(os.row_lock_count), + row_lock_wait_count = SUM(os.row_lock_wait_count), + row_lock_wait_in_ms = SUM(os.row_lock_wait_in_ms), + page_lock_count = SUM(os.page_lock_count), + page_lock_wait_count = SUM(os.page_lock_wait_count), + page_lock_wait_in_ms = SUM(os.page_lock_wait_in_ms), + page_latch_wait_count = SUM(os.page_latch_wait_count), + page_latch_wait_in_ms = SUM(os.page_latch_wait_in_ms), + page_io_latch_wait_count = SUM(os.page_io_latch_wait_count), + page_io_latch_wait_in_ms = SUM(os.page_io_latch_wait_in_ms), + forwarded_fetch_count = SUM(os.forwarded_fetch_count), + leaf_insert_count = SUM(os.leaf_insert_count), + leaf_update_count = SUM(os.leaf_update_count), + leaf_delete_count = SUM(os.leaf_delete_count) + FROM #partition_stats AS ps + LEFT JOIN #index_details AS id + ON id.database_id = ps.database_id + AND id.object_id = ps.object_id + AND id.index_id = ps.index_id + AND id.is_included_column = 0 + AND id.key_ordinal > 0 + LEFT JOIN #operational_stats AS os + ON os.database_id = ps.database_id + AND os.object_id = ps.object_id + AND os.index_id = ps.index_id + GROUP BY + ps.database_name, + ps.database_id, + ps.schema_name, + ps.schema_id, + ps.table_name, + ps.object_id + OPTION(RECOMPILE); + + /* We're not doing index-level summaries - focusing on database and table level reports */ + + /* + Return the consolidated results in a single result set + Results are ordered by: + 1. Summary information (overall stats, savings estimates) + 2. Merge scripts (includes merges and unique conversions) - sort_order 5 + 3. Disable scripts (for redundant indexes) - sort_order 20 + 4. Constraint scripts (for unique constraints to disable) + 5. Compression scripts (for tables eligible for compression) + 6. Partition-specific compression scripts + 7. Ineligible objects (tables that can't be compressed) + 8. Kept indexes - sort_order 95 + + Note: Merge target scripts are sorted higher in the results (sort_order 5) + so that new merged indexes are created before subset indexes are disabled. + + Within each category, indexes are sorted by size and impact for better prioritization. + */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_cleanup_results, RESULTS', 0, 0) WITH NOWAIT; + END; + + SELECT + /* First, show the information needed to understand the script */ + script_type = CASE WHEN ir.result_type = 'KEPT' AND ir.script_type IS NULL THEN 'KEPT' ELSE ir.script_type END, + ir.additional_info, + /* Then show identifying information for the index */ + ir.database_name, + ir.schema_name, + ir.table_name, + ir.index_name, + /* Then show relationship information */ + ir.consolidation_rule, + ir.target_index_name, + /* Include superseded_by info for winning indexes */ + superseded_info = + CASE + WHEN ia.superseded_by IS NOT NULL + THEN ia.superseded_by + ELSE ir.superseded_info + END, + /* Add size and usage metrics */ + index_size_gb = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0.0000' + ELSE FORMAT(ISNULL(ir.index_size_gb, 0), 'N4') + END, + index_rows = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0' + ELSE FORMAT(ISNULL(ir.index_rows, 0), 'N0') + END, + index_reads = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0' + ELSE FORMAT(ISNULL(ir.index_reads, 0), 'N0') + END, + index_writes = + CASE + WHEN ir.result_type = 'SUMMARY' + THEN '0' + ELSE FORMAT(ISNULL(ir.index_writes, 0), 'N0') + END, + ia.original_index_definition, + /* Finally show the actual script */ + ir.script + FROM + ( + /* Use a subquery with ROW_NUMBER to ensure we only get one row per index */ + SELECT *, + ROW_NUMBER() OVER( + PARTITION BY database_name, schema_name, table_name, index_name + ORDER BY result_type DESC /* Prefer non-NULL result types */ + ) AS rn + FROM #index_cleanup_results + ) AS ir + LEFT JOIN #index_analysis AS ia + ON ir.database_name = ia.database_name + AND ir.schema_name = ia.schema_name + AND ir.table_name = ia.table_name + AND ir.index_name = ia.index_name + WHERE ir.rn = 1 /* Take only the first row for each index */ + ORDER BY + ir.sort_order, + ir.database_name, + /* Within each sort_order group, prioritize by size and usage */ + CASE + /* For SUMMARY, keep the original order */ + WHEN ir.result_type = 'SUMMARY' + THEN 0 + /* For script categories, order by size and impact */ + ELSE ISNULL(ir.index_size_gb, 0) + END DESC, + CASE + /* For SUMMARY, keep the original order */ + WHEN ir.result_type = 'SUMMARY' + THEN 0 + /* For script categories, consider rows as secondary sort */ + ELSE ISNULL(ir.index_rows, 0) + END DESC, + /* Then by database, schema, table, index name for consistent ordering */ + ir.schema_name, + ir.table_name, + ir.index_name + OPTION(RECOMPILE); + + /* Insert overall summary information */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats insert, SUMMARY', 0, 0) WITH NOWAIT; + END; + + INSERT INTO + #index_reporting_stats + WITH + (TABLOCK) + ( + summary_level, + server_uptime_days, + uptime_warning, + tables_analyzed, + index_count, + indexes_to_disable, + indexes_to_merge, + avg_indexes_per_table, + space_saved_gb, + compression_min_savings_gb, + compression_max_savings_gb, + total_min_savings_gb, + total_max_savings_gb, + total_rows + ) + SELECT + summary_level = 'SUMMARY', + server_uptime_days = @uptime_days, + uptime_warning = @uptime_warning, + tables_analyzed = + COUNT_BIG(DISTINCT CONCAT(ia.database_id, N'.', ia.schema_id, N'.', ia.object_id)), + index_count = + COUNT_BIG(*), + indexes_to_disable = + SUM + ( + CASE + WHEN ia.action = N'DISABLE' + THEN 1 + ELSE 0 + END + ), + indexes_to_merge = + SUM + ( + CASE + WHEN ia.action IN (N'MERGE INCLUDES', N'MAKE UNIQUE') + THEN 1 + ELSE 0 + END + ), + avg_indexes_per_table = + COUNT_BIG(*) * 1.0 / + NULLIF + ( + COUNT_BIG(DISTINCT CONCAT(ia.database_id, N'.', ia.schema_id, N'.', ia.object_id)), + 0 + ), + /* Space savings from cleanup */ + space_saved_gb = + SUM + ( + CASE + WHEN ia.action IN (N'DISABLE', N'MERGE INCLUDES', N'MAKE UNIQUE') + THEN ps.total_space_gb + ELSE 0 + END + ), + /* Conservative compression savings estimate (20%) */ + compression_min_savings_gb = + SUM + ( + CASE + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.20 + ELSE 0 + END + ), + /* Optimistic compression savings estimate (60%) */ + compression_max_savings_gb = + SUM + ( + CASE + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.60 + ELSE 0 + END + ), + /* Total conservative savings */ + total_min_savings_gb = + SUM + ( + CASE + WHEN ia.action IN (N'DISABLE', N'MERGE INCLUDES', N'MAKE UNIQUE') + THEN ps.total_space_gb + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.20 + ELSE 0 + END + ), + /* Total optimistic savings */ + total_max_savings_gb = + SUM + ( + CASE + WHEN ia.action IN (N'DISABLE', N'MERGE INCLUDES', N'MAKE UNIQUE') + THEN ps.total_space_gb + WHEN (ia.action IS NULL OR ia.action = N'KEEP') + AND ce.can_compress = 1 + THEN ps.total_space_gb * 0.60 + ELSE 0 + END + ), + /* Get total rows from database unique tables */ + total_rows = + ( + SELECT + SUM(t.row_count) + FROM + ( + SELECT + ps_distinct.object_id, + row_count = + MAX + ( + CASE + WHEN ps_distinct.index_id IN (0, 1) + THEN ps_distinct.total_rows + ELSE 0 + END + ) + FROM #partition_stats AS ps_distinct + WHERE ps_distinct.index_id IN (0, 1) + GROUP BY + ps_distinct.object_id + ) AS t + ) + FROM #index_analysis AS ia + LEFT JOIN #partition_stats AS ps + ON ia.database_id = ps.database_id + AND ia.object_id = ps.object_id + AND ia.index_id = ps.index_id + LEFT JOIN #compression_eligibility AS ce + ON ia.database_id = ce.database_id + AND ia.object_id = ce.object_id + AND ia.index_id = ce.index_id + OPTION(RECOMPILE); + + /* Return enhanced database impact summaries */ + IF @debug = 1 + BEGIN + RAISERROR('Generating enhanced summary reports', 0, 0) WITH NOWAIT; + END; + + /* + This section now REPLACES the existing summary view rather than supplementing it + We'll modify the existing query below rather than creating new output panes + */ + + /* Return streamlined reporting statistics focused on key metrics */ + IF @debug = 1 + BEGIN + RAISERROR('Generating #index_reporting_stats, REPORT', 0, 0) WITH NOWAIT; + END; + + SELECT + /* Basic identification with enhanced naming */ + level = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN '=== OVERALL ANALYSIS ===' + ELSE irs.summary_level + END, + + /* Server info (for summary) or database name */ + database_info = + CASE + WHEN irs.summary_level = 'SUMMARY' + AND irs.uptime_warning = 1 + THEN 'WARNING: Server uptime only ' + + CONVERT(varchar(10), irs.server_uptime_days) + + ' days - usage data may be incomplete!' + WHEN irs.summary_level = 'SUMMARY' + THEN 'Server uptime: ' + + CONVERT(varchar(10), irs.server_uptime_days) + + ' days' + ELSE irs.database_name + END, + + /* Schema and table names (except for summary) */ + schema_name = ISNULL(irs.schema_name, 'N/A'), + table_name = ISNULL(irs.table_name, 'N/A'), + + /* ===== Section 1: Index Counts ===== */ + /* Tables analyzed (summary only) */ + tables_analyzed = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT(irs.tables_analyzed, 'N0') + ELSE FORMAT(0, 'N0') /* Show 0 instead of NULL */ + END, + + /* Total indexes */ + total_indexes = FORMAT(ISNULL(irs.index_count, 0), 'N0'), + + /* Removable indexes - report consistent values across levels */ + removable_indexes = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT(ISNULL(irs.indexes_to_disable, 0), 'N0') /* Indexes that will be disabled based on analysis */ + ELSE FORMAT(ISNULL(irs.unused_indexes, 0), 'N0') /* Unused indexes at database/table level */ + END, + + /* Show mergeable indexes across all levels */ + mergeable_indexes = FORMAT(ISNULL(irs.indexes_to_merge, 0), 'N0'), + + /* Percent of indexes that can be removed */ + percent_removable = + CASE + WHEN irs.summary_level = 'SUMMARY' AND irs.index_count > 0 + THEN FORMAT(100.0 * ISNULL(irs.indexes_to_disable, 0) / NULLIF(irs.index_count, 0), 'N1') + '%' + WHEN irs.index_count > 0 + THEN FORMAT(100.0 * ISNULL(irs.unused_indexes, 0) / NULLIF(irs.index_count, 0), 'N1') + '%' + ELSE '0.0%' + END, + + /* ===== Section 2: Size and Space Savings with Before/After comparison ===== */ + /* Current size in GB */ + current_size_gb = FORMAT(ISNULL(irs.total_size_gb, 0), 'N2'), + + /* Size after cleanup - added this as new metric */ + size_after_cleanup_gb = FORMAT(ISNULL(irs.total_size_gb, 0) - ISNULL(irs.space_saved_gb, 0), 'N2'), + + /* Size that can be saved through cleanup */ + space_saved_gb = + CASE + WHEN irs.summary_level = 'SUMMARY' + THEN FORMAT(ISNULL(irs.space_saved_gb, 0), 'N2') + ELSE FORMAT(ISNULL(irs.unused_size_gb, 0), 'N2') + END, + + /* Space reduction percentage - added this as new metric */ + space_reduction_percent = + CASE + WHEN ISNULL(irs.total_size_gb, 0) > 0 + THEN FORMAT((ISNULL(irs.space_saved_gb, 0) / NULLIF(irs.total_size_gb, 0)) * 100, 'N1') + '%' + ELSE '0.0%' + END, + + /* ===== Section 3: Table and Usage Statistics ===== */ + /* Row count */ + total_rows = FORMAT(ISNULL(irs.total_rows, 0), 'N0'), + + /* Total reads - combined total and breakdown */ + reads_breakdown = + CASE + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.total_reads, 0), 'N0') + + ' (' + + FORMAT(ISNULL(irs.user_seeks, 0), 'N0') + + ' seeks, ' + + FORMAT(ISNULL(irs.user_scans, 0), 'N0') + + ' scans, ' + + FORMAT(ISNULL(irs.user_lookups, 0), 'N0') + + ' lookups)' + ELSE 'N/A' + END, + + /* Total writes */ + writes = + CASE + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.total_writes, 0), 'N0') + ELSE 'N/A' + END, + + /* Daily write operations saved - added as new metric */ + daily_write_ops_saved = + CASE + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.user_updates / NULLIF(CONVERT(DECIMAL(10,2), + (SELECT TOP (1) irs2.server_uptime_days FROM #index_reporting_stats AS irs2 WHERE irs2.summary_level = 'DATABASE')), 0) * + (ISNULL(irs.unused_indexes, 0) / NULLIF(CONVERT(DECIMAL(10,2), irs.index_count), 0)), 0), 'N0') + ELSE 'N/A' + END, + + /* ===== Section 4: Consolidated Performance Metrics ===== */ + /* Total count of lock waits (row + page) */ + lock_wait_count = + CASE + WHEN irs.summary_level <> 'SUMMARY' + THEN FORMAT(ISNULL(irs.row_lock_wait_count, 0) + + ISNULL(irs.page_lock_wait_count, 0), 'N0') + ELSE '0' + END, + + /* Average lock wait time in ms */ + avg_lock_wait_ms = + CASE + WHEN irs.summary_level <> 'SUMMARY' + AND (ISNULL(irs.row_lock_wait_count, 0) + + ISNULL(irs.page_lock_wait_count, 0)) > 0 + THEN FORMAT(1.0 * (ISNULL(irs.row_lock_wait_in_ms, 0) + + ISNULL(irs.page_lock_wait_in_ms, 0)) / + NULLIF(ISNULL(irs.row_lock_wait_count, 0) + + ISNULL(irs.page_lock_wait_count, 0), 0), 'N2') + ELSE '0.00' + END, + + /* Combined latch wait time in ms */ + avg_latch_wait_ms = + CASE + WHEN irs.summary_level <> 'SUMMARY' + AND (ISNULL(irs.page_latch_wait_count, 0) + + ISNULL(irs.page_io_latch_wait_count, 0)) > 0 + THEN FORMAT(1.0 * (ISNULL(irs.page_latch_wait_in_ms, 0) + + ISNULL(irs.page_io_latch_wait_in_ms, 0)) / + NULLIF(ISNULL(irs.page_latch_wait_count, 0) + + ISNULL(irs.page_io_latch_wait_count, 0), 0), 'N2') + ELSE '0.00' + END + FROM #index_reporting_stats AS irs + WHERE irs.summary_level IN ('SUMMARY', 'DATABASE', 'TABLE') /* Filter out INDEX level */ + ORDER BY + /* Order by level - summary first */ + CASE + WHEN irs.summary_level = 'SUMMARY' THEN 0 + WHEN irs.summary_level = 'DATABASE' THEN 1 + WHEN irs.summary_level = 'TABLE' THEN 2 + ELSE 3 + END, + /* Then by database name */ + irs.database_name, + /* For tables, sort by potential savings and size */ + CASE + WHEN irs.summary_level = 'TABLE' THEN irs.unused_size_gb + ELSE 0 + END DESC, + CASE + WHEN irs.summary_level = 'TABLE' THEN irs.total_size_gb + ELSE 0 + END DESC, + /* Then by schema, table */ + irs.schema_name, + irs.table_name + OPTION(RECOMPILE); + +END TRY +BEGIN CATCH + THROW; +END CATCH; +END; /*Final End*/ +GO diff --git a/sp_LogHunter/README.md b/sp_LogHunter/README.md new file mode 100644 index 00000000..9fde86cd --- /dev/null +++ b/sp_LogHunter/README.md @@ -0,0 +1,61 @@ + + +# sp_LogHunter + +The SQL Server error log can have a lot of good information in it about what's going on, whether it's right or wrong. + +The problem is that it's hard to know *what* to look for, and what else was going on once you filter it. + +It's another notoriously bad Microsoft GUI, just like Query Store and Extended Events. + +I created sp_LogHunter to search through your error logs for the important stuff, with some configurability for you, and return everything ordered by log entry time. + +It helps you give you a fuller, better picture of any bad stuff happening. + +## Parameters + +| parameter_name | data_type | description | valid_inputs | defaults | +|----------------------|-----------|------------------------------------------------|------------------------------------------------------------------------------|--------------| +| @days_back | integer | how many days back you want to search the logs | an integer; will be converted to a negative number automatically | -7 | +| @start_date | datetime | if you want to search a specific time frame | a datetime value | NULL | +| @end_date | datetime | if you want to search a specific time frame | a datetime value | NULL | +| @custom_message | nvarchar | if you want to search for a custom string | something specific you want to search for. no wildcards or substitions. | NULL | +| @custom_message_only | bit | only search for the custom string | NULL, 0, 1 | 0 | +| @first_log_only | bit | only search through the first error log | NULL, 0, 1 | 0 | +| @language_id | integer | to use something other than English | SELECT DISTINCT m.language_id FROM sys.messages AS m ORDER BY m.language_id; | 1033 | +| @help | bit | how you got here | NULL, 0, 1 | 0 | +| @debug | bit | dumps raw temp table contents | NULL, 0, 1 | 0 | +| @version | varchar | OUTPUT; for support | OUTPUT; for support | none; OUTPUT | +| @version_date | datetime | OUTPUT; for support | OUTPUT; for support | none; OUTPUT | + +## Examples + +```sql +-- Basic execution to search the last 7 days of error logs +EXECUTE dbo.sp_LogHunter; + +-- Search logs for the last 30 days +EXECUTE dbo.sp_LogHunter + @days_back = -30; + +-- Search a specific time period +EXECUTE dbo.sp_LogHunter + @start_date = '2025-01-01 00:00:00', + @end_date = '2025-01-02 00:00:00'; + +-- Search for a specific custom message +EXECUTE dbo.sp_LogHunter + @custom_message = 'login failed'; + +-- Only search for the custom message, ignore other errors +EXECUTE dbo.sp_LogHunter + @custom_message = 'login failed', + @custom_message_only = 1; + +-- Only search the current error log +EXECUTE dbo.sp_LogHunter + @first_log_only = 1; +``` + +## Resources +* [YouTube introduction](https://youtu.be/L_yJ6zPjHfs) \ No newline at end of file diff --git a/sp_LogHunter/sp_LogHunter.sql b/sp_LogHunter/sp_LogHunter.sql index 4de77f25..4f0035ec 100644 --- a/sp_LogHunter/sp_LogHunter.sql +++ b/sp_LogHunter/sp_LogHunter.sql @@ -36,7 +36,7 @@ EXECUTE sp_LogHunter @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com EXECUTE sp_LogHunter; @@ -72,8 +72,8 @@ SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; BEGIN SELECT - @version = '2.1', - @version_date = '20250101'; + @version = '2.4', + @version_date = '20250401'; IF @help = 1 BEGIN diff --git a/sp_PressureDetector/README.md b/sp_PressureDetector/README.md new file mode 100644 index 00000000..14f39294 --- /dev/null +++ b/sp_PressureDetector/README.md @@ -0,0 +1,72 @@ + + +# sp_PressureDetector + +Is your client/server relationship on the rocks? Are queries timing out, dragging along, or causing CPU fans to spin out of control? + +All you need to do is hit F5 to get information about: +* Wait stats since startup +* Database file size, stall, and activity +* tempdb configuration details +* Memory consumers +* Low memory indicators +* Memory configuration and allocation +* Current query memory grants, along with other execution details +* CPU configuration and retained utilization details +* Thread count and current usage +* Any current THREADPOOL waits (best observed with the DAC) +* Currently executing queries, along with other execution details + +## Parameters + +| parameter_name | data_type | description | valid_inputs | defaults | +|----------------------------|-----------|--------------------------------------------------------------------------------|------------------------------------------------------|--------------| +| @what_to_check | varchar | areas to check for pressure | "all", "cpu", and "memory" | all | +| @skip_queries | bit | if you want to skip looking at running queries | 0 or 1 | 0 | +| @skip_plan_xml | bit | if you want to skip getting plan XML | 0 or 1 | 0 | +| @minimum_disk_latency_ms | smallint | low bound for reporting disk latency | a reasonable number of milliseconds for disk latency | 100 | +| @cpu_utilization_threshold | smallint | low bound for reporting high cpu utlization | a reasonable cpu utlization percentage | 50 | +| @skip_waits | bit | skips waits when you do not need them on every run | 0 or 1 | 0 | +| @skip_perfmon | bit | skips perfmon counters when you do not need them on every run | a valid tinyint: 0-255 | 0 | +| @sample_seconds | tinyint | take a sample of your server's metrics | 0 or 1 | 0 | +| @log_to_table | bit | enable logging to permanent tables | 0 or 1 | 0 | +| @log_database_name | sysname | database to store logging tables | valid database name | NULL | +| @log_schema_name | sysname | schema to store logging tables | valid schema name | NULL | +| @log_table_name_prefix | sysname | prefix for all logging tables | valid table name prefix | 'PressureDetector' | +| @log_retention_days | integer | Number of days to keep logs, 0 = keep indefinitely | integer | 30 | +| @help | bit | how you got here | 0 or 1 | 0 | +| @debug | bit | prints dynamic sql, displays parameter and variable values, and table contents | 0 or 1 | 0 | +| @version | varchar | OUTPUT; for support | none | none; OUTPUT | +| @version_date | datetime | OUTPUT; for support | none | none; OUTPUT | + +## Examples + +```sql +-- Basic execution to check all pressure types +EXECUTE dbo.sp_PressureDetector; + +-- Check only CPU pressure +EXECUTE dbo.sp_PressureDetector + @what_to_check = 'cpu'; + +-- Check only memory pressure +EXECUTE dbo.sp_PressureDetector + @what_to_check = 'memory'; + +-- Skip looking at executing queries +EXECUTE dbo.sp_PressureDetector + @skip_queries = 1; + +-- Take a 10-second sample of server metrics +EXECUTE dbo.sp_PressureDetector + @sample_seconds = 10; + +-- Log results to a table +EXECUTE dbo.sp_PressureDetector + @log_to_table = 1, + @log_database_name = 'DBA', + @log_schema_name = 'dbo'; +``` + +## Resources +* [Video walkthrough](https://www.erikdarlingdata.com/sp_pressuredetector/) \ No newline at end of file diff --git a/sp_PressureDetector/sp_PressureDetector.sql b/sp_PressureDetector/sp_PressureDetector.sql index 21ec2ede..caa64fe2 100644 --- a/sp_PressureDetector/sp_PressureDetector.sql +++ b/sp_PressureDetector/sp_PressureDetector.sql @@ -37,7 +37,7 @@ EXECUTE sp_PressureDetector @debug = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ @@ -57,6 +57,11 @@ ALTER PROCEDURE @skip_waits bit = 0, /*skips waits when you do not need them on every run*/ @skip_perfmon bit = 0, /*skips perfmon counters when you do not need them on every run*/ @sample_seconds tinyint = 0, /*take a sample of your server's metrics*/ + @log_to_table bit = 0, /*enable logging to permanent tables*/ + @log_database_name sysname = NULL, /*database to store logging tables*/ + @log_schema_name sysname = NULL, /*schema to store logging tables*/ + @log_table_name_prefix sysname = 'PressureDetector', /*prefix for all logging tables*/ + @log_retention_days integer = 30, /*Number of days to keep logs, 0 = keep indefinitely*/ @help bit = 0, /*how you got here*/ @debug bit = 0, /*prints dynamic sql, displays parameter and variable values, and table contents*/ @version varchar(5) = NULL OUTPUT, /*OUTPUT; for support*/ @@ -71,8 +76,8 @@ SET XACT_ABORT ON; SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT - @version = '5.1', - @version_date = '20250101'; + @version = '5.4', + @version_date = '20250401'; IF @help = 1 @@ -110,6 +115,11 @@ BEGIN WHEN N'@skip_waits' THEN N'skips waits when you do not need them on every run' WHEN N'@skip_perfmon' THEN N'skips perfmon counters when you do not need them on every run' WHEN N'@sample_seconds' THEN N'take a sample of your server''s metrics' + WHEN N'@log_to_table' THEN N'enable logging to permanent tables instead of returning results' + WHEN N'@log_database_name' THEN N'database to store logging tables' + WHEN N'@log_schema_name' THEN N'schema to store logging tables' + WHEN N'@log_table_name_prefix' THEN N'prefix for all logging tables' + WHEN N'@log_retention_days' THEN N'how many days of data to retain' WHEN N'@help' THEN N'how you got here' WHEN N'@debug' THEN N'prints dynamic sql, displays parameter and variable values, and table contents' WHEN N'@version' THEN N'OUTPUT; for support' @@ -126,6 +136,11 @@ BEGIN WHEN N'@skip_waits' THEN N'0 or 1' WHEN N'@skip_perfmon' THEN N'0 or 1' WHEN N'@sample_seconds' THEN N'a valid tinyint: 0-255' + WHEN N'@log_to_table' THEN N'0 or 1' + WHEN N'@log_database_name' THEN N'any valid database name' + WHEN N'@log_schema_name' THEN N'any valid schema name' + WHEN N'@log_table_name_prefix' THEN N'any valid identifier' + WHEN N'@log_retention_days' THEN N'a positive integer' WHEN N'@help' THEN N'0 or 1' WHEN N'@debug' THEN N'0 or 1' WHEN N'@version' THEN N'none' @@ -142,15 +157,20 @@ BEGIN WHEN N'@skip_waits' THEN N'0' WHEN N'@skip_perfmon' THEN N'0' WHEN N'@sample_seconds' THEN N'0' + WHEN N'@log_to_table' THEN N'0' + WHEN N'@log_database_name' THEN N'NULL (current database)' + WHEN N'@log_schema_name' THEN N'NULL (dbo)' + WHEN N'@log_table_name_prefix' THEN N'PressureDetector' + WHEN N'@log_retention_days' THEN N'30' WHEN N'@help' THEN N'0' WHEN N'@debug' THEN N'0' WHEN N'@version' THEN N'none; OUTPUT' WHEN N'@version_date' THEN N'none; OUTPUT' END FROM sys.all_parameters AS ap - INNER JOIN sys.all_objects AS o + JOIN sys.all_objects AS o ON ap.object_id = o.object_id - INNER JOIN sys.types AS t + JOIN sys.types AS t ON ap.system_type_id = t.system_type_id AND ap.user_type_id = t.user_type_id WHERE o.name = N'sp_PressureDetector' @@ -214,6 +234,58 @@ END; /*End help section*/ @what_to_check = 'all'; END; + IF @log_to_table = 1 + AND @cpu_utilization_threshold > 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Setting @cpu_utilization_threshold to 0 to capture all CPU utilization data when logging to tables', 0, 1) WITH NOWAIT; + END; + SELECT + @cpu_utilization_threshold = 0; + END; + + IF @log_to_table = 1 + AND @sample_seconds <> 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Logging to tables is not compatible with @sample_seconds. Using @sample_seconds = 0', 0, 1) WITH NOWAIT; + END; + SELECT + @sample_seconds = 0; + END; + + IF @log_to_table = 1 + AND @what_to_check <> 'all' + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('@what_to_check was set to %s, setting to all when logging to tables', 0, 1, @what_to_check) WITH NOWAIT; + END; + SELECT + @what_to_check = 'all'; + END; + + IF @log_to_table = 1 + AND + ( + @skip_queries = 1 + OR @skip_plan_xml = 1 + OR @skip_waits = 1 + OR @skip_perfmon = 1 + ) + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('reverting skip options for table logging', 0, 1, @what_to_check) WITH NOWAIT; + END; + SELECT + @skip_queries = 0, + @skip_plan_xml = 0, + @skip_waits = 0, + @skip_perfmon = 0; + END; /* Declarations of Variablependence @@ -235,7 +307,7 @@ END; /*End help section*/ THEN 1 ELSE 0 END, - @pool_sql nvarchar(MAX) = N'', + @pool_sql nvarchar(max) = N'', @pages_kb bit = CASE WHEN @@ -249,7 +321,7 @@ END; /*End help section*/ THEN 1 ELSE 0 END, - @mem_sql nvarchar(MAX) = N'', + @mem_sql nvarchar(max) = N'', @helpful_new_columns bit = CASE WHEN @@ -267,7 +339,7 @@ END; /*End help section*/ THEN 1 ELSE 0 END, - @cpu_sql nvarchar(MAX) = N'', + @cpu_sql nvarchar(max) = N'', @cool_new_columns bit = CASE WHEN @@ -286,7 +358,7 @@ END; /*End help section*/ ELSE 0 END, @reserved_worker_count_out varchar(10) = '0', - @reserved_worker_count nvarchar(MAX) = N' + @reserved_worker_count nvarchar(max) = N' SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT @@ -295,10 +367,10 @@ SELECT FROM sys.dm_exec_query_memory_grants AS deqmg OPTION(MAXDOP 1, RECOMPILE); ', - @cpu_details nvarchar(MAX) = N'', + @cpu_details nvarchar(max) = N'', @cpu_details_output xml = N'', - @cpu_details_columns nvarchar(MAX) = N'', - @cpu_details_select nvarchar(MAX) = N' + @cpu_details_columns nvarchar(max) = N'', + @cpu_details_select nvarchar(max) = N' SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT @@ -308,19 +380,19 @@ SELECT offline_cpus = (SELECT COUNT_BIG(*) FROM sys.dm_os_schedulers dos WHERE dos.is_online = 0), ', - @cpu_details_from nvarchar(MAX) = N' + @cpu_details_from nvarchar(max) = N' FROM sys.dm_os_sys_info AS osi FOR XML PATH(''cpu_details''), TYPE ) OPTION(MAXDOP 1, RECOMPILE);', - @database_size_out nvarchar(MAX) = N'', - @database_size_out_gb varchar(10) = '0', + @database_size_out nvarchar(max) = N'', + @database_size_out_gb nvarchar(10) = '0', @total_physical_memory_gb bigint, @cpu_utilization xml = N'', @low_memory xml = N'', - @disk_check nvarchar(MAX) = N'', + @disk_check nvarchar(max) = N'', @live_plans bit = CASE WHEN OBJECT_ID('sys.dm_exec_query_statistics_xml') IS NOT NULL @@ -359,7 +431,573 @@ OPTION(MAXDOP 1, RECOMPILE);', N'%', @memory_grant_cap xml, @cache_xml xml, - @cache_sql nvarchar(MAX) = N''; + @cache_sql nvarchar(max) = N'', + @resource_semaphores nvarchar(max) = N'', + @cpu_threads nvarchar(max) = N'', + /*Log to table stuff*/ + @log_table_waits sysname, + @log_table_file_metrics sysname, + @log_table_perfmon sysname, + @log_table_memory sysname, + @log_table_cpu sysname, + @log_table_memory_consumers sysname, + @log_table_memory_queries sysname, + @log_table_cpu_queries sysname, + @log_table_cpu_events sysname, + @cleanup_date datetime2(7), + @check_sql nvarchar(max) = N'', + @create_sql nvarchar(max) = N'', + @insert_sql nvarchar(max) = N'', + @delete_sql nvarchar(max) = N'', + @log_database_schema nvarchar(1024); + + /* Validate logging parameters */ + IF @log_to_table = 1 + BEGIN + /* Default database name to current database if not specified */ + SELECT @log_database_name = ISNULL(@log_database_name, DB_NAME()); + + /* Default schema name to dbo if not specified */ + SELECT @log_schema_name = ISNULL(@log_schema_name, N'dbo'); + + /* Validate database exists */ + IF NOT EXISTS + ( + SELECT + 1/0 + FROM sys.databases AS d + WHERE d.name = @log_database_name + ) + BEGIN + RAISERROR('The specified logging database %s does not exist. Logging will be disabled.', 11, 1, @log_database_name) WITH NOWAIT; + RETURN; + END; + + SET + @log_database_schema = + QUOTENAME(@log_database_name) + + N'.' + + QUOTENAME(@log_schema_name) + + N'.'; + + /* Generate fully qualified table names */ + SELECT + @log_table_waits = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_Waits'), + @log_table_file_metrics = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_FileMetrics'), + @log_table_perfmon = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_Perfmon'), + @log_table_memory = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_Memory'), + @log_table_cpu = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_CPU'), + @log_table_memory_consumers = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_MemoryConsumers'), + @log_table_memory_queries = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_MemoryQueries'), + @log_table_cpu_queries = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_CPUQueries'), + @log_table_cpu_events = + @log_database_schema + + QUOTENAME(@log_table_name_prefix + N'_CPUEvents'); + + /* Check if schema exists and create it if needed */ + SET @check_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + WHERE s.name = @schema_name + ) + BEGIN + DECLARE + @create_schema_sql nvarchar(max) = N''CREATE SCHEMA '' + QUOTENAME(@schema_name); + + EXECUTE ' + QUOTENAME(@log_database_name) + N'.sys.sp_executesql @create_schema_sql; + IF @debug = 1 BEGIN RAISERROR(''Created schema %s in database %s for logging.'', 0, 1, @schema_name, @db_name) WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @check_sql, + N'@schema_name sysname, + @db_name sysname, + @debug bit', + @log_schema_name, + @log_database_name, + @debug; + + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_Waits'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_waits + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + hours_uptime integer NULL, + hours_cpu_time decimal(38,2) NULL, + wait_type nvarchar(60) NOT NULL, + description nvarchar(60) NULL, + hours_wait_time decimal(38,2) NULL, + avg_ms_per_wait decimal(38,2) NULL, + percent_signal_waits decimal(38,2) NULL, + waiting_tasks_count bigint NULL, + sample_time datetime NULL, + sorting bigint NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for wait stats logging.'', 0, 1, ''' + @log_table_waits + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_FileMetrics'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_file_metrics + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + hours_uptime integer NULL, + drive nvarchar(255) NOT NULL, + database_name nvarchar(128) NOT NULL, + database_file_details nvarchar(1000) NULL, + file_size_gb decimal(38,2) NULL, + total_gb_read decimal(38,2) NULL, + total_mb_read decimal(38,2) NULL, + total_read_count bigint NULL, + avg_read_stall_ms decimal(38,2) NULL, + total_gb_written decimal(38,2) NULL, + total_mb_written decimal(38,2) NULL, + total_write_count bigint NULL, + avg_write_stall_ms decimal(38,2) NULL, + io_stall_read_ms bigint NULL, + io_stall_write_ms bigint NULL, + sample_time datetime NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for file metrics logging.'', 0, 1, ''' + @log_table_file_metrics + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_Perfmon'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_perfmon + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + sample_time datetime NULL, + object_name sysname NOT NULL, + counter_name sysname NOT NULL, + counter_name_clean sysname NULL, + instance_name sysname NOT NULL, + cntr_value bigint NULL, + cntr_type bigint NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for perfmon logging.'', 0, 1, ''' + @log_table_perfmon + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_Memory'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_memory + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + resource_semaphore_id integer NOT NULL, + total_database_size_gb varchar(20) NULL, + total_physical_memory_gb bigint NULL, + max_server_memory_gb bigint NULL, + max_memory_grant_cap xml NULL, + memory_model nvarchar(128) NULL, + target_memory_gb decimal(38,2) NULL, + max_target_memory_gb decimal(38,2) NULL, + total_memory_gb decimal(38,2) NULL, + available_memory_gb decimal(38,2) NULL, + granted_memory_gb decimal(38,2) NULL, + used_memory_gb decimal(38,2) NULL, + grantee_count integer NULL, + waiter_count integer NULL, + timeout_error_count integer NULL, + forced_grant_count integer NULL, + total_reduced_memory_grant_count bigint NULL, + pool_id integer NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for memory logging.'', 0, 1, ''' + @log_table_memory + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_CPU'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_cpu + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + total_threads integer NULL, + used_threads integer NULL, + available_threads integer NULL, + reserved_worker_count varchar(10) NULL, + threads_waiting_for_cpu integer NULL, + requests_waiting_for_threads integer NULL, + current_workers integer NULL, + total_active_request_count integer NULL, + total_queued_request_count integer NULL, + total_blocked_task_count integer NULL, + total_active_parallel_thread_count integer NULL, + avg_runnable_tasks_count float NULL, + high_runnable_percent varchar(100) NULL, + cpu_details_output xml NULL, + cpu_utilization_over_threshold xml NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for CPU logging.'', 0, 1, ''' + @log_table_cpu + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Memory Consumers table */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_MemoryConsumers'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_memory_consumers + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + memory_source nvarchar(128) NOT NULL, + memory_consumer nvarchar(128) NOT NULL, + memory_consumed_gb decimal(38,2) NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for memory consumers logging.'', 0, 1, ''' + @log_database_schema + QUOTENAME(@log_table_name_prefix + N'_MemoryConsumers') + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Memory Query Grants table */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_MemoryQueries'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_memory_queries + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + session_id integer NOT NULL, + database_name nvarchar(128) NULL, + duration varchar(30) NULL, + request_time datetime NULL, + grant_time datetime NULL, + wait_time_seconds decimal(38,2) NULL, + requested_memory_gb decimal(38,2) NULL, + granted_memory_gb decimal(38,2) NULL, + used_memory_gb decimal(38,2) NULL, + max_used_memory_gb decimal(38,2) NULL, + ideal_memory_gb decimal(38,2) NULL, + required_memory_gb decimal(38,2) NULL, + queue_id integer NULL, + wait_order integer NULL, + is_next_candidate bit NULL, + wait_type nvarchar(60) NULL, + wait_duration_seconds decimal(38,2) NULL, + dop integer NULL, + reserved_worker_count integer NULL, + used_worker_count integer NULL, + plan_handle varbinary(64) NULL, + sql_text xml NULL, + query_plan_xml xml NULL, + live_query_plan xml NULL + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for memory queries logging.'', 0, 1, ''' + @log_database_schema + QUOTENAME(@log_table_name_prefix + N'_MemoryQueries') + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* CPU Queries table */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_CPUQueries'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_cpu_queries + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + session_id integer NOT NULL, + database_name nvarchar(128) NULL, + duration varchar(30) NULL, + status nvarchar(30) NULL, + blocking_session_id integer NULL, + wait_type nvarchar(60) NULL, + wait_time_ms bigint NULL, + wait_resource nvarchar(512) NULL, + cpu_time_ms bigint NULL, + total_elapsed_time_ms bigint NULL, + reads bigint NULL, + writes bigint NULL, + logical_reads bigint NULL, + granted_query_memory_gb decimal(38,2) NULL, + transaction_isolation_level nvarchar(30) NULL, + dop integer NULL, + parallel_worker_count integer NULL, + plan_handle varbinary(64) NULL, + sql_text xml NULL, + query_plan_xml xml NULL, + live_query_plan xml NULL, + statement_start_offset integer NULL, + statement_end_offset integer NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for CPU queries logging.'', 0, 1, ''' + @log_database_schema + QUOTENAME(@log_table_name_prefix + N'_CPUQueries') + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* CPU Utilization Events table */ + SET @create_sql = N' + IF NOT EXISTS + ( + SELECT + 1/0 + FROM ' + QUOTENAME(@log_database_name) + N'.sys.tables AS t + JOIN ' + QUOTENAME(@log_database_name) + N'.sys.schemas AS s + ON t.schema_id = s.schema_id + WHERE t.name = @table_name + N''_CPUEvents'' + AND s.name = @schema_name + ) + BEGIN + CREATE TABLE ' + @log_table_cpu_events + N' + ( + id bigint IDENTITY, + collection_time datetime2(7) NOT NULL DEFAULT SYSDATETIME(), + sample_time datetime NULL, + sqlserver_cpu_utilization integer NULL, + other_process_cpu_utilization integer NULL, + total_cpu_utilization integer NULL, + PRIMARY KEY CLUSTERED (collection_time, id) + ); + IF @debug = 1 BEGIN RAISERROR(''Created table %s for CPU utilization events logging.'', 0, 1, ''' + @log_database_schema + QUOTENAME(@log_table_name_prefix + N'_CPUEvents') + N''') WITH NOWAIT; END; + END'; + + EXECUTE sys.sp_executesql + @create_sql, + N'@schema_name sysname, + @table_name sysname, + @debug bit', + @log_schema_name, + @log_table_name_prefix, + @debug; + + /* Handle log retention if specified */ + IF @log_to_table = 1 AND @log_retention_days > 0 + BEGIN + IF @debug = 1 + BEGIN + RAISERROR('Cleaning up log tables older than %i days', 0, 1, @log_retention_days) WITH NOWAIT; + END; + + SET @cleanup_date = + DATEADD + ( + DAY, + -@log_retention_days, + SYSDATETIME() + ); + + /* Clean up each log table */ + SET @delete_sql = N' + DELETE FROM ' + @log_table_waits + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_file_metrics + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_perfmon + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_memory + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_cpu + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_memory_consumers + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_memory_queries + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_cpu_queries + ' + WHERE collection_time < @cleanup_date; + + DELETE FROM ' + @log_table_cpu_events + ' + WHERE collection_time < @cleanup_date;'; + + IF @debug = 1 BEGIN PRINT @delete_sql; END; + + EXECUTE sys.sp_executesql + @delete_sql, + N'@cleanup_date datetime2(7)', + @cleanup_date; + + IF @debug = 1 + BEGIN + RAISERROR('Log cleanup complete', 0, 1) WITH NOWAIT; + END; + END; + + END; /*End log to tables validation checks here*/ DECLARE @waits table @@ -445,6 +1083,7 @@ OPTION(MAXDOP 1, RECOMPILE);', ( @what_to_check = 'all' AND @pass = 1 + AND @log_to_table = 0 ) BEGIN IF @debug = 1 @@ -760,100 +1399,153 @@ OPTION(MAXDOP 1, RECOMPILE);', dows.waiting_tasks_count DESC OPTION(MAXDOP 1, RECOMPILE); - IF @sample_seconds = 0 + IF @log_to_table = 0 BEGIN - SELECT - w.wait_type, - w.description, - w.hours_uptime, - w.hours_cpu_time, - w.hours_wait_time, - w.avg_ms_per_wait, - w.percent_signal_waits, - waiting_tasks_count = - REPLACE - ( - CONVERT + IF @sample_seconds = 0 + BEGIN + SELECT + w.wait_type, + w.description, + w.hours_uptime, + w.hours_cpu_time, + w.hours_wait_time, + w.avg_ms_per_wait, + w.percent_signal_waits, + waiting_tasks_count = + REPLACE ( - nvarchar(30), CONVERT ( - money, - w.waiting_tasks_count + nvarchar(30), + CONVERT + ( + money, + w.waiting_tasks_count + ), + 1 ), - 1 + N'.00', + N'' + ) + FROM @waits AS w + WHERE w.waiting_tasks_count_n > 0 + ORDER BY + w.sorting + OPTION(MAXDOP 1, RECOMPILE); + END; + + IF + ( + @sample_seconds > 0 + AND @pass = 1 + ) + BEGIN + SELECT + w.wait_type, + w.description, + sample_cpu_time_seconds = + CONVERT + ( + decimal(38,2), + (w2.hours_cpu_time - w.hours_cpu_time) / 1000. ), - N'.00', - N'' - ) - FROM @waits AS w - WHERE w.waiting_tasks_count_n > 0 - ORDER BY - w.sorting - OPTION(MAXDOP 1, RECOMPILE); - END; - - IF - ( - @sample_seconds > 0 - AND @pass = 1 - ) - BEGIN - SELECT - w.wait_type, - w.description, - sample_cpu_time_seconds = - CONVERT - ( - decimal(38,2), - (w2.hours_cpu_time - w.hours_cpu_time) / 1000. - ), - wait_time_seconds = - CONVERT - ( - decimal(38,2), - (w2.hours_wait_time - w.hours_wait_time) / 1000. - ), - avg_ms_per_wait = - CONVERT - ( - decimal(38,1), - (w2.avg_ms_per_wait + w.avg_ms_per_wait) / 2 - ), - percent_signal_waits = - CONVERT - ( - decimal(38,1), - (w2.percent_signal_waits + w.percent_signal_waits) / 2 - ), - waiting_tasks_count = - REPLACE - ( + wait_time_seconds = CONVERT ( - nvarchar(30), + decimal(38,2), + (w2.hours_wait_time - w.hours_wait_time) / 1000. + ), + avg_ms_per_wait = + CONVERT + ( + decimal(38,1), + (w2.avg_ms_per_wait + w.avg_ms_per_wait) / 2 + ), + percent_signal_waits = + CONVERT + ( + decimal(38,1), + (w2.percent_signal_waits + w.percent_signal_waits) / 2 + ), + waiting_tasks_count = + REPLACE + ( CONVERT ( - money, - (w2.waiting_tasks_count_n - w.waiting_tasks_count_n) + nvarchar(30), + CONVERT + ( + money, + (w2.waiting_tasks_count_n - w.waiting_tasks_count_n) + ), + 1 ), - 1 + N'.00', + N'' ), - N'.00', - N'' - ), - sample_seconds = - DATEDIFF(SECOND, w.sample_time, w2.sample_time) - FROM @waits AS w - JOIN @waits AS w2 - ON w.wait_type = w2.wait_type - AND w.sample_time < w2.sample_time - AND (w2.waiting_tasks_count_n - w.waiting_tasks_count_n) > 0 - ORDER BY - wait_time_seconds DESC - OPTION(MAXDOP 1, RECOMPILE); + sample_seconds = + DATEDIFF(SECOND, w.sample_time, w2.sample_time) + FROM @waits AS w + JOIN @waits AS w2 + ON w.wait_type = w2.wait_type + AND w.sample_time < w2.sample_time + AND (w2.waiting_tasks_count_n - w.waiting_tasks_count_n) > 0 + ORDER BY + wait_time_seconds DESC + OPTION(MAXDOP 1, RECOMPILE); + END; END; - END; + + IF @log_to_table = 1 + BEGIN + + SELECT + w.* + INTO #waits + FROM @waits AS w + OPTION(RECOMPILE); + + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_waits + N' + ( + hours_uptime, + hours_cpu_time, + wait_type, + description, + hours_wait_time, + avg_ms_per_wait, + percent_signal_waits, + waiting_tasks_count, + sample_time, + sorting + ) + SELECT + w.hours_uptime, + w.hours_cpu_time, + w.wait_type, + w.description, + w.hours_wait_time, + w.avg_ms_per_wait, + w.percent_signal_waits, + w.waiting_tasks_count_n, + w.sample_time, + w.sorting + FROM #waits AS w; + '; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql; + + DROP TABLE IF EXISTS + #waits; + END; + END; /*End wait stats*/ /* This section looks at disk metrics */ @@ -1011,7 +1703,7 @@ OPTION(MAXDOP 1, RECOMPILE);', JOIN ' + CONVERT ( - nvarchar(MAX), + nvarchar(max), CASE WHEN @azure = 1 THEN N'sys.database_files AS f @@ -1059,22 +1751,188 @@ OPTION(MAXDOP 1, RECOMPILE);', EXECUTE sys.sp_executesql @disk_check; - IF @sample_seconds = 0 + IF @log_to_table = 0 BEGIN - WITH - file_metrics AS - ( + IF @sample_seconds = 0 + BEGIN + WITH + file_metrics AS + ( + SELECT + fm.hours_uptime, + fm.drive, + fm.database_name, + fm.database_file_details, + fm.file_size_gb, + fm.avg_read_stall_ms, + fm.avg_write_stall_ms, + fm.total_gb_read, + fm.total_gb_written, + total_read_count = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + fm.total_read_count + ), + 1 + ), + N'.00', + N'' + ), + total_write_count = + REPLACE + ( + CONVERT + ( + nvarchar(30), + CONVERT + ( + money, + fm.total_write_count + ), + 1 + ), + N'.00', + N'' + ), + total_avg_stall_ms = + fm.avg_read_stall_ms + + fm.avg_write_stall_ms + FROM @file_metrics AS fm + WHERE fm.avg_read_stall_ms > @minimum_disk_latency_ms + OR fm.avg_write_stall_ms > @minimum_disk_latency_ms + ) SELECT - fm.hours_uptime, fm.drive, fm.database_name, fm.database_file_details, + fm.hours_uptime, fm.file_size_gb, fm.avg_read_stall_ms, fm.avg_write_stall_ms, + fm.total_avg_stall_ms, fm.total_gb_read, fm.total_gb_written, - total_read_count = + fm.total_read_count, + fm.total_write_count + FROM file_metrics AS fm + + UNION ALL + + SELECT + drive = N'Nothing to see here', + database_name = N'By default, only >100 ms latency is reported', + database_file_details = N'Use the @minimum_disk_latency_ms parameter to adjust what you see', + hours_uptime = 0, + file_size_gb = 0, + avg_read_stall_ms = 0, + avg_write_stall_ms = 0, + total_avg_stall = 0, + total_gb_read = 0, + total_gb_written = 0, + total_read_count = N'0', + total_write_count = N'0' + WHERE NOT EXISTS + ( + SELECT + 1/0 + FROM file_metrics AS fm + ) + ORDER BY + total_avg_stall_ms DESC + OPTION(MAXDOP 1, RECOMPILE); + END; + + IF + ( + @sample_seconds > 0 + AND @pass = 1 + ) + BEGIN + WITH + f AS + ( + SELECT + fm.drive, + fm.database_name, + fm.database_file_details, + fm.file_size_gb, + avg_read_stall_ms = + CASE + WHEN (fm2.total_read_count - fm.total_read_count) = 0 + THEN 0.00 + ELSE + CONVERT + ( + decimal(38, 2), + (fm2.io_stall_read_ms - fm.io_stall_read_ms) / + (fm2.total_read_count - fm.total_read_count) + ) + END, + avg_write_stall_ms = + CASE + WHEN (fm2.total_write_count - fm.total_write_count) = 0 + THEN 0.00 + ELSE + CONVERT + ( + decimal(38, 2), + (fm2.io_stall_write_ms - fm.io_stall_write_ms) / + (fm2.total_write_count - fm.total_write_count) + ) + END, + total_avg_stall = + CASE + WHEN (fm2.total_read_count - fm.total_read_count) + + (fm2.total_write_count - fm.total_write_count) = 0 + THEN 0.00 + ELSE + CONVERT + ( + decimal(38,2), + ( + (fm2.io_stall_read_ms - fm.io_stall_read_ms) + + (fm2.io_stall_write_ms - fm.io_stall_write_ms) + ) / + ( + (fm2.total_read_count - fm.total_read_count) + + (fm2.total_write_count - fm.total_write_count) + ) + ) + END, + total_mb_read = + (fm2.total_mb_read - fm.total_mb_read), + total_mb_written = + (fm2.total_mb_written - fm.total_mb_written), + total_read_count = + (fm2.total_read_count - fm.total_read_count), + total_write_count = + (fm2.total_write_count - fm.total_write_count), + sample_time_o = + fm.sample_time, + sample_time_t = + fm2.sample_time + FROM @file_metrics AS fm + JOIN @file_metrics AS fm2 + ON fm.drive = fm2.drive + AND fm.database_name = fm2.database_name + AND fm.database_file_details = fm2.database_file_details + AND fm.sample_time < fm2.sample_time + ) + SELECT + f.drive, + f.database_name, + f.database_file_details, + f.file_size_gb, + f.avg_read_stall_ms, + f.avg_write_stall_ms, + f.total_avg_stall, + total_mb_read = REPLACE ( CONVERT @@ -1083,14 +1941,14 @@ OPTION(MAXDOP 1, RECOMPILE);', CONVERT ( money, - fm.total_read_count + f.total_mb_read ), 1 ), N'.00', N'' ), - total_write_count = + total_mb_written = REPLACE ( CONVERT @@ -1099,220 +1957,119 @@ OPTION(MAXDOP 1, RECOMPILE);', CONVERT ( money, - fm.total_write_count + f.total_mb_written ), - 1 - ), - N'.00', - N'' - ), - total_avg_stall_ms = - fm.avg_read_stall_ms + - fm.avg_write_stall_ms - FROM @file_metrics AS fm - WHERE fm.avg_read_stall_ms > @minimum_disk_latency_ms - OR fm.avg_write_stall_ms > @minimum_disk_latency_ms - ) - SELECT - fm.drive, - fm.database_name, - fm.database_file_details, - fm.hours_uptime, - fm.file_size_gb, - fm.avg_read_stall_ms, - fm.avg_write_stall_ms, - fm.total_avg_stall_ms, - fm.total_gb_read, - fm.total_gb_written, - fm.total_read_count, - fm.total_write_count - FROM file_metrics AS fm - - UNION ALL - - SELECT - drive = N'Nothing to see here', - database_name = N'By default, only >100 ms latency is reported', - database_file_details = N'Use the @minimum_disk_latency_ms parameter to adjust what you see', - hours_uptime = 0, - file_size_gb = 0, - avg_read_stall_ms = 0, - avg_write_stall_ms = 0, - total_avg_stall = 0, - total_gb_read = 0, - total_gb_written = 0, - total_read_count = N'0', - total_write_count = N'0' - WHERE NOT EXISTS - ( - SELECT - 1/0 - FROM file_metrics AS fm - ) - ORDER BY - total_avg_stall_ms DESC - OPTION(MAXDOP 1, RECOMPILE); - END; - - IF - ( - @sample_seconds > 0 - AND @pass = 1 - ) - BEGIN - WITH - f AS - ( - SELECT - fm.drive, - fm.database_name, - fm.database_file_details, - fm.file_size_gb, - avg_read_stall_ms = - CASE - WHEN (fm2.total_read_count - fm.total_read_count) = 0 - THEN 0.00 - ELSE - CONVERT - ( - decimal(38, 2), - (fm2.io_stall_read_ms - fm.io_stall_read_ms) / - (fm2.total_read_count - fm.total_read_count) - ) - END, - avg_write_stall_ms = - CASE - WHEN (fm2.total_write_count - fm.total_write_count) = 0 - THEN 0.00 - ELSE - CONVERT - ( - decimal(38, 2), - (fm2.io_stall_write_ms - fm.io_stall_write_ms) / - (fm2.total_write_count - fm.total_write_count) - ) - END, - total_avg_stall = - CASE - WHEN (fm2.total_read_count - fm.total_read_count) + - (fm2.total_write_count - fm.total_write_count) = 0 - THEN 0.00 - ELSE - CONVERT - ( - decimal(38,2), - ( - (fm2.io_stall_read_ms - fm.io_stall_read_ms) + - (fm2.io_stall_write_ms - fm.io_stall_write_ms) - ) / - ( - (fm2.total_read_count - fm.total_read_count) + - (fm2.total_write_count - fm.total_write_count) - ) - ) - END, - total_mb_read = - (fm2.total_mb_read - fm.total_mb_read), - total_mb_written = - (fm2.total_mb_written - fm.total_mb_written), - total_read_count = - (fm2.total_read_count - fm.total_read_count), - total_write_count = - (fm2.total_write_count - fm.total_write_count), - sample_time_o = - fm.sample_time, - sample_time_t = - fm2.sample_time - FROM @file_metrics AS fm - JOIN @file_metrics AS fm2 - ON fm.drive = fm2.drive - AND fm.database_name = fm2.database_name - AND fm.database_file_details = fm2.database_file_details - AND fm.sample_time < fm2.sample_time - ) - SELECT - f.drive, - f.database_name, - f.database_file_details, - f.file_size_gb, - f.avg_read_stall_ms, - f.avg_write_stall_ms, - f.total_avg_stall, - total_mb_read = - REPLACE - ( - CONVERT - ( - nvarchar(30), - CONVERT - ( - money, - f.total_mb_read - ), - 1 - ), - N'.00', - N'' - ), - total_mb_written = - REPLACE - ( - CONVERT - ( - nvarchar(30), - CONVERT - ( - money, - f.total_mb_written + 1 ), - 1 + N'.00', + N'' ), - N'.00', - N'' - ), - total_read_count = - REPLACE - ( - CONVERT + total_read_count = + REPLACE ( - nvarchar(30), CONVERT ( - money, - f.total_read_count + nvarchar(30), + CONVERT + ( + money, + f.total_read_count + ), + 1 ), - 1 + N'.00', + N'' ), - N'.00', - N'' - ), - total_write_count = - REPLACE - ( - CONVERT + total_write_count = + REPLACE ( - nvarchar(30), CONVERT ( - money, - f.total_write_count + nvarchar(30), + CONVERT + ( + money, + f.total_write_count + ), + 1 ), - 1 + N'.00', + N'' ), - N'.00', - N'' - ), - sample_seconds = - DATEDIFF(SECOND, f.sample_time_o, f.sample_time_t) - FROM f - WHERE - ( - f.total_read_count > 0 - OR f.total_write_count > 0 - ) - ORDER BY - f.total_avg_stall DESC - OPTION(MAXDOP 1, RECOMPILE); + sample_seconds = + DATEDIFF(SECOND, f.sample_time_o, f.sample_time_t) + FROM f + WHERE + ( + f.total_read_count > 0 + OR f.total_write_count > 0 + ) + ORDER BY + f.total_avg_stall DESC + OPTION(MAXDOP 1, RECOMPILE); + END; + END; + + IF @log_to_table = 1 + BEGIN + + SELECT + fm.* + INTO #file_metrics + FROM @file_metrics AS fm + OPTION(RECOMPILE); + + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_file_metrics + N' + ( + hours_uptime, + drive, + database_name, + database_file_details, + file_size_gb, + total_gb_read, + total_mb_read, + total_read_count, + avg_read_stall_ms, + total_gb_written, + total_mb_written, + total_write_count, + avg_write_stall_ms, + io_stall_read_ms, + io_stall_write_ms, + sample_time + ) + SELECT + fm.hours_uptime, + fm.drive, + fm.database_name, + fm.database_file_details, + fm.file_size_gb, + fm.total_gb_read, + fm.total_mb_read, + fm.total_read_count, + fm.avg_read_stall_ms, + fm.total_gb_written, + fm.total_mb_written, + fm.total_write_count, + fm.avg_write_stall_ms, + fm.io_stall_read_ms, + fm.io_stall_write_ms, + fm.sample_time + FROM #file_metrics AS fm; + '; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql; + + DROP TABLE IF EXISTS + #file_metrics; END; END; /*End file stats*/ @@ -1395,112 +2152,160 @@ OPTION(MAXDOP 1, RECOMPILE);', N'Active parallel threads', N'Active requests', N'Blocked tasks', N'Query optimizations/sec', N'Queued requests', N'Reduced memory grants/sec' ); - IF @sample_seconds = 0 + + IF @log_to_table = 0 BEGIN - WITH - p AS - ( - SELECT - hours_uptime = - ( - SELECT + IF @sample_seconds = 0 + BEGIN + WITH + p AS + ( + SELECT + hours_uptime = + ( + SELECT + DATEDIFF + ( + HOUR, + dopc.sample_time, + SYSDATETIME() + ) + ), + dopc.object_name, + dopc.counter_name, + dopc.instance_name, + dopc.cntr_value, + total = + FORMAT(dopc.cntr_value, 'N0'), + total_per_second = + FORMAT + ( + dopc.cntr_value / DATEDIFF ( - HOUR, + SECOND, dopc.sample_time, SYSDATETIME() - ) - ), - dopc.object_name, - dopc.counter_name, - dopc.instance_name, - dopc.cntr_value, - total = - FORMAT(dopc.cntr_value, 'N0'), - total_per_second = - FORMAT - ( - dopc.cntr_value / - DATEDIFF - ( - SECOND, - dopc.sample_time, - SYSDATETIME() - ), - 'N0' - ) - FROM @dm_os_performance_counters AS dopc - ) - SELECT - p.object_name, - p.counter_name, - p.instance_name, - p.hours_uptime, - p.total, - p.total_per_second - FROM p - WHERE p.cntr_value > 0 - ORDER BY - p.object_name, - p.counter_name, - p.cntr_value DESC - OPTION(MAXDOP 1, RECOMPILE); - END; - - IF - ( - @sample_seconds > 0 - AND @pass = 1 - ) - BEGIN - WITH - p AS - ( + ), + 'N0' + ) + FROM @dm_os_performance_counters AS dopc + ) SELECT - dopc.object_name, - dopc.counter_name, - dopc.instance_name, - first_cntr_value = - FORMAT(dopc.cntr_value, 'N0'), - second_cntr_value = - FORMAT(dopc2.cntr_value, 'N0'), - total_difference = - FORMAT((dopc2.cntr_value - dopc.cntr_value), 'N0'), - total_difference_per_second = - FORMAT((dopc2.cntr_value - dopc.cntr_value) / - DATEDIFF(SECOND, dopc.sample_time, dopc2.sample_time), 'N0'), - sample_seconds = - DATEDIFF(SECOND, dopc.sample_time, dopc2.sample_time), - first_sample_time = - dopc.sample_time, - second_sample_time = - dopc2.sample_time, - total_difference_i = - (dopc2.cntr_value - dopc.cntr_value) - FROM @dm_os_performance_counters AS dopc - JOIN @dm_os_performance_counters AS dopc2 - ON dopc.object_name = dopc2.object_name - AND dopc.counter_name = dopc2.counter_name - AND dopc.instance_name = dopc2.instance_name - AND dopc.sample_time < dopc2.sample_time - WHERE (dopc2.cntr_value - dopc.cntr_value) <> 0 + p.object_name, + p.counter_name, + p.instance_name, + p.hours_uptime, + p.total, + p.total_per_second + FROM p + WHERE p.cntr_value > 0 + ORDER BY + p.object_name, + p.counter_name, + p.cntr_value DESC + OPTION(MAXDOP 1, RECOMPILE); + END; + + IF + ( + @sample_seconds > 0 + AND @pass = 1 ) - SELECT - p.object_name, - p.counter_name, - p.instance_name, - p.first_cntr_value, - p.second_cntr_value, - p.total_difference, - p.total_difference_per_second, - p.sample_seconds - FROM p - ORDER BY - p.object_name, - p.counter_name, - p.total_difference_i DESC - OPTION(MAXDOP 1, RECOMPILE); + BEGIN + WITH + p AS + ( + SELECT + dopc.object_name, + dopc.counter_name, + dopc.instance_name, + first_cntr_value = + FORMAT(dopc.cntr_value, 'N0'), + second_cntr_value = + FORMAT(dopc2.cntr_value, 'N0'), + total_difference = + FORMAT((dopc2.cntr_value - dopc.cntr_value), 'N0'), + total_difference_per_second = + FORMAT((dopc2.cntr_value - dopc.cntr_value) / + DATEDIFF(SECOND, dopc.sample_time, dopc2.sample_time), 'N0'), + sample_seconds = + DATEDIFF(SECOND, dopc.sample_time, dopc2.sample_time), + first_sample_time = + dopc.sample_time, + second_sample_time = + dopc2.sample_time, + total_difference_i = + (dopc2.cntr_value - dopc.cntr_value) + FROM @dm_os_performance_counters AS dopc + JOIN @dm_os_performance_counters AS dopc2 + ON dopc.object_name = dopc2.object_name + AND dopc.counter_name = dopc2.counter_name + AND dopc.instance_name = dopc2.instance_name + AND dopc.sample_time < dopc2.sample_time + WHERE (dopc2.cntr_value - dopc.cntr_value) <> 0 + ) + SELECT + p.object_name, + p.counter_name, + p.instance_name, + p.first_cntr_value, + p.second_cntr_value, + p.total_difference, + p.total_difference_per_second, + p.sample_seconds + FROM p + ORDER BY + p.object_name, + p.counter_name, + p.total_difference_i DESC + OPTION(MAXDOP 1, RECOMPILE); + END; END; + + IF @log_to_table = 1 + BEGIN + + SELECT + dopc.* + INTO #dm_os_performance_counters + FROM @dm_os_performance_counters AS dopc + OPTION(RECOMPILE); + + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_perfmon + N' + ( + sample_time, + object_name, + counter_name, + counter_name_clean, + instance_name, + cntr_value, + cntr_type + ) + SELECT + dopc.sample_time, + dopc.object_name, + dopc.counter_name, + dopc.counter_name_clean, + dopc.instance_name, + dopc.cntr_value, + dopc.cntr_type + FROM #dm_os_performance_counters AS dopc; + '; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql; + + DROP TABLE IF EXISTS + #dm_os_performance_counters; + END; END; /*End Perfmon*/ /* @@ -1511,6 +2316,7 @@ OPTION(MAXDOP 1, RECOMPILE);', @azure = 0 AND @what_to_check = 'all' AND @pass = 1 + AND @log_to_table = 0 ) BEGIN IF @debug = 1 @@ -1677,7 +2483,7 @@ OPTION(MAXDOP 1, RECOMPILE);', ' + CONVERT ( - nvarchar(MAX), + nvarchar(max), CASE @pages_kb WHEN 1 THEN @@ -1686,7 +2492,7 @@ OPTION(MAXDOP 1, RECOMPILE);', N'domc.single_pages_kb + domc.multi_pages_kb + ' END - ) + ) + N' domc.virtual_memory_committed_kb + domc.awe_allocated_kb + @@ -1739,7 +2545,10 @@ OPTION(MAXDOP 1, RECOMPILE);', decimal(38, 2), SUM ( - ' + CASE @pages_kb + ' + CONVERT + ( + nvarchar(max), + CASE @pages_kb WHEN 1 THEN N' domc.pages_kb ' @@ -1764,7 +2573,7 @@ OPTION(MAXDOP 1, RECOMPILE);', ELSE N'domc.single_pages_kb + domc.multi_pages_kb ' - END + N' + END ) + N' ) / 1024. / 1024. > 0. ORDER BY memory_used_gb DESC @@ -1777,8 +2586,38 @@ OPTION(MAXDOP 1, RECOMPILE);', PRINT @pool_sql; END; - EXECUTE sys.sp_executesql - @pool_sql; + IF @log_to_table = 0 + BEGIN + EXECUTE sys.sp_executesql + @pool_sql; + END; + + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_memory_consumers + N' + ( + memory_source, + memory_consumer, + memory_consumed_gb + ) + ' + + REPLACE + ( + @pool_sql, + N'SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;', + N'' + ); + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql; + END; /*Checking total database size*/ IF @azure = 1 @@ -1819,7 +2658,7 @@ OPTION(MAXDOP 1, RECOMPILE);', EXECUTE sys.sp_executesql @database_size_out, - N'@database_size_out_gb varchar(10) OUTPUT', + N'@database_size_out_gb nvarchar(10) OUTPUT', @database_size_out_gb OUTPUT; /*Check physical memory in the server*/ @@ -2039,13 +2878,16 @@ OPTION(MAXDOP 1, RECOMPILE);', IF @debug = 1 BEGIN - RAISERROR('%s', 0, 1, @cache_sql) WITH NOWAIT; + PRINT @cache_sql; END; + IF @log_to_table = 0 + BEGIN EXECUTE sys.sp_executesql @cache_sql, N'@cache_xml xml OUTPUT', @cache_xml OUTPUT; + END; IF @cache_xml IS NULL BEGIN @@ -2060,11 +2902,14 @@ OPTION(MAXDOP 1, RECOMPILE);', ); END; - SELECT - low_memory = - @low_memory, - cache_memory = - @cache_xml; + IF @log_to_table = 0 + BEGIN + SELECT + low_memory = + @low_memory, + cache_memory = + @cache_xml; + END; SELECT @memory_grant_cap = @@ -2111,6 +2956,8 @@ OPTION(MAXDOP 1, RECOMPILE);', ); END; + SELECT + @resource_semaphores += N' SELECT deqrs.resource_semaphore_id, total_database_size_gb = @@ -2126,7 +2973,7 @@ OPTION(MAXDOP 1, RECOMPILE);', c.value_in_use ) FROM sys.configurations AS c - WHERE c.name = N'max server memory (MB)' + WHERE c.name = N''max server memory (MB)'' ) / 1024, max_memory_grant_cap = @memory_grant_cap, @@ -2192,6 +3039,61 @@ OPTION(MAXDOP 1, RECOMPILE);', ORDER BY deqrs.pool_id OPTION(MAXDOP 1, RECOMPILE); + '; + + IF @log_to_table = 0 + BEGIN + EXECUTE sys.sp_executesql + @resource_semaphores, + N'@database_size_out_gb nvarchar(10), + @total_physical_memory_gb bigint, + @memory_grant_cap xml', + @database_size_out_gb, + @total_physical_memory_gb, + @memory_grant_cap; + END + + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_memory + N' + ( + resource_semaphore_id, + total_database_size_gb, + total_physical_memory_gb, + max_server_memory_gb, + max_memory_grant_cap, + memory_model, + target_memory_gb, + max_target_memory_gb, + total_memory_gb, + available_memory_gb, + granted_memory_gb, + used_memory_gb, + grantee_count, + waiter_count, + timeout_error_count, + forced_grant_count, + total_reduced_memory_grant_count, + pool_id + )' + + @resource_semaphores; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@database_size_out_gb nvarchar(10), + @total_physical_memory_gb bigint, + @memory_grant_cap xml', + @database_size_out_gb, + @total_physical_memory_gb, + @memory_grant_cap; + END; END; /*End memory checks*/ /* @@ -2306,7 +3208,7 @@ OPTION(MAXDOP 1, RECOMPILE);', ),' + CONVERT ( - nvarchar(MAX), + nvarchar(max), CASE WHEN @skip_plan_xml = 0 THEN N' @@ -2411,8 +3313,78 @@ OPTION(MAXDOP 1, RECOMPILE);', PRINT SUBSTRING(@mem_sql, 4001, 8000); END; + IF @log_to_table = 0 + BEGIN EXECUTE sys.sp_executesql @mem_sql; + END + + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_memory_queries + N' + ( + session_id, + database_name, + duration, + sql_text, + query_plan_xml' + + CASE + WHEN @live_plans = 1 + THEN N', + live_query_plan' + ELSE N'' + END + N', + request_time, + grant_time, + wait_time_seconds, + requested_memory_gb, + granted_memory_gb, + used_memory_gb, + max_used_memory_gb, + ideal_memory_gb, + required_memory_gb, + queue_id, + wait_order, + is_next_candidate, + wait_type, + wait_duration_seconds, + dop' + + CASE + WHEN @helpful_new_columns = 1 + THEN N', + reserved_worker_count, + used_worker_count' + ELSE N'' + END + N', + plan_handle + ) ' + + REPLACE + ( + REPLACE + ( + REPLACE + ( + @mem_sql, + N'SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;', + N'' + ), + N'SET LOCK_TIMEOUT 1000;', + N'' + ), + N'SET LOCK_TIMEOUT -1;', + N'' + ); + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql; + END; END; /* @@ -2563,13 +3535,47 @@ OPTION(MAXDOP 1, RECOMPILE);', ); END; - SELECT - cpu_details_output = - @cpu_details_output, - cpu_utilization_over_threshold = + IF @log_to_table = 0 + BEGIN + SELECT + cpu_details_output = + @cpu_details_output, + cpu_utilization_over_threshold = + @cpu_utilization; + END; + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_cpu_events + N' + ( + sample_time, + sqlserver_cpu_utilization, + other_process_cpu_utilization, + total_cpu_utilization + ) + SELECT + sample_time = event.value(''(./sample_time)[1]'', ''datetime''), + sqlserver_cpu_utilization = event.value(''(./sqlserver_cpu_utilization)[1]'', ''integer''), + other_process_cpu_utilization = event.value(''(./other_process_cpu_utilization)[1]'', ''integer''), + total_cpu_utilization = event.value(''(./total_cpu_utilization)[1]'', ''integer'') + FROM @cpu_utilization.nodes(''/cpu_utilization'') AS cpu(event);'; + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql, + N'@cpu_utilization xml', @cpu_utilization; + END; /*Thread usage*/ + SELECT + @cpu_threads += N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT total_threads = MAX(osi.max_workers_count), @@ -2577,7 +3583,7 @@ OPTION(MAXDOP 1, RECOMPILE);', SUM(dos.active_workers_count), available_threads = MAX(osi.max_workers_count) - SUM(dos.active_workers_count), - reserved_worker_count = + reserved_worker_count = ' + CASE @helpful_new_columns WHEN 1 THEN ISNULL @@ -2586,7 +3592,7 @@ OPTION(MAXDOP 1, RECOMPILE);', N'0' ) ELSE N'N/A' - END, + END + N', threads_waiting_for_cpu = SUM(dos.runnable_tasks_count), requests_waiting_for_threads = @@ -2620,11 +3626,11 @@ OPTION(MAXDOP 1, RECOMPILE);', ( SELECT high_runnable_percent = - '' + + '''' + RTRIM(y.runnable_pct) + - '% of ' + + ''% of '' + RTRIM(y.total) + - ' queries are waiting to get on a CPU.' + '' queries are waiting to get on a CPU.'' FROM ( SELECT @@ -2647,7 +3653,7 @@ OPTION(MAXDOP 1, RECOMPILE);', SUM ( CASE - WHEN der.status = N'runnable' + WHEN der.status = N''runnable'' THEN 1 ELSE 0 END @@ -2659,38 +3665,65 @@ OPTION(MAXDOP 1, RECOMPILE);', WHERE y.runnable_pct >= 10 AND y.total >= 4 ) AS r - WHERE dos.status = N'VISIBLE ONLINE' + WHERE dos.status = N''VISIBLE ONLINE'' OPTION(MAXDOP 1, RECOMPILE); + '; + + IF @log_to_table = 0 + BEGIN + EXECUTE sys.sp_executesql + @cpu_threads; + END; + + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_cpu + N' + ( + total_threads, + used_threads, + available_threads, + reserved_worker_count, + threads_waiting_for_cpu, + requests_waiting_for_threads, + current_workers, + total_active_request_count, + total_queued_request_count, + total_blocked_task_count, + total_active_parallel_thread_count, + avg_runnable_tasks_count, + high_runnable_percent + )' + + REPLACE + ( + @cpu_threads, + N'SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;', + N'' + ); + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql; + END; /* Any current threadpool waits? */ - INSERT - @threadpool_waits - ( - session_id, - wait_duration_ms, - threadpool_waits - ) - SELECT - dowt.session_id, - dowt.wait_duration_ms, - threadpool_waits = - dowt.wait_type - FROM sys.dm_os_waiting_tasks AS dowt - WHERE dowt.wait_type = N'THREADPOOL' - ORDER BY - dowt.wait_duration_ms DESC - OPTION(MAXDOP 1, RECOMPILE); - - IF @@ROWCOUNT = 0 - BEGIN - SELECT - THREADPOOL = N'No current THREADPOOL waits'; - END; - ELSE + IF @log_to_table = 0 BEGIN + INSERT + @threadpool_waits + ( + session_id, + wait_duration_ms, + threadpool_waits + ) SELECT dowt.session_id, dowt.wait_duration_ms, @@ -2701,6 +3734,25 @@ OPTION(MAXDOP 1, RECOMPILE);', ORDER BY dowt.wait_duration_ms DESC OPTION(MAXDOP 1, RECOMPILE); + + IF @@ROWCOUNT = 0 + BEGIN + SELECT + THREADPOOL = N'No current THREADPOOL waits'; + END; + ELSE + BEGIN + SELECT + dowt.session_id, + dowt.wait_duration_ms, + threadpool_waits = + dowt.wait_type + FROM sys.dm_os_waiting_tasks AS dowt + WHERE dowt.wait_type = N'THREADPOOL' + ORDER BY + dowt.wait_duration_ms DESC + OPTION(MAXDOP 1, RECOMPILE); + END; END; @@ -2815,7 +3867,7 @@ OPTION(MAXDOP 1, RECOMPILE);', + CONVERT ( - nvarchar(MAX), + nvarchar(max), CASE WHEN @skip_plan_xml = 0 THEN N' @@ -2851,7 +3903,7 @@ OPTION(MAXDOP 1, RECOMPILE);', ) + CONVERT ( - nvarchar(MAX), + nvarchar(max), N' statement_start_offset = (der.statement_start_offset / 2) + 1, @@ -2918,7 +3970,7 @@ OPTION(MAXDOP 1, RECOMPILE);', WHEN @cool_new_columns = 1 THEN CONVERT ( - nvarchar(MAX), + nvarchar(max), N', der.dop, der.parallel_worker_count' @@ -2927,7 +3979,7 @@ OPTION(MAXDOP 1, RECOMPILE);', END + CONVERT ( - nvarchar(MAX), + nvarchar(max), N' FROM sys.dm_exec_requests AS der OUTER APPLY sys.dm_exec_sql_text(der.plan_handle) AS dest @@ -2969,8 +4021,77 @@ OPTION(MAXDOP 1, RECOMPILE);', PRINT SUBSTRING(@cpu_sql, 4001, 8000); END; - EXECUTE sys.sp_executesql - @cpu_sql; + IF @log_to_table = 0 + BEGIN + EXECUTE sys.sp_executesql + @cpu_sql; + END; + + IF @log_to_table = 1 + BEGIN + SET @insert_sql = N' + SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; + INSERT INTO ' + @log_table_cpu_queries + N' + ( + session_id, + database_name, + duration, + sql_text, + query_plan_xml' + + CASE + WHEN @live_plans = 1 + THEN N', + live_query_plan' + ELSE N'' + END + N', + statement_start_offset, + statement_end_offset, + plan_handle, + status, + blocking_session_id, + wait_type, + wait_time_ms, + wait_resource, + cpu_time_ms, + total_elapsed_time_ms, + reads, + writes, + logical_reads, + granted_query_memory_gb, + transaction_isolation_level' + + CASE + WHEN @cool_new_columns = 1 + THEN N', + dop, + parallel_worker_count' + ELSE N'' + END + N' + )' + + REPLACE + ( + REPLACE + ( + REPLACE + ( + @cpu_sql, + N'SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;', + N'' + ), + N'SET LOCK_TIMEOUT 1000;', + N'' + ), + N'SET LOCK_TIMEOUT -1;', + N'' + ); + + IF @debug = 1 + BEGIN + PRINT @insert_sql; + END; + + EXECUTE sys.sp_executesql + @insert_sql; + END; END; /*End not skipping queries*/ END; /*End CPU checks*/ @@ -3104,6 +4225,38 @@ OPTION(MAXDOP 1, RECOMPILE);', memory_grant_cap = @memory_grant_cap; + SELECT + pattern = + 'logging parameters', + log_to_table = + @log_to_table, + log_database_name = + @log_database_name, + log_schema_name = + @log_schema_name, + log_table_name_prefix = + @log_table_name_prefix, + log_database_schema = + @log_database_schema, + log_table_waits = + @log_table_waits, + log_table_file_metrics = + @log_table_file_metrics, + log_table_perfmon = + @log_table_perfmon, + log_table_memory = + @log_table_memory, + log_table_cpu = + @log_table_cpu, + log_table_memory_consumers = + @log_table_memory_consumers, + log_table_memory_queries = + @log_table_memory_queries, + log_table_cpu_queries = + @log_table_cpu_queries, + log_table_cpu_events = + @log_table_cpu_events; + END; /*End Debug*/ END; /*Final End*/ GO diff --git a/sp_QuickieStore/Examples.sql b/sp_QuickieStore/Examples.sql deleted file mode 100644 index 0e39a6e2..00000000 --- a/sp_QuickieStore/Examples.sql +++ /dev/null @@ -1,437 +0,0 @@ -/* -███████╗██╗ ██╗ █████╗ ███╗ ███╗██████╗ ██╗ ███████╗ -██╔════╝╚██╗██╔╝██╔══██╗████╗ ████║██╔══██╗██║ ██╔════╝ -█████╗ ╚███╔╝ ███████║██╔████╔██║██████╔╝██║ █████╗ -██╔══╝ ██╔██╗ ██╔══██║██║╚██╔╝██║██╔═══╝ ██║ ██╔══╝ -███████╗██╔╝ ██╗██║ ██║██║ ╚═╝ ██║██║ ███████╗███████╗ -╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚══════╝╚══════╝ - - ██████╗ █████╗ ██╗ ██╗ ███████╗ -██╔════╝██╔══██╗██║ ██║ ██╔════╝ -██║ ███████║██║ ██║ ███████╗ -██║ ██╔══██║██║ ██║ ╚════██║ -╚██████╗██║ ██║███████╗███████╗███████║ - ╚═════╝╚═╝ ╚═╝╚══════╝╚══════╝╚══════╝ - -Copyright 2025 Darling Data, LLC -https://www.erikdarling.com/ - -For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData -*/ - -/*Get help!*/ -EXECUTE dbo.sp_QuickieStore - @help = 1; - -/*The default is finding the top 10 sorted by CPU in the last seven days.*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013'; - -/*Find top 10 sorted by memory*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory', - @top = 10; - -/*Find top 10 in each user database sorted by cpu*/ -EXECUTE dbo.sp_QuickieStore - @get_all_databases = 1, - @sort_order = 'cpu', - @top = 10; - -/*Search for specific query_ids*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @include_query_ids = '13977, 13978'; - - -/*Search for specific plan_ids*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory', - @top = 10, - @start_date = '20210320', - @include_plan_ids = '1896, 1897'; - - -/*Ignore for specific query_ids*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @ignore_query_ids = '13977, 13978'; - - -/*Ignore for specific plan_ids*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory', - @top = 10, - @start_date = '20210320', - @ignore_plan_ids = '1896, 1897'; - - -/*Search for queries within a date range*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory', - @top = 10, - @start_date = '20210320', - @end_date = '20210321'; - -/*Filter out weekends and anything outside of your choice of hours.*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @workdays = 1, - @work_start = '8am', - @work_end = '6pm' - - -/*Search for queries with a minimum execution count*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @execution_count = 10; - - -/*Search for queries over a specific duration*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @duration_ms = 10000; - - -/*Use wait filter to search for queries responsible for high waits*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @wait_filter = 'memory', - @sort_order = 'memory'; - -/*We also support using wait types as a sort order, see the documentation for the full list. -The wait-related sort orders are special because we add an extra column for the duration of the wait type you are asking for. -It's all the way over on the right. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory waits'; - -/*You can also sort by total wait time across all waits. */ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'total waits'; - - -/*Search for queries with a specific execution type -When we do not provide this parameter, we grab all types. -This example grabs "aborted" queries, which are queries cancelled by the client. -This is a great way to find timeouts. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @execution_type_desc = 'aborted'; - -/*Search for queries that errored -As above, but for "exception" queries. -This grabs queries that were cancelled by throwing exceptions. -It's no substitute for proper error monitoring, but it can be a good early warning. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @execution_type_desc = 'exception'; - -/*Search for queries that finished normally -As above, but for "regular" queries. -This grabs queries that were not cancelled. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @execution_type_desc = 'regular'; - - -/*Search for a specific stored procedure*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @procedure_name = 'top_percent_sniffer'; - -/*Search for a specific stored procedure in a specific schema*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @procedure_schema = 'not_dbo' - @procedure_name = 'top_percent_sniffer'; - -/*Search for specific query text*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @query_text_search = 'WITH Comment'; - -/*Search for specific query text, with brackets automatically escaped. -Commonly needed when dealing with ORM queries. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @query_text_search = 'FROM [users] AS [t0]', - @escape_brackets = 1; - -/*By default, We use '\' to escape when @escape_brackets = 1 is set. -Maybe you want something else. -Provide it with @escape_character. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @query_text_search = 'FROM foo\bar AS [t0]', - @escape_character = '~' - @escape_brackets = 1; - - -/*Find every reference to a particular table in your Query Store data, sorted by their execution counts. -Quite expensive! -Handy when tuning or finding dependencies, but only as good as what your Query Store has captured. -Makes use of @get_all_databases = 1, which lets you search all user databases. -Note the abuse of @start_date. By setting it very far back in the past and leaving @end_date unspecified, we cover all of the data. -We also abuse @top by setting it very high. -*/ -EXECUTE dbo.sp_QuickieStore - @get_all_databases = 1, - @start_date = '20000101', - @sort_order = 'executions', - @query_text_search = 'MyTable', - @top = 100; - -/*Filter out certain query text with @query_text_search_not. -Good for when @query_text_search gets false positives. -After all, it's only doing string searching. -*/ -EXECUTE dbo.sp_QuickieStore - @get_all_databases = 1, - @start_date = '20000101', - @sort_order = 'executions', - @query_text_search = 'MyTable', - @query_text_search_not = 'MyTable_secret_backup' - @top = 100; - - -/*What happened recently on a database?*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013' - @sort_order = 'recent'; - -/*What happened recently that referenced my database? -Good for finding cross-database queries, such as when checking if a database is dead code. -Don't forget that queries in a database do not need to reference it explicitly! -*/ -EXECUTE dbo.sp_QuickieStore - @get_all_databases = 1, - @start_date = '20000101', - @sort_order = 'recent', - @query_text_search = 'StackOverflow2013' - @top = 10; - -/*Only return queries with feedback (2022+)*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @only_query_with_feedback = 1; - -/*Only return queries with variants (2022+)*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @only_query_with_variants = 1; - -/*Only return queries with forced plans (2022+)*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @only_query_with_forced_plans = 1; - -/*Only return queries with forced plan failures (2022+)*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @only_query_with_forced_plan_failures = 1; - -/*Only return queries with query hints (2022+)*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @only_query_with_hints = 1; - -/*Use expert mode to return additional columns*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory', - @top = 10, - @expert_mode = 1; - - -/*Use format output to add commas to larger numbers -This is enabled by default. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory', - @top = 10, - @format_output = 1; - -/*Disable format output to remove commas.*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'memory', - @top = 10, - @format_output = 0; - -/*Change the timezone show in your outputs. -This is only an output-formatting change. -It does not change how dates are processed. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @timezone = 'Egypt Standard Time'; - -/*Debugging something complex? -Hide the bottom table with @hide_help_table = 1 when you need more room. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @hide_help_table = 1, - @sort_order = 'latch waits', - @top = 50; - -/*Search by query hashes*/ -EXECUTE dbo.sp_QuickieStore - @include_query_hashes = '0x1AB614B461F4D769,0x1CD777B461F4D769'; - -/*Search by plan hashes*/ -EXECUTE dbo.sp_QuickieStore - @include_plan_hashes = '0x6B84B820B8B38564,0x6B84B999D7B38564'; - -/*Search by SQL Handles -Do you need to find if one Query Store is tracking the same query that is present in another database's Query Store? If so, use the statement_sql_handle to do that. -This helps with scenarios where you have multiple production databases which have the same schema and you want to compare performance across Query Stores. -*/ -EXECUTE dbo.sp_QuickieStore - @include_sql_handles = - '0x0900F46AC89E66DF744C8A0AD4FD3D3306B90000000000000000000000000000000000000000000000000000,0x0200000AC89E66DF744C8A0AD4FD3D3306B90000000000000000000000000000000000000000000000000000'; - -/*Search, but ignoring some query hashes*/ -EXECUTE dbo.sp_QuickieStore - @ignore_query_hashes = '0x1AB614B461F4D769,0x1CD777B461F4D769'; - -/*Search, but ignoring some plan hashes*/ -EXECUTE dbo.sp_QuickieStore - @ignore_plan_hashes = '0x6B84B820B8B38564,0x6B84B999D7B38564'; - -/*Search, but ignoring some SQL Handles*/ -EXECUTE dbo.sp_QuickieStore - @ignore_sql_handles = - '0x0900F46AC89E66DF744C8A0AD4FD3D3306B90000000000000000000000000000000000000000000000000000,0x0200000AC89E66DF744C8A0AD4FD3D3306B90000000000000000000000000000000000000000000000000000'; - -/*What query hashes have the most plans? -This sort order is special because it needs to return multiple rows for each of the @top hashes it looks at. -It is also special because it adds some new columns all the way over on the right of the output. -*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'plan count by hashes'; - -/*Check for regressions. -Specifically, this checks for queries that did more logical reads last week than this week. -The default dates are helpful here. The default @start_date and @end_date specify last week for us and @regression_baseline_end_date defaults to being one week after @regression_baseline_start_date. -However, we need to specify @regression_baseline_start_date so that sp_QuickieStore knows to check for regressions. -Searches by query hash, so you will won't be caught out by identical queries with different query ids. -*/ -DECLARE @TwoWeekAgo datetimeoffset(7) = DATEADD(WEEK, -2, SYSDATETIMEOFFSET()); - -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'logical reads', - @regression_baseline_start_date = @TwoWeekAgo; - -/*Check for improved queries. -I deleted some indexes yesterday. -Let's see what's doing less writes today. -Since we're checking for improvements rather than regressions, we use @regression_direction = 'improved'. -This is a good chance to point out that the @end_date parameters do comparisons with < rather than <=. -The @start_data parameters, of course, use >=. -*/ -DECLARE @StartOfYesterday datetimeoffset(7) = CONVERT(date, DATEADD(DAY, -1, SYSDATETIMEOFFSET())), - @StartOfToday datetimeoffset(7) = CONVERT(date, SYSDATETIMEOFFSET()), - @StartOfTomorrow datetimeoffset(7) = CONVERT(date, DATEADD(DAY, 1, SYSDATETIMEOFFSET())); - -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'writes', - @regression_direction = 'improved', - @regression_baseline_start_date = @StartOfYesterday, - @regression_baseline_end_date = @StartOfToday, - @start_date = @StartOfToday, - @end_date = @StartOfTomorrow; - -/*Check for percentage changes in performance. -By default, our @regression parameters have us check for changes in the raw numbers. -It's just plain subtraction: new minus old. -This means that a query that used to read hardly anything from disk but now reads triple that is indistinguishable from the noise in a query that reads lots. -To get percentage changes instead, specify @regression_comparator = 'relative'. -The default is @regression_comparator = 'absolute'. - -To see the difference, run `sp_QuickieStore` twice. -To save space on your screen, we will specify @hide_help_table = 1 to hide the table normally at the bottom of the normal output. -*/ -DECLARE @TwoWeekAgo datetimeoffset(7) = DATEADD(WEEK, -2, SYSDATETIMEOFFSET()); - -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'physical reads', - @hide_help_table = 1, - @regression_comparator = 'relative', - @regression_baseline_start_date = @TwoWeekAgo; - -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @sort_order = 'physical reads', - @hide_help_table = 1, - @regression_comparator = 'absolute', - @regression_baseline_start_date = @TwoWeekAgo; - -/*Check for changes in modulus. -What if you're looking for sheer size of changes, rather than the direction of the change? -For example, you might care about a 30 second reduction in duration just as much as a 30 second increase. -Use @regression_direction = 'absolute'. -And while we're at it, let's check all user databases with @get_all_databases = 1. -*/ -DECLARE @TwoWeekAgo datetimeoffset(7) = DATEADD(WEEK, -2, SYSDATETIMEOFFSET()); - -EXECUTE dbo.sp_QuickieStore - @get_all_databases = 1, - @sort_order = 'duration', - @regression_direction = 'absolute', - @regression_baseline_start_date = @TwoWeekAgo; - -/*Get version info.*/ -DECLARE @version_output varchar(30), - @version_date_output datetime; - -EXECUTE sp_QuickieStore - @version = @version_output OUTPUT, - @version_date = @version_date_output OUTPUT; - -SELECT - Version = @version_output, - VersionDate = @version_date_output; - -/*Search for queries that take a while and return lots of rows on average*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @top = 10, - @sort_order = 'rows', - @duration_ms = 20000; - - -/*Troubleshoot performance*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @troubleshoot_performance = 1; - -/*Debug dynamic SQL and temp table contents*/ -EXECUTE dbo.sp_QuickieStore - @database_name = 'StackOverflow2013', - @debug = 1; diff --git a/sp_QuickieStore/README.md b/sp_QuickieStore/README.md new file mode 100644 index 00000000..82d60123 --- /dev/null +++ b/sp_QuickieStore/README.md @@ -0,0 +1,132 @@ + + +# sp_QuickieStore + +This procedure will dig into Query Store data for a specific database, or all databases with Query Store enabled. + +It's designed to run as quickly as possible, but there are some circumstances that prevent me from realizing my ultimate dream. + +The big upside of using this stored procedure over the GUI is that you can search for specific items in Query Store, by: +* query_id +* plan_id +* query hash +* sql handle +* module name +* query text +* query type (ad hoc or from a module) + +You can also choose to filter out specific queries by those, too. + +And you can do all that without worrying about incorrect data from the GUI, which doesn't handle UTC conversion correctly when filtering data. + +By default, it will return the top 10 queries by average CPU. You can configure all sorts of things to look at queries by other metrics, or just specific queries. + +Use the `@expert_mode` parameter to return additional details. + +## Parameters + +| parameter_name | data_type | description | valid_inputs | defaults | +|-----------------------------------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------| +| @database_name | sysname | the name of the database you want to look at query store in | a database name with query store enabled | NULL; current database name if NULL | +| @sort_order | varchar | the runtime metric you want to prioritize results by | cpu, logical reads, physical reads, writes, duration, memory, tempdb, executions, recent, plan count by hashes, cpu waits, lock waits, locks waits, latch waits, latches waits, buffer latch waits, buffer latches waits, buffer io waits, log waits, log io waits, network waits, network io waits, parallel waits, parallelism waits, memory waits, total waits, rows | cpu | +| @top | bigint | the number of queries you want to pull back | a positive integer between 1 and 9,223,372,036,854,775,807 | 10 | +| @start_date | datetimeoffset | the begin date of your search, will be converted to UTC internally | January 1, 1753, through December 31, 9999 | the last seven days | +| @end_date | datetimeoffset | the end date of your search, will be converted to UTC internally | January 1, 1753, through December 31, 9999 | NULL | +| @timezone | sysname | user specified time zone to override dates displayed in results | SELECT tzi.* FROM sys.time_zone_info AS tzi; | NULL | +| @execution_count | bigint | the minimum number of executions a query must have | a positive integer between 1 and 9,223,372,036,854,775,807 | NULL | +| @duration_ms | bigint | the minimum duration a query must have to show up in results | a positive integer between 1 and 9,223,372,036,854,775,807 | NULL | +| @execution_type_desc | nvarchar | the type of execution you want to filter by (regular, aborted, exception) | regular, aborted, exception | NULL | +| @procedure_schema | sysname | the schema of the procedure you're searching for | a valid schema in your database | NULL; dbo if NULL and procedure name is not NULL | +| @procedure_name | sysname | the name of the programmable object you're searching for | a valid programmable object in your database, can use wildcards | NULL | +| @include_plan_ids | nvarchar | a list of plan ids to search for | a string; comma separated for multiple ids | NULL | +| @include_query_ids | nvarchar | a list of query ids to search for | a string; comma separated for multiple ids | NULL | +| @include_query_hashes | nvarchar | a list of query hashes to search for | a string; comma separated for multiple hashes | NULL | +| @include_plan_hashes | nvarchar | a list of query plan hashes to search for | a string; comma separated for multiple hashes | NULL | +| @include_sql_handles | nvarchar | a list of sql handles to search for | a string; comma separated for multiple handles | NULL | +| @ignore_plan_ids | nvarchar | a list of plan ids to ignore | a string; comma separated for multiple ids | NULL | +| @ignore_query_ids | nvarchar | a list of query ids to ignore | a string; comma separated for multiple ids | NULL | +| @ignore_query_hashes | nvarchar | a list of query hashes to ignore | a string; comma separated for multiple hashes | NULL | +| @ignore_plan_hashes | nvarchar | a list of query plan hashes to ignore | a string; comma separated for multiple hashes | NULL | +| @ignore_sql_handles | nvarchar | a list of sql handles to ignore | a string; comma separated for multiple handles | NULL | +| @query_text_search | nvarchar | query text to search for | a string; leading and trailing wildcards will be added if missing | NULL | +| @query_text_search_not | nvarchar | query text to exclude | a string; leading and trailing wildcards will be added if missing | NULL | +| @escape_brackets | bit | Set this bit to 1 to search for query text containing square brackets (common in .NET Entity Framework and other ORM queries) | 0 or 1 | 0 | +| @escape_character | nchar | Sets the ESCAPE character for special character searches, defaults to the SQL standard backslash (\) character | some escape character, SQL standard is backslash (\) | \ | +| @only_queries_with_hints | bit | only return queries with query hints | 0 or 1 | 0 | +| @only_queries_with_feedback | bit | only return queries with query feedback | 0 or 1 | 0 | +| @only_queries_with_variants | bit | only return queries with query variants | 0 or 1 | 0 | +| @only_queries_with_forced_plans | bit | only return queries with forced plans | 0 or 1 | 0 | +| @only_queries_with_forced_plan_failures | bit | only return queries with forced plan failures | 0 or 1 | 0 | +| @wait_filter | varchar | wait category to search for; category details are below | cpu, lock, latch, buffer latch, buffer io, log io, network io, parallelism, memory | NULL | +| @query_type | varchar | filter for only ad hoc queries or only from queries from modules | ad hoc, adhoc, proc, procedure, whatever. | NULL | +| @expert_mode | bit | returns additional columns and results | 0 or 1 | 0 | +| @hide_help_table | bit | hides the "bottom table" that shows help and support information | 0 or 1 | 0 | +| @format_output | bit | returns numbers formatted with commas | 0 or 1 | 1 | +| @get_all_databases | bit | looks for query store enabled user databases and returns combined results from all of them | 0 or 1 | 0 | +| @include_databases | nvarchar(4000) | comma-separated list of databases to include (only when @get_all_databases = 1) | a string; comma separated database names | NULL | +| @exclude_databases | nvarchar(4000) | comma-separated list of databases to exclude (only when @get_all_databases = 1) | a string; comma separated database names | NULL | +| @workdays | bit | use this to filter out weekends and after-hours queries | 0 or 1 | 0 | +| @work_start | time | use this to set a specific start of your work days | a time like 8am, 9am or something | 9am | +| @work_end | time | use this to set a specific end of your work days | a time like 5pm, 6pm or something | 5pm | +| @regression_baseline_start_date | datetimeoffset | the begin date of the baseline that you are checking for regressions against (if any), will be converted to UTC internally | January 1, 1753, through December 31, 9999 | NULL | +| @regression_baseline_end_date | datetimeoffset | the end date of the baseline that you are checking for regressions against (if any), will be converted to UTC internally | January 1, 1753, through December 31, 9999 | NULL; One week after @regression_baseline_start_date if that is specified | +| @regression_comparator | varchar | what difference to use ('relative' or 'absolute') when comparing @sort_order's metric for the normal time period with any regression time period. | relative, absolute | NULL; absolute if @regression_baseline_start_date is specified | +| @regression_direction | varchar | when comparing against any regression baseline, what do you want the results sorted by ('magnitude', 'improved', or 'regressed')? | regressed, worse, improved, better, magnitude, absolute, whatever | NULL; regressed if @regression_baseline_start_date is specified | +| @include_query_hash_totals | bit | will add an additional column to final output with total resource usage by query hash | 0 or 1 | 0 | +| @help | bit | how you got here | 0 or 1 | 0 | +| @debug | bit | prints dynamic sql, statement length, parameter and variable values, and raw temp table contents | 0 or 1 | 0 | +| @troubleshoot_performance | bit | set statistics xml on for queries against views | 0 or 1 | 0 | +| @version | varchar | OUTPUT; for support | none; OUTPUT | none; OUTPUT | +| @version_date | datetime | OUTPUT; for support | none; OUTPUT | none; OUTPUT | + +## Examples + +```sql +-- Basic execution - returns top 10 queries by CPU +EXECUTE dbo.sp_QuickieStore; + +-- Look at top 20 queries by logical reads +EXECUTE dbo.sp_QuickieStore + @sort_order = 'logical reads', + @top = 20; + +-- Search for a specific query text +EXECUTE dbo.sp_QuickieStore + @query_text_search = 'SELECT * FROM Orders'; + +-- Find queries from a specific procedure +EXECUTE dbo.sp_QuickieStore + @procedure_name = 'usp_GetCustomerOrders'; + +-- Filter to queries that executed at least 1000 times +EXECUTE dbo.sp_QuickieStore + @execution_count = 1000; + +-- Show queries with a minimum duration of 500ms +EXECUTE dbo.sp_QuickieStore + @duration_ms = 500; + +-- Look for regressions against a baseline period +EXECUTE dbo.sp_QuickieStore + @regression_baseline_start_date = '2025-01-01', + @regression_baseline_end_date = '2025-01-08', + @regression_direction = 'regressed'; + +-- Expert mode for additional details +EXECUTE dbo.sp_QuickieStore + @expert_mode = 1; + +-- Get data from all databases with Query Store enabled, except for specific ones +EXECUTE dbo.sp_QuickieStore + @get_all_databases = 1, + @exclude_databases = 'Head, Shoulders, Knees, Toes'; + +-- Get data from only specific databases with Query Store enabled +EXECUTE dbo.sp_QuickieStore + @get_all_databases = 1, + @include_databases = 'StacOverflow2013, StackOverflow2010'; +``` + +## Resources +* [YouTube playlist](https://www.youtube.com/playlist?list=PLt4QZ-7lfQie1XZHEm0HN-Zt1S7LFEx1P) +* [Blog post](https://www.erikdarlingdata.com/sp_quickiestore/) \ No newline at end of file diff --git a/sp_QuickieStore/sp_QuickieStore.sql b/sp_QuickieStore/sp_QuickieStore.sql index c1a548d3..3b1ab85f 100644 --- a/sp_QuickieStore/sp_QuickieStore.sql +++ b/sp_QuickieStore/sp_QuickieStore.sql @@ -41,7 +41,7 @@ EXECUTE sp_QuickieStore @troubleshoot_performance = 1; For support, head over to GitHub: -https://github.com/erikdarlingdata/DarlingData +https://code.erikdarling.com */ @@ -90,6 +90,8 @@ ALTER PROCEDURE @hide_help_table bit = 0, /*hides the "bottom table" that shows help and support information*/ @format_output bit = 1, /*returns numbers formatted with commas*/ @get_all_databases bit = 0, /*looks for query store enabled user databases and returns combined results from all of them*/ + @include_databases nvarchar(max) = NULL, /*comma-separated list of databases to include (only when @get_all_databases = 1)*/ + @exclude_databases nvarchar(max) = NULL, /*comma-separated list of databases to exclude (only when @get_all_databases = 1)*/ @workdays bit = 0, /*Use this to filter out weekends and after-hours queries*/ @work_start time(0) = '9am', /*Use this to set a specific start of your work days*/ @work_end time(0) = '5pm', /*Use this to set a specific end of your work days*/ @@ -109,6 +111,7 @@ AS BEGIN SET STATISTICS XML OFF; SET NOCOUNT ON; +SET XACT_ABORT OFF; SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; BEGIN TRY @@ -136,8 +139,8 @@ END; These are for your outputs. */ SELECT - @version = '5.1', - @version_date = '20250101'; + @version = '5.4', + @version_date = '20250401'; /* Helpful section! For help. @@ -202,6 +205,8 @@ BEGIN WHEN N'@hide_help_table' THEN 'hides the "bottom table" that shows help and support information' WHEN N'@format_output' THEN 'returns numbers formatted with commas' WHEN N'@get_all_databases' THEN 'looks for query store enabled user databases and returns combined results from all of them' + WHEN N'@include_databases' THEN 'comma-separated list of databases to include (only when @get_all_databases = 1)' + WHEN N'@exclude_databases' THEN 'comma-separated list of databases to exclude (only when @get_all_databases = 1)' WHEN N'@workdays' THEN 'use this to filter out weekends and after-hours queries' WHEN N'@work_start' THEN 'use this to set a specific start of your work days' WHEN N'@work_end' THEN 'use this to set a specific end of your work days' @@ -255,6 +260,8 @@ BEGIN WHEN N'@hide_help_table' THEN '0 or 1' WHEN N'@format_output' THEN '0 or 1' WHEN N'@get_all_databases' THEN '0 or 1' + WHEN N'@include_databases' THEN 'a string; comma separated database names' + WHEN N'@exclude_databases' THEN 'a string; comma separated database names' WHEN N'@workdays' THEN '0 or 1' WHEN N'@work_start' THEN 'a time like 8am, 9am or something' WHEN N'@work_end' THEN 'a time like 5pm, 6pm or something' @@ -308,6 +315,8 @@ BEGIN WHEN N'@hide_help_table' THEN '0' WHEN N'@format_output' THEN '1' WHEN N'@get_all_databases' THEN '0' + WHEN N'@include_databases' THEN 'NULL' + WHEN N'@exclude_databases' THEN 'NULL' WHEN N'@workdays' THEN '0' WHEN N'@work_start' THEN '9am' WHEN N'@work_end' THEN '5pm' @@ -552,11 +561,11 @@ query hash has. CREATE TABLE #plan_ids_with_query_hashes ( - database_id int NOT NULL, + database_id integer NOT NULL, plan_id bigint NOT NULL, query_hash binary(8) NOT NULL, - plan_hash_count_for_query_hash int NOT NULL, - PRIMARY KEY CLUSTERED (database_id, plan_id, query_hash) + plan_hash_count_for_query_hash integer NOT NULL, + PRIMARY KEY CLUSTERED (database_id, plan_id, query_hash) ); /* @@ -572,11 +581,11 @@ therefore every sp_executesql). CREATE TABLE #plan_ids_with_total_waits ( - database_id int NOT NULL, + database_id integer NOT NULL, plan_id bigint NOT NULL, from_regression_baseline varchar(3) NOT NULL, total_query_wait_time_ms bigint NOT NULL, - PRIMARY KEY CLUSTERED(database_id, plan_id, from_regression_baseline) + PRIMARY KEY CLUSTERED(database_id, plan_id, from_regression_baseline) ); /* @@ -618,11 +627,11 @@ on to our final output. CREATE TABLE #regression_changes ( - database_id int NOT NULL, + database_id integer NOT NULL, plan_id bigint NOT NULL, query_hash binary(8) NOT NULL, change_since_regression_time_period float NULL, - PRIMARY KEY CLUSTERED (database_id, plan_id, query_hash) + PRIMARY KEY CLUSTERED (database_id, plan_id, query_hash) ); /* @@ -791,7 +800,7 @@ Query Store Setup CREATE TABLE #database_query_store_options ( - database_id int NOT NULL, + database_id integer NOT NULL, desired_state_desc nvarchar(60) NULL, actual_state_desc nvarchar(60) NULL, readonly_reason nvarchar(100) NULL, @@ -802,10 +811,10 @@ CREATE TABLE stale_query_threshold_days bigint NULL, max_plans_per_query bigint NULL, query_capture_mode_desc nvarchar(60) NULL, - capture_policy_execution_count int NULL, + capture_policy_execution_count integer NULL, capture_policy_total_compile_cpu_time_ms bigint NULL, capture_policy_total_execution_cpu_time_ms bigint NULL, - capture_policy_stale_threshold_hours int NULL, + capture_policy_stale_threshold_hours integer NULL, size_based_cleanup_mode_desc nvarchar(60) NULL, wait_stats_capture_mode_desc nvarchar(60) NULL ); @@ -816,7 +825,7 @@ Query Store Trouble CREATE TABLE #query_store_trouble ( - database_id int NOT NULL, + database_id integer NOT NULL, desired_state_desc nvarchar(60) NULL, actual_state_desc nvarchar(60) NULL, readonly_reason nvarchar(100) NULL, @@ -836,7 +845,7 @@ Plans and Plan information CREATE TABLE #query_store_plan ( - database_id int NOT NULL, + database_id integer NOT NULL, plan_id bigint NOT NULL, query_id bigint NOT NULL, all_plan_ids varchar(MAX), @@ -870,7 +879,7 @@ Queries and Compile Information CREATE TABLE #query_store_query ( - database_id int NOT NULL, + database_id integer NOT NULL, query_id bigint NOT NULL, query_text_id bigint NOT NULL, context_settings_id bigint NOT NULL, @@ -947,7 +956,7 @@ Query Text And Columns From sys.dm_exec_query_stats CREATE TABLE #query_store_query_text ( - database_id int NOT NULL, + database_id integer NOT NULL, query_text_id bigint NOT NULL, query_sql_text xml NULL, statement_sql_handle varbinary(64) NULL, @@ -1010,7 +1019,7 @@ Runtime stats information CREATE TABLE #query_store_runtime_stats ( - database_id int NOT NULL, + database_id integer NOT NULL, runtime_stats_id bigint NOT NULL, plan_id bigint NOT NULL, runtime_stats_interval_id bigint NOT NULL, @@ -1113,7 +1122,7 @@ Wait Stats, When Available (2017+) CREATE TABLE #query_store_wait_stats ( - database_id int NOT NULL, + database_id integer NOT NULL, plan_id bigint NOT NULL, wait_category_desc nvarchar(60) NOT NULL, total_query_wait_time_ms bigint NOT NULL, @@ -1129,17 +1138,17 @@ Context is everything CREATE TABLE #query_context_settings ( - database_id int NOT NULL, + database_id integer NOT NULL, context_settings_id bigint NOT NULL, set_options varbinary(8) NULL, language_id smallint NOT NULL, date_format smallint NOT NULL, date_first tinyint NOT NULL, status varbinary(2) NULL, - required_cursor_options int NOT NULL, - acceptable_cursor_options int NOT NULL, + required_cursor_options integer NOT NULL, + acceptable_cursor_options integer NOT NULL, merge_action_type smallint NOT NULL, - default_schema_id int NOT NULL, + default_schema_id integer NOT NULL, is_replication_specific bit NOT NULL, is_contained varbinary(1) NULL ); @@ -1150,14 +1159,14 @@ Feed me Seymour CREATE TABLE #query_store_plan_feedback ( - database_id int NOT NULL, - plan_feedback_id bigint, - plan_id bigint, - feature_desc nvarchar(120), - feedback_data nvarchar(MAX), - state_desc nvarchar(120), - create_time datetimeoffset(7), - last_updated_time datetimeoffset(7) + database_id integer NOT NULL, + plan_feedback_id bigint NOT NULL, + plan_id bigint NULL, + feature_desc nvarchar(120) NULL, + feedback_data nvarchar(MAX) NULL, + state_desc nvarchar(120) NULL, + create_time datetimeoffset(7) NOT NULL, + last_updated_time datetimeoffset(7) NULL ); /* @@ -1166,13 +1175,13 @@ America's Most Hinted CREATE TABLE #query_store_query_hints ( - database_id int NOT NULL, - query_hint_id bigint, - query_id bigint, - query_hint_text nvarchar(MAX), - last_query_hint_failure_reason_desc nvarchar(256), - query_hint_failure_count bigint, - source_desc nvarchar(256) + database_id integer NOT NULL, + query_hint_id bigint NOT NULL, + query_id bigint NOT NULL, + query_hint_text nvarchar(MAX) NULL, + last_query_hint_failure_reason_desc nvarchar(256) NULL, + query_hint_failure_count bigint NOT NULL, + source_desc nvarchar(256) NULL ); /* @@ -1181,22 +1190,22 @@ Variant? Deviant? You decide! CREATE TABLE #query_store_query_variant ( - database_id int NOT NULL, - query_variant_query_id bigint, - parent_query_id bigint, - dispatcher_plan_id bigint + database_id integer NOT NULL, + query_variant_query_id bigint NOT NULL, + parent_query_id bigint NOT NULL, + dispatcher_plan_id bigint NOT NULL ); /* Replicants */ CREATE TABLE - #query_store_replicas + #query_store_replicas ( - database_id int NOT NULL, - replica_group_id bigint, - role_type smallint, - replica_name nvarchar(1288) + database_id integer NOT NULL, + replica_group_id bigint NOT NULL, + role_type smallint NOT NULL, + replica_name nvarchar(1288) NULL ); /* @@ -1205,11 +1214,11 @@ Location, location, location CREATE TABLE #query_store_plan_forcing_locations ( - database_id int NOT NULL, - plan_forcing_location_id bigint, - query_id bigint, - plan_id bigint, - replica_group_id bigint + database_id integer NOT NULL, + plan_forcing_location_id bigint NOT NULL, + query_id bigint NOT NULL, + plan_id bigint NOT NULL, + replica_group_id bigint NOT NULL ); /* @@ -1218,10 +1227,10 @@ Trouble Loves Me CREATE TABLE #troubleshoot_performance ( - id bigint IDENTITY, - current_table nvarchar(100), - start_time datetime, - end_time datetime, + id bigint IDENTITY PRIMARY KEY CLUSTERED, + current_table nvarchar(100) NOT NULL, + start_time datetime NOT NULL, + end_time datetime NOT NULL, runtime_ms AS FORMAT ( @@ -1235,6 +1244,24 @@ CREATE TABLE ) ); +/*Gonna try gathering this based on*/ +CREATE TABLE + #query_hash_totals +( + database_id integer NOT NULL, + query_hash binary(8) NOT NULL, + total_executions bigint NOT NULL, + total_duration_ms decimal(19,2) NOT NULL, + total_cpu_time_ms decimal(19,2) NOT NULL, + total_logical_reads_mb decimal(19,2) NOT NULL, + total_physical_reads_mb decimal(19,2) NOT NULL, + total_logical_writes_mb decimal(19,2) NOT NULL, + total_clr_time_ms decimal(19,2) NOT NULL, + total_memory_mb decimal(19,2) NOT NULL, + total_rowcount decimal(19,2) NOT NULL, + PRIMARY KEY CLUSTERED(query_hash, database_id) +); + /*GET ALL THOSE DATABASES*/ CREATE TABLE #databases @@ -1242,6 +1269,318 @@ CREATE TABLE database_name sysname PRIMARY KEY CLUSTERED ); +/* Create tables for database filtering */ +CREATE TABLE + #include_databases +( + database_name sysname PRIMARY KEY +); + +CREATE TABLE + #exclude_databases +( + database_name sysname PRIMARY KEY +); + +CREATE TABLE + #requested_but_skipped_databases +( + database_name sysname PRIMARY KEY, + reason varchar(100) NOT NULL +); + +/* Create a table variable to store ALL column definitions with logical ordering */ +DECLARE + @ColumnDefinitions table +( + column_id integer + PRIMARY KEY CLUSTERED, /* Controls the ordering of columns in output */ + metric_group nvarchar(50) NOT NULL, /* Grouping (duration, cpu, etc.) */ + metric_type nvarchar(20) NOT NULL, /* Type within group (avg, total, last, min, max) */ + column_name nvarchar(100) NOT NULL, /* Column name as it appears in output */ + column_source nvarchar(MAX) NOT NULL, /* Source expression or formula */ + is_conditional bit NOT NULL, /* Is this a conditional column (depends on a parameter) */ + condition_param nvarchar(50) NULL, /* Parameter name this column depends on */ + condition_value sql_variant NULL, /* Value the parameter must have */ + expert_only bit NOT NULL, /* Only include in expert mode */ + format_pattern nvarchar(20) NULL /* Format pattern (e.g., 'N0', 'P2', NULL for no formatting) */ +); + +/* Fill the table with ALL columns, including SQL 2022 views and regression columns */ + +/* Basic metadata columns (still part of prefix, but in the table) */ +INSERT INTO + @ColumnDefinitions +( + column_id, metric_group, metric_type, column_name, column_source, is_conditional, condition_param, condition_value, expert_only, format_pattern +) +VALUES + (20, 'metadata', 'force_count', 'force_failure_count', 'qsp.force_failure_count', 0, NULL, NULL, 0, NULL), + (30, 'metadata', 'force_reason', 'last_force_failure_reason_desc', 'qsp.last_force_failure_reason_desc', 0, NULL, NULL, 0, NULL), + /* SQL 2022 specific columns */ + (40, 'sql_2022', 'feedback', 'has_query_feedback', 'CASE WHEN EXISTS (SELECT 1/0 FROM #query_store_plan_feedback AS qspf WHERE qspf.plan_id = qsp.plan_id) THEN ''Yes'' ELSE ''No'' END', 1, 'sql_2022_views', 1, 0, NULL), + (50, 'sql_2022', 'hints', 'has_query_store_hints', 'CASE WHEN EXISTS (SELECT 1/0 FROM #query_store_query_hints AS qsqh WHERE qsqh.query_id = qsp.query_id) THEN ''Yes'' ELSE ''No'' END', 1, 'sql_2022_views', 1, 0, NULL), + (60, 'sql_2022', 'variants', 'has_plan_variants', 'CASE WHEN EXISTS (SELECT 1/0 FROM #query_store_query_variant AS qsqv WHERE qsqv.query_variant_query_id = qsp.query_id) THEN ''Yes'' ELSE ''No'' END', 1, 'sql_2022_views', 1, 0, NULL), + (70, 'sql_2022', 'replay', 'has_compile_replay_script', 'qsp.has_compile_replay_script', 1, 'sql_2022_views', 1, 0, NULL), + (80, 'sql_2022', 'opt_forcing', 'is_optimized_plan_forcing_disabled', 'qsp.is_optimized_plan_forcing_disabled', 1, 'sql_2022_views', 1, 0, NULL), + (90, 'sql_2022', 'plan_type', 'plan_type_desc', 'qsp.plan_type_desc', 1, 'sql_2022_views', 1, 0, NULL), + /* New version features */ + (95, 'new_features', 'forcing_type', 'plan_forcing_type_desc', 'qsp.plan_forcing_type_desc', 1, 'new', 1, 0, NULL), + (97, 'new_features', 'top_waits', 'top_waits', 'w.top_waits', 1, 'new', 1, 0, NULL), + /* Date/time columns (not conditional, always included) */ + (100, 'execution_time', 'first', 'first_execution_time', 'CASE WHEN @timezone IS NULL THEN SWITCHOFFSET(qsrs.first_execution_time, @utc_offset_string) WHEN @timezone IS NOT NULL THEN qsrs.first_execution_time AT TIME ZONE @timezone END', 0, NULL, NULL, 0, NULL), + (110, 'execution_time', 'first_utc', 'first_execution_time_utc', 'qsrs.first_execution_time', 0, NULL, NULL, 0, NULL), + (120, 'execution_time', 'last', 'last_execution_time', 'CASE WHEN @timezone IS NULL THEN SWITCHOFFSET(qsrs.last_execution_time, @utc_offset_string) WHEN @timezone IS NOT NULL THEN qsrs.last_execution_time AT TIME ZONE @timezone END', 0, NULL, NULL, 0, NULL), + (130, 'execution_time', 'last_utc', 'last_execution_time_utc', 'qsrs.last_execution_time', 0, NULL, NULL, 0, NULL), + /* Regression mode columns */ + (140, 'regression', 'baseline', 'from_regression_baseline_time_period', 'qsrs.from_regression_baseline', 1, 'regression_mode', 1, 0, NULL), + (150, 'regression', 'hash', 'query_hash_from_regression_checking', 'regression.query_hash', 1, 'regression_mode', 1, 0, NULL), + /* Execution columns */ + (200, 'executions', 'count', 'count_executions', 'qsrs.count_executions', 0, NULL, NULL, 0, 'N0'), + (210, 'executions', 'per_second', 'executions_per_second', 'qsrs.executions_per_second', 0, NULL, NULL, 0, 'N0'), + /* Hash totals - conditionally added */ + (215, 'executions', 'count_hash', 'count_executions_by_query_hash', 'qht.total_executions', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* Duration metrics (group together avg, total, last, min, max) */ + (300, 'duration', 'avg', 'avg_duration_ms', 'qsrs.avg_duration_ms', 0, NULL, NULL, 0, 'N0'), + (310, 'duration', 'total', 'total_duration_ms', 'qsrs.total_duration_ms', 0, NULL, NULL, 0, 'N0'), + (320, 'duration', 'last', 'last_duration_ms', 'qsrs.last_duration_ms', 0, NULL, NULL, 1, 'N0'), + (330, 'duration', 'min', 'min_duration_ms', 'qsrs.min_duration_ms', 0, NULL, NULL, 1, 'N0'), + (340, 'duration', 'max', 'max_duration_ms', 'qsrs.max_duration_ms', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for duration */ + (315, 'duration', 'total_hash', 'total_duration_ms_by_query_hash', 'qht.total_duration_ms', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* CPU metrics */ + (400, 'cpu', 'avg', 'avg_cpu_time_ms', 'qsrs.avg_cpu_time_ms', 0, NULL, NULL, 0, 'N0'), + (410, 'cpu', 'total', 'total_cpu_time_ms', 'qsrs.total_cpu_time_ms', 0, NULL, NULL, 0, 'N0'), + (420, 'cpu', 'last', 'last_cpu_time_ms', 'qsrs.last_cpu_time_ms', 0, NULL, NULL, 1, 'N0'), + (430, 'cpu', 'min', 'min_cpu_time_ms', 'qsrs.min_cpu_time_ms', 0, NULL, NULL, 1, 'N0'), + (440, 'cpu', 'max', 'max_cpu_time_ms', 'qsrs.max_cpu_time_ms', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for CPU */ + (415, 'cpu', 'total_hash', 'total_cpu_time_ms_by_query_hash', 'qht.total_cpu_time_ms', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* Logical IO Reads */ + (500, 'logical_io_reads', 'avg', 'avg_logical_io_reads_mb', 'qsrs.avg_logical_io_reads_mb', 0, NULL, NULL, 0, 'N0'), + (510, 'logical_io_reads', 'total', 'total_logical_io_reads_mb', 'qsrs.total_logical_io_reads_mb', 0, NULL, NULL, 0, 'N0'), + (520, 'logical_io_reads', 'last', 'last_logical_io_reads_mb', 'qsrs.last_logical_io_reads_mb', 0, NULL, NULL, 1, 'N0'), + (530, 'logical_io_reads', 'min', 'min_logical_io_reads_mb', 'qsrs.min_logical_io_reads_mb', 0, NULL, NULL, 1, 'N0'), + (540, 'logical_io_reads', 'max', 'max_logical_io_reads_mb', 'qsrs.max_logical_io_reads_mb', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for logical reads */ + (515, 'logical_io_reads', 'total_hash', 'total_logical_io_reads_mb_by_query_hash', 'qht.total_logical_reads_mb', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* Logical IO Writes */ + (600, 'logical_io_writes', 'avg', 'avg_logical_io_writes_mb', 'qsrs.avg_logical_io_writes_mb', 0, NULL, NULL, 0, 'N0'), + (610, 'logical_io_writes', 'total', 'total_logical_io_writes_mb', 'qsrs.total_logical_io_writes_mb', 0, NULL, NULL, 0, 'N0'), + (620, 'logical_io_writes', 'last', 'last_logical_io_writes_mb', 'qsrs.last_logical_io_writes_mb', 0, NULL, NULL, 1, 'N0'), + (630, 'logical_io_writes', 'min', 'min_logical_io_writes_mb', 'qsrs.min_logical_io_writes_mb', 0, NULL, NULL, 1, 'N0'), + (640, 'logical_io_writes', 'max', 'max_logical_io_writes_mb', 'qsrs.max_logical_io_writes_mb', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for logical writes */ + (615, 'logical_io_writes', 'total_hash', 'total_logical_io_writes_mb_by_query_hash', 'qht.total_logical_writes_mb', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* Physical IO Reads */ + (700, 'physical_io_reads', 'avg', 'avg_physical_io_reads_mb', 'qsrs.avg_physical_io_reads_mb', 0, NULL, NULL, 0, 'N0'), + (710, 'physical_io_reads', 'total', 'total_physical_io_reads_mb', 'qsrs.total_physical_io_reads_mb', 0, NULL, NULL, 0, 'N0'), + (720, 'physical_io_reads', 'last', 'last_physical_io_reads_mb', 'qsrs.last_physical_io_reads_mb', 0, NULL, NULL, 1, 'N0'), + (730, 'physical_io_reads', 'min', 'min_physical_io_reads_mb', 'qsrs.min_physical_io_reads_mb', 0, NULL, NULL, 1, 'N0'), + (740, 'physical_io_reads', 'max', 'max_physical_io_reads_mb', 'qsrs.max_physical_io_reads_mb', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for physical reads */ + (715, 'physical_io_reads', 'total_hash', 'total_physical_io_reads_mb_by_query_hash', 'qht.total_physical_reads_mb', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* CLR Time */ + (800, 'clr_time', 'avg', 'avg_clr_time_ms', 'qsrs.avg_clr_time_ms', 0, NULL, NULL, 0, 'N0'), + (810, 'clr_time', 'total', 'total_clr_time_ms', 'qsrs.total_clr_time_ms', 0, NULL, NULL, 0, 'N0'), + (820, 'clr_time', 'last', 'last_clr_time_ms', 'qsrs.last_clr_time_ms', 0, NULL, NULL, 1, 'N0'), + (830, 'clr_time', 'min', 'min_clr_time_ms', 'qsrs.min_clr_time_ms', 0, NULL, NULL, 1, 'N0'), + (840, 'clr_time', 'max', 'max_clr_time_ms', 'qsrs.max_clr_time_ms', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for CLR time */ + (815, 'clr_time', 'total_hash', 'total_clr_time_ms_by_query_hash', 'qht.total_clr_time_ms', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* DOP (Degree of Parallelism) */ + (900, 'dop', 'last', 'last_dop', 'qsrs.last_dop', 0, NULL, NULL, 1, NULL), + (910, 'dop', 'min', 'min_dop', 'qsrs.min_dop', 0, NULL, NULL, 0, NULL), + (920, 'dop', 'max', 'max_dop', 'qsrs.max_dop', 0, NULL, NULL, 0, NULL), + /* Memory metrics */ + (1000, 'memory', 'avg', 'avg_query_max_used_memory_mb', 'qsrs.avg_query_max_used_memory_mb', 0, NULL, NULL, 0, 'N0'), + (1010, 'memory', 'total', 'total_query_max_used_memory_mb', 'qsrs.total_query_max_used_memory_mb', 0, NULL, NULL, 0, 'N0'), + (1020, 'memory', 'last', 'last_query_max_used_memory_mb', 'qsrs.last_query_max_used_memory_mb', 0, NULL, NULL, 1, 'N0'), + (1030, 'memory', 'min', 'min_query_max_used_memory_mb', 'qsrs.min_query_max_used_memory_mb', 0, NULL, NULL, 1, 'N0'), + (1040, 'memory', 'max', 'max_query_max_used_memory_mb', 'qsrs.max_query_max_used_memory_mb', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for memory */ + (1015, 'memory', 'total_hash', 'total_query_max_used_memory_mb_by_query_hash', 'qht.total_memory_mb', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* Row counts */ + (1100, 'rowcount', 'avg', 'avg_rowcount', 'qsrs.avg_rowcount', 0, NULL, NULL, 0, 'N0'), + (1110, 'rowcount', 'total', 'total_rowcount', 'qsrs.total_rowcount', 0, NULL, NULL, 0, 'N0'), + (1120, 'rowcount', 'last', 'last_rowcount', 'qsrs.last_rowcount', 0, NULL, NULL, 1, 'N0'), + (1130, 'rowcount', 'min', 'min_rowcount', 'qsrs.min_rowcount', 0, NULL, NULL, 1, 'N0'), + (1140, 'rowcount', 'max', 'max_rowcount', 'qsrs.max_rowcount', 0, NULL, NULL, 0, 'N0'), + /* Hash totals for row counts */ + (1115, 'rowcount', 'total_hash', 'total_rowcount_by_query_hash', 'qht.total_rowcount', 1, 'include_query_hash_totals', 1, 0, 'N0'), + /* New metrics for newer versions */ + /* Physical IO Reads (for newer versions) */ + (1200, 'num_physical_io_reads', 'avg', 'avg_num_physical_io_reads_mb', 'qsrs.avg_num_physical_io_reads_mb', 1, 'new', 1, 0, 'N0'), + (1210, 'num_physical_io_reads', 'total', 'total_num_physical_io_reads_mb', 'qsrs.total_num_physical_io_reads_mb', 1, 'new', 1, 0, 'N0'), + (1220, 'num_physical_io_reads', 'last', 'last_num_physical_io_reads_mb', 'qsrs.last_num_physical_io_reads_mb', 1, 'new', 1, 1, 'N0'), + (1230, 'num_physical_io_reads', 'min', 'min_num_physical_io_reads_mb', 'qsrs.min_num_physical_io_reads_mb', 1, 'new', 1, 1, 'N0'), + (1240, 'num_physical_io_reads', 'max', 'max_num_physical_io_reads_mb', 'qsrs.max_num_physical_io_reads_mb', 1, 'new', 1, 0, 'N0'), + /* Hash totals for new physical IO reads */ + (1215, 'num_physical_io_reads', 'total_hash', 'total_num_physical_io_reads_mb_by_query_hash', 'SUM(qsrs.total_num_physical_io_reads_mb) OVER (PARTITION BY qsq.query_hash ORDER BY qsq.query_hash)', 1, 'new_with_hash_totals', 1, 0, 'N0'), + /* Finish adding the remaining columns (log bytes and tempdb usage) */ + /* Log bytes used */ + (1300, 'log_bytes', 'avg', 'avg_log_bytes_used_mb', 'qsrs.avg_log_bytes_used_mb', 1, 'new', 1, 0, 'N0'), + (1310, 'log_bytes', 'total', 'total_log_bytes_used_mb', 'qsrs.total_log_bytes_used_mb', 1, 'new', 1, 0, 'N0'), + (1320, 'log_bytes', 'last', 'last_log_bytes_used_mb', 'qsrs.last_log_bytes_used_mb', 1, 'new', 1, 1, 'N0'), + (1330, 'log_bytes', 'min', 'min_log_bytes_used_mb', 'qsrs.min_log_bytes_used_mb', 1, 'new', 1, 1, 'N0'), + (1340, 'log_bytes', 'max', 'max_log_bytes_used_mb', 'qsrs.max_log_bytes_used_mb', 1, 'new', 1, 0, 'N0'), + /* Hash totals for log bytes */ + (1315, 'log_bytes', 'total_hash', 'total_log_bytes_used_mb_by_query_hash', 'SUM(qsrs.total_log_bytes_used_mb) OVER (PARTITION BY qsq.query_hash ORDER BY qsq.query_hash)', 1, 'new_with_hash_totals', 1, 0, 'N0'), + /* TempDB usage */ + (1400, 'tempdb', 'avg', 'avg_tempdb_space_used_mb', 'qsrs.avg_tempdb_space_used_mb', 1, 'new', 1, 0, 'N0'), + (1410, 'tempdb', 'total', 'total_tempdb_space_used_mb', 'qsrs.total_tempdb_space_used_mb', 1, 'new', 1, 0, 'N0'), + (1420, 'tempdb', 'last', 'last_tempdb_space_used_mb', 'qsrs.last_tempdb_space_used_mb', 1, 'new', 1, 1, 'N0'), + (1430, 'tempdb', 'min', 'min_tempdb_space_used_mb', 'qsrs.min_tempdb_space_used_mb', 1, 'new', 1, 1, 'N0'), + (1440, 'tempdb', 'max', 'max_tempdb_space_used_mb', 'qsrs.max_tempdb_space_used_mb', 1, 'new', 1, 0, 'N0'), + /* Hash totals for tempdb */ + (1415, 'tempdb', 'total_hash', 'total_tempdb_space_used_mb_by_query_hash', 'SUM(qsrs.total_tempdb_space_used_mb) OVER (PARTITION BY qsq.query_hash ORDER BY qsq.query_hash)', 1, 'new_with_hash_totals', 1, 0, 'N0'), + /* Context settings and sorting columns */ + (1500, 'metadata', 'context', 'context_settings', 'qsrs.context_settings', 0, NULL, NULL, 0, NULL); + +/* Add special sorting columns based on @sort_order */ +/* Plan hash count for 'plan count by hashes' sort */ +IF @sort_order = 'plan count by hashes' +BEGIN + INSERT INTO + @ColumnDefinitions (column_id, metric_group, metric_type, column_name, column_source, is_conditional, condition_param, condition_value, expert_only, format_pattern) + VALUES + (1600, 'sort_order', 'plan_hash_count', 'plan_hash_count_for_query_hash', 'hashes.plan_hash_count_for_query_hash', 0, NULL, NULL, 0, 'N0'), + (1610, 'sort_order', 'query_hash', 'query_hash_from_hash_counting', 'hashes.query_hash', 0, NULL, NULL, 0, NULL); +END; + +/* Dynamic regression change column based on formatting and comparator */ +IF @regression_baseline_start_date IS NOT NULL AND @regression_comparator = 'relative' AND @format_output = 1 +BEGIN + INSERT INTO + @ColumnDefinitions (column_id, metric_group, metric_type, column_name, column_source, is_conditional, condition_param, condition_value, expert_only, format_pattern) + VALUES (160, 'regression', 'change', 'change_in_average_for_query_hash_since_regression_time_period', 'regression.change_since_regression_time_period', 1, 'regression_mode', 1, 0, 'P2'); +END; +ELSE IF @regression_baseline_start_date IS NOT NULL AND @format_output = 1 +BEGIN + INSERT INTO + @ColumnDefinitions (column_id, metric_group, metric_type, column_name, column_source, is_conditional, condition_param, condition_value, expert_only, format_pattern) + VALUES (160, 'regression', 'change', 'change_in_average_for_query_hash_since_regression_time_period', 'regression.change_since_regression_time_period', 1, 'regression_mode', 1, 0, 'N2'); +END; +ELSE IF @regression_baseline_start_date IS NOT NULL +BEGIN + INSERT INTO + @ColumnDefinitions (column_id, metric_group, metric_type, column_name, column_source, is_conditional, condition_param, condition_value, expert_only, format_pattern) + VALUES (160, 'regression', 'change', 'change_in_average_for_query_hash_since_regression_time_period', 'regression.change_since_regression_time_period', 1, 'regression_mode', 1, 0, NULL); +END; + +/* Wait time for wait-based sorting */ +IF LOWER(@sort_order) LIKE N'%waits' +BEGIN + INSERT INTO + @ColumnDefinitions (column_id, metric_group, metric_type, column_name, column_source, is_conditional, condition_param, condition_value, expert_only, format_pattern) + VALUES + (1620, 'sort_order', 'wait_time', 'total_wait_time_from_sort_order_ms', 'waits.total_query_wait_time_ms', 0, NULL, NULL, 0, 'N0'); +END; + +/* ROW_NUMBER window function for sorting */ +INSERT INTO + @ColumnDefinitions (column_id, metric_group, metric_type, column_name, column_source, is_conditional, condition_param, condition_value, expert_only, format_pattern) +VALUES + ( + 2000, + 'metadata', + 'n', + 'n', + 'ROW_NUMBER() OVER (PARTITION BY qsrs.plan_id ORDER BY ' + + CASE WHEN @regression_baseline_start_date IS NOT NULL THEN + /* As seen when populating #regression_changes */ + CASE @regression_direction + WHEN 'regressed' THEN 'regression.change_since_regression_time_period' + WHEN 'worse' THEN 'regression.change_since_regression_time_period' + WHEN 'improved' THEN 'regression.change_since_regression_time_period * -1.0' + WHEN 'better' THEN 'regression.change_since_regression_time_period * -1.0' + WHEN 'magnitude' THEN 'ABS(regression.change_since_regression_time_period)' + WHEN 'absolute' THEN 'ABS(regression.change_since_regression_time_period)' + END + ELSE + CASE @sort_order + WHEN 'cpu' THEN 'qsrs.avg_cpu_time_ms' + WHEN 'logical reads' THEN 'qsrs.avg_logical_io_reads_mb' + WHEN 'physical reads' THEN 'qsrs.avg_physical_io_reads_mb' + WHEN 'writes' THEN 'qsrs.avg_logical_io_writes_mb' + WHEN 'duration' THEN 'qsrs.avg_duration_ms' + WHEN 'memory' THEN 'qsrs.avg_query_max_used_memory_mb' + WHEN 'tempdb' THEN 'qsrs.avg_tempdb_space_used_mb' /*This gets validated later*/ + WHEN 'executions' THEN 'qsrs.count_executions' + WHEN 'recent' THEN 'qsrs.last_execution_time' + WHEN 'rows' THEN 'qsrs.avg_rowcount' + WHEN 'plan count by hashes' THEN 'hashes.plan_hash_count_for_query_hash DESC, hashes.query_hash' + ELSE CASE WHEN LOWER(@sort_order) LIKE N'%waits' THEN 'waits.total_query_wait_time_ms' + ELSE 'qsrs.avg_cpu_time' END + END + END + ' DESC)', + 0, + NULL, + NULL, + 0, + NULL + ); + +/* Create a table variable to define parameter processing */ +DECLARE + @FilterParameters table +( + parameter_name nvarchar(100) NOT NULL, + parameter_value nvarchar(4000) NOT NULL, + temp_table_name sysname NOT NULL, + column_name sysname NOT NULL, + data_type sysname NOT NULL, + is_include bit NOT NULL, + requires_secondary_processing bit NOT NULL +); + +/* Populate with parameter definitions*/ +INSERT INTO + @FilterParameters +( + parameter_name, + parameter_value, + temp_table_name, + column_name, + data_type, + is_include, + requires_secondary_processing +) +SELECT + v.parameter_name, + v.parameter_value, + v.temp_table_name, + v.column_name, + v.data_type, + v.is_include, + v.requires_secondary_processing +FROM +( + VALUES + /* Include parameters */ + ('include_plan_ids', @include_plan_ids, '#include_plan_ids', 'plan_id', 'bigint', 1, 0), + ('include_query_ids', @include_query_ids, '#include_query_ids', 'query_id', 'bigint', 1, 1), + ('include_query_hashes', @include_query_hashes, '#include_query_hashes', 'query_hash_s', 'varchar', 1, 1), + ('include_plan_hashes', @include_plan_hashes, '#include_plan_hashes', 'plan_hash_s', 'varchar', 1, 1), + ('include_sql_handles', @include_sql_handles, '#include_sql_handles', 'sql_handle_s', 'varchar', 1, 1), + /* Ignore parameters */ + ('ignore_plan_ids', @ignore_plan_ids, '#ignore_plan_ids', 'plan_id', 'bigint', 0, 0), + ('ignore_query_ids', @ignore_query_ids, '#ignore_query_ids', 'query_id', 'bigint', 0, 1), + ('ignore_query_hashes', @ignore_query_hashes, '#ignore_query_hashes', 'query_hash_s', 'varchar', 0, 1), + ('ignore_plan_hashes', @ignore_plan_hashes, '#ignore_plan_hashes', 'plan_hash_s', 'varchar', 0, 1), + ('ignore_sql_handles', @ignore_sql_handles, '#ignore_sql_handles', 'sql_handle_s', 'varchar', 0, 1) + ) AS v + ( + parameter_name, + parameter_value, + temp_table_name, + column_name, + data_type, + is_include, + requires_secondary_processing + ) +WHERE v.parameter_value IS NOT NULL; + /* Try to be helpful by subbing in a database name if null */ @@ -1332,7 +1671,18 @@ DECLARE @regression_baseline_start_date_original datetimeoffset(7), @regression_baseline_end_date_original datetimeoffset(7), @regression_mode bit, - @regression_where_clause nvarchar(max); + @regression_where_clause nvarchar(MAX), + @column_sql nvarchar(MAX), + @param_name nvarchar(100), + @param_value nvarchar(4000), + @temp_table sysname, + @column_name sysname, + @data_type sysname, + @is_include bit, + @requires_secondary_processing bit, + @split_sql nvarchar(MAX), + @error_msg nvarchar(2000), + @conflict_list nvarchar(max) = N''; /* In cases where we are escaping @query_text_search and @@ -1593,6 +1943,126 @@ are assigned for the specific database that is currently being looked at */ +/* +Look at databases to include or exclude +*/ +IF @get_all_databases = 1 +BEGIN + /* Check for contradictory parameters */ + IF @database_name IS NOT NULL + BEGIN + IF @debug = 1 + BEGIN + RAISERROR(N'@database name being ignored since @get_all_databases is set to 1', 0, 0) WITH NOWAIT; + END; + SET @database_name = NULL; + END; + + /* Parse @include_databases if specified using XML for compatibility */ + IF @include_databases IS NOT NULL + BEGIN + INSERT + #include_databases + WITH + (TABLOCK) + ( + database_name + ) + SELECT DISTINCT + database_name = + LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) + FROM + ( + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE + ( + @include_databases, + N',', + N'' + ) + + N'' + ) + ) AS a + CROSS APPLY x.nodes(N'//i') AS t(c) + WHERE LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) <> N'' + OPTION(RECOMPILE); + END; + + /* Parse @exclude_databases if specified using XML for compatibility */ + IF @exclude_databases IS NOT NULL + BEGIN + INSERT + #exclude_databases + WITH + (TABLOCK) + ( + database_name + ) + SELECT DISTINCT + database_name = + LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) + FROM + ( + SELECT + x = CONVERT + ( + xml, + N'' + + REPLACE + ( + @exclude_databases, + N',', + N'' + ) + + N'' + ) + ) AS a + CROSS APPLY x.nodes(N'//i') AS t(c) + WHERE LTRIM(RTRIM(c.value(N'(./text())[1]', N'sysname'))) <> N'' + OPTION(RECOMPILE); + + /* Check for databases in both include and exclude lists */ + IF @include_databases IS NOT NULL + BEGIN + /* Build list of conflicting databases */ + SELECT + @conflict_list = + @conflict_list + + ed.database_name + N', ' + FROM #exclude_databases AS ed + WHERE EXISTS + ( + SELECT + 1/0 + FROM #include_databases AS id + WHERE id.database_name = ed.database_name + ) + OPTION(RECOMPILE); + + /* If we found any conflicts, raise an error */ + IF LEN(@conflict_list) > 0 + BEGIN + /* Remove trailing comma and space */ + SET @conflict_list = LEFT(@conflict_list, LEN(@conflict_list) - 2); + + SET @error_msg = + N'The following databases appear in both @include_databases and @exclude_databases, which creates ambiguity: ' + + @conflict_list + N'. Please remove these databases from one of the lists.'; + + RAISERROR(@error_msg, 16, 1); + RETURN; + END; + END; + END; +END; + +/* +Build up the databases to process +*/ IF ( SELECT @@ -1604,7 +2074,9 @@ SELECT ) IN (5, 8) BEGIN INSERT INTO - #databases WITH(TABLOCK) + #databases + WITH + (TABLOCK) ( database_name ) @@ -1625,12 +2097,65 @@ BEGIN AND d.state = 0 AND d.is_in_standby = 0 AND d.is_read_only = 0 + AND ( + @include_databases IS NULL + OR EXISTS (SELECT 1/0 FROM #include_databases AS id WHERE id.database_name = d.name) + ) + AND ( + @exclude_databases IS NULL + OR NOT EXISTS (SELECT 1/0 FROM #exclude_databases AS ed WHERE ed.database_name = d.name) + ) OPTION(RECOMPILE); -END + + /* Track which requested databases were skipped */ + IF @include_databases IS NOT NULL + AND @get_all_databases = 1 + BEGIN + INSERT + #requested_but_skipped_databases + WITH + (TABLOCK) + ( + database_name, + reason + ) + SELECT + id.database_name, + reason = + CASE + WHEN d.name IS NULL + THEN 'Database does not exist' + WHEN d.state <> 0 + THEN 'Database not online' + WHEN d.is_query_store_on = 0 + THEN 'Query Store not enabled' + WHEN d.is_in_standby = 1 + THEN 'Database is in standby' + WHEN d.is_read_only = 1 + THEN 'Database is read-only' + WHEN d.database_id <= 4 + THEN 'System database' + ELSE 'Other issue' + END + FROM #include_databases AS id + LEFT JOIN sys.databases AS d + ON id.database_name = d.name + WHERE NOT EXISTS + ( + SELECT + 1/0 + FROM #databases AS db + WHERE db.database_name = id.database_name + ) + OPTION(RECOMPILE); + END; +END; ELSE BEGIN INSERT - #databases WITH(TABLOCK) + #databases + WITH + (TABLOCK) ( database_name ) @@ -1647,7 +2172,7 @@ BEGIN FROM sys.databases AS d WHERE @get_all_databases = 1 AND d.is_query_store_on = 1 - AND d.database_id > 4 + AND d.name NOT IN (N'master', N'model', N'msdb', N'tempdb', N'rdsadmin') AND d.state = 0 AND d.is_in_standby = 0 AND d.is_read_only = 0 @@ -1663,18 +2188,76 @@ BEGIN AND s.role_desc <> N'PRIMARY' AND DATABASEPROPERTYEX(c.database_name, N'Updateability') <> N'READ_WRITE' ) + AND ( + @include_databases IS NULL + OR EXISTS (SELECT 1/0 FROM #include_databases AS id WHERE id.database_name = d.name) + ) + AND ( + @exclude_databases IS NULL + OR NOT EXISTS (SELECT 1/0 FROM #exclude_databases AS ed WHERE ed.database_name = d.name) + ) OPTION(RECOMPILE); + + /* Track which requested databases were skipped */ + IF @include_databases IS NOT NULL + AND @get_all_databases = 1 + BEGIN + INSERT + #requested_but_skipped_databases + WITH + (TABLOCK) + ( + database_name, + reason + ) + SELECT + id.database_name, + reason = + CASE + WHEN d.name IS NULL THEN 'Database does not exist' + WHEN d.state <> 0 THEN 'Database not online' + WHEN d.is_query_store_on = 0 THEN 'Query Store not enabled' + WHEN d.is_in_standby = 1 THEN 'Database is in standby' + WHEN d.is_read_only = 1 THEN 'Database is read-only' + WHEN d.database_id <= 4 THEN 'System database' + WHEN EXISTS + ( + SELECT + 1/0 + FROM sys.dm_hadr_availability_replica_states AS s + JOIN sys.availability_databases_cluster AS c + ON s.group_id = c.group_id + AND d.name = c.database_name + WHERE s.is_local <> 1 + AND s.role_desc <> N'PRIMARY' + AND DATABASEPROPERTYEX(c.database_name, N'Updateability') <> N'READ_WRITE' + ) THEN 'AG replica issues' + ELSE 'Other issue' + END + FROM #include_databases AS id + LEFT JOIN sys.databases AS d + ON id.database_name = d.name + WHERE NOT EXISTS + ( + SELECT + 1/0 + FROM #databases AS db + WHERE db.database_name = id.database_name + ) + OPTION(RECOMPILE); + END; END; DECLARE @database_cursor CURSOR; SET - @database_cursor = CURSOR - LOCAL - SCROLL - DYNAMIC - READ_ONLY + @database_cursor = + CURSOR + LOCAL + SCROLL + DYNAMIC + READ_ONLY FOR SELECT d.database_name @@ -1688,12 +2271,113 @@ INTO @database_name; WHILE @@FETCH_STATUS = 0 BEGIN +/* +These tables need to get cleared out +to avoid result pollution and +primary key violations +*/ +IF @debug = 1 +BEGIN + RAISERROR('Truncating per-database temp tables for the next iteration', 0, 0) WITH NOWAIT; +END; + +TRUNCATE TABLE + #regression_baseline_runtime_stats; + +TRUNCATE TABLE + #regression_current_runtime_stats; + +TRUNCATE TABLE + #distinct_plans; + +TRUNCATE TABLE + #procedure_plans; + +TRUNCATE TABLE + #procedure_object_ids; + +TRUNCATE TABLE + #maintenance_plans; + +TRUNCATE TABLE + #query_text_search; + +TRUNCATE TABLE + #query_text_search_not; + +TRUNCATE TABLE + #dm_exec_query_stats; + +TRUNCATE TABLE + #query_types; + +TRUNCATE TABLE + #wait_filter; + +TRUNCATE TABLE + #only_queries_with_hints; + +TRUNCATE TABLE + #only_queries_with_feedback; + +TRUNCATE TABLE + #only_queries_with_variants; + +TRUNCATE TABLE + #forced_plans_failures; + +TRUNCATE TABLE + #include_plan_ids; + +TRUNCATE TABLE + #include_query_ids; + +TRUNCATE TABLE + #include_query_hashes; + +TRUNCATE TABLE + #include_plan_hashes; + +TRUNCATE TABLE + #include_sql_handles; + +TRUNCATE TABLE + #ignore_plan_ids; + +TRUNCATE TABLE + #ignore_query_ids; + +TRUNCATE TABLE + #ignore_query_hashes; + +TRUNCATE TABLE + #ignore_plan_hashes; + +TRUNCATE TABLE + #ignore_sql_handles; + +TRUNCATE TABLE + #only_queries_with_hints; + +TRUNCATE TABLE + #only_queries_with_feedback; + +TRUNCATE TABLE + #only_queries_with_variants; + +TRUNCATE TABLE + #forced_plans_failures; + +TRUNCATE TABLE + #query_hash_totals; + + /* Some variable assignment, because why not? */ IF @debug = 1 BEGIN - RAISERROR('Starting analysis for database %s', 0, 1, @database_name) WITH NOWAIT; + RAISERROR('Starting analysis for database %s', 0, 0, @database_name) WITH NOWAIT; END; SELECT @@ -1890,7 +2574,9 @@ SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;', OPTION(RECOMPILE);', @troubleshoot_insert = N' INSERT - #troubleshoot_performance WITH(TABLOCK) + #troubleshoot_performance + WITH + (TABLOCK) ( current_table, start_time @@ -2109,19 +2795,19 @@ We set both _date_original variables earlier. */ SELECT @regression_baseline_start_date = - DATEADD - ( - MINUTE, - @utc_minutes_difference, - @regression_baseline_start_date_original - ), + DATEADD + ( + MINUTE, + @utc_minutes_difference, + @regression_baseline_start_date_original + ), @regression_baseline_end_date = - DATEADD - ( - MINUTE, - @utc_minutes_difference, - @regression_baseline_end_date_original - ), + DATEADD + ( + MINUTE, + @utc_minutes_difference, + @regression_baseline_end_date_original + ), @regression_comparator = ISNULL(@regression_comparator, 'absolute'), @regression_direction = @@ -2147,7 +2833,7 @@ BEGIN BEGIN IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2168,7 +2854,7 @@ BEGIN RAISERROR('Not all Azure offerings are supported, please try avoiding memes', 11, 1) WITH NOWAIT; IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2195,7 +2881,7 @@ BEGIN RAISERROR('Azure databases in compatibility levels under 130 are not supported', 11, 1) WITH NOWAIT; IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2284,7 +2970,7 @@ BEGIN BEGIN IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2364,7 +3050,9 @@ BEGIN END; INSERT - #query_store_trouble WITH (TABLOCK) + #query_store_trouble +WITH + (TABLOCK) ( database_id, desired_state_desc, @@ -2416,7 +3104,7 @@ BEGIN IF @procedure_schema IS NULL BEGIN SELECT - @procedure_schema = N'dbo' + @procedure_schema = N'dbo'; END; SELECT @current_table = 'checking procedure existence', @@ -2445,7 +3133,8 @@ FROM ' + @database_name_quoted + N'.sys.procedures AS p JOIN ' + @database_name_quoted + N'.sys.schemas AS s ON p.schema_id = s.schema_id WHERE s.name = @procedure_schema -AND p.name LIKE @procedure_name;' + @nc10; +AND p.name LIKE @procedure_name +OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN @@ -2454,7 +3143,9 @@ AND p.name LIKE @procedure_name;' + @nc10; END; INSERT - #procedure_object_ids WITH(TABLOCK) + #procedure_object_ids + WITH + (TABLOCK) ( [object_id] ) @@ -2657,7 +3348,7 @@ Check that you spelled everything correctly and you''re in the right database', BEGIN IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2727,7 +3418,7 @@ END; Checks if the sort order is for a wait. Cuts out a lot of repetition. */ -IF @sort_order IN +IF LOWER(@sort_order) IN ( 'cpu waits', 'lock waits', @@ -2747,7 +3438,6 @@ IF @sort_order IN 'total waits' ) BEGIN - SELECT @sort_order_is_a_wait = 1; END; @@ -2811,7 +3501,7 @@ BEGIN BEGIN IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2835,7 +3525,7 @@ BEGIN BEGIN IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2876,7 +3566,7 @@ BEGIN BEGIN IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -2959,7 +3649,7 @@ OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN RAISERROR('Query Store wait stats are not enabled for database %s', 10, 1, @database_name_quoted) WITH NOWAIT; - END + END; END; END; /*End wait stats checks*/ @@ -2977,7 +3667,7 @@ BEGIN RAISERROR('The time zone you chose (%s) is not valid. Please check sys.time_zone_info for a valid list.', 10, 1, @timezone) WITH NOWAIT; IF @debug = 1 BEGIN - GOTO DEBUG + GOTO DEBUG; END; ELSE BEGIN @@ -3019,7 +3709,7 @@ BEGIN ELSE 0 END OPTION(RECOMPILE); - END + END; END; /* @@ -3177,12 +3867,12 @@ SELECT DISTINCT FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp ON qsq.query_id = qsp.query_id -WHERE ' +WHERE '; IF CHARINDEX(N'%', @procedure_name) = 0 BEGIN SELECT - @sql += N'qsq.object_id = OBJECT_ID(@procedure_name_quoted)' + @sql += N'qsq.object_id = OBJECT_ID(@procedure_name_quoted)'; END; IF CHARINDEX(N'%', @procedure_name) > 0 @@ -3194,7 +3884,7 @@ BEGIN 1/0 FROM #procedure_object_ids AS poi WHERE poi.[object_id] = qsq.[object_id] -)' +)'; END; SELECT @@ -3208,7 +3898,9 @@ OPTION(RECOMPILE);' + @nc10; END; INSERT - #procedure_plans WITH(TABLOCK) + #procedure_plans + WITH + (TABLOCK) ( plan_id ) @@ -3288,7 +3980,9 @@ OPTION(RECOMPILE);' + @nc10; END; INSERT - #query_types WITH(TABLOCK) + #query_types + WITH + (TABLOCK) ( plan_id ) @@ -3332,97 +4026,337 @@ IF OR @include_query_ids IS NOT NULL OR @ignore_plan_ids IS NOT NULL OR @ignore_query_ids IS NOT NULL +OR @include_query_hashes IS NOT NULL +OR @include_plan_hashes IS NOT NULL +OR @include_sql_handles IS NOT NULL +OR @ignore_query_hashes IS NOT NULL +OR @ignore_plan_hashes IS NOT NULL +OR @ignore_sql_handles IS NOT NULL ) BEGIN - IF @include_plan_ids IS NOT NULL - BEGIN - SELECT - @include_plan_ids = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@include_plan_ids)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); - - SELECT - @current_table = 'inserting #include_plan_ids'; - - INSERT - #include_plan_ids WITH(TABLOCK) - ( - plan_id - ) - EXECUTE sys.sp_executesql - @string_split_ints, - N'@ids nvarchar(4000)', - @include_plan_ids; - - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #include_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; /*End include plan ids*/ - - IF @ignore_plan_ids IS NOT NULL - BEGIN - SELECT - @ignore_plan_ids = + DECLARE + @filter_cursor CURSOR; + + SET @filter_cursor = + CURSOR + LOCAL + FORWARD_ONLY + STATIC + READ_ONLY + FOR + SELECT + parameter_name, + parameter_value, + temp_table_name, + column_name, + data_type, + is_include, + requires_secondary_processing + FROM @FilterParameters AS fp; + + OPEN @filter_cursor; + + FETCH NEXT + FROM @filter_cursor + INTO + @param_name, + @param_value, + @temp_table, + @column_name, + @data_type, + @is_include, + @requires_secondary_processing; + + WHILE @@FETCH_STATUS = 0 + BEGIN + /* Clean parameter value */ + SELECT + @param_value = REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@ignore_plan_ids)), - CHAR(10), N''), CHAR(13), N''), + LTRIM(RTRIM(@param_value)), + CHAR(10), N''), CHAR(13), N''), NCHAR(10), N''), NCHAR(13), N''); - - SELECT - @current_table = 'inserting #ignore_plan_ids'; - - INSERT - #ignore_plan_ids WITH(TABLOCK) + + /* Log current operation if debugging */ + IF @debug = 1 + BEGIN + RAISERROR('Processing %s with value %s', 0, 0, @param_name, @param_value) WITH NOWAIT; + END; + + /* Set current table name for troubleshooting */ + SELECT + @current_table = 'inserting ' + @temp_table; + + /* Choose appropriate string split function based on data type */ + IF @data_type = N'bigint' + BEGIN + SELECT @split_sql = @string_split_ints; + END + ELSE + BEGIN + SELECT @split_sql = @string_split_strings; + END; + + /* Execute the initial insert with troubleshooting if enabled */ + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; + + SET STATISTICS XML ON; + END; + + /* Execute the dynamic SQL to populate the temporary table */ + DECLARE @dynamic_sql nvarchar(MAX) = N' + INSERT INTO + ' + @temp_table + N' + WITH + (TABLOCK) ( - plan_id - ) + ' + @column_name + + N') EXECUTE sys.sp_executesql - @string_split_ints, - N'@ids nvarchar(4000)', - @ignore_plan_ids; - - SELECT - @where_clause += N'AND NOT EXISTS - ( - SELECT - 1/0 - FROM #ignore_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; /*End ignore plan ids*/ + @split_sql, + N''@ids nvarchar(4000)'', + @param_value;'; + + EXEC sys.sp_executesql + @dynamic_sql, + N'@split_sql nvarchar(max), + @param_value nvarchar(4000)', + @split_sql, + @param_value; + + IF @troubleshoot_performance = 1 + BEGIN + SET STATISTICS XML OFF; + + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; + + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @split_sql, + @current_table; + END; + + /* Secondary processing (for parameters that need to populate plan IDs) */ + IF @requires_secondary_processing = 1 + BEGIN + SELECT + @current_table = 'inserting #include_plan_ids for ' + @param_name; + + /* Build appropriate SQL based on parameter type */ + DECLARE + @secondary_sql nvarchar(MAX) = N''; + + IF @param_name = 'include_query_ids' + OR @param_name = 'ignore_query_ids' + BEGIN + SELECT @secondary_sql = N' + SELECT DISTINCT + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE EXISTS + ( + SELECT + 1/0 + FROM #' + + CASE + WHEN @is_include = 1 + THEN N'include' + ELSE N'ignore' + END + + N'_query_ids AS iqi + WHERE iqi.query_id = qsp.query_id + ) + OPTION(RECOMPILE);'; + END; + ELSE + IF @param_name = 'include_query_hashes' + OR @param_name = 'ignore_query_hashes' + BEGIN + SELECT @secondary_sql = N' + SELECT DISTINCT + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + WHERE qsq.query_id = qsp.query_id + AND EXISTS + ( + SELECT + 1/0 + FROM #' + + CASE + WHEN @is_include = 1 + THEN N'include' + ELSE N'ignore' + END + + N'_query_hashes AS iqh + WHERE iqh.query_hash = qsq.query_hash + ) + ) + OPTION(RECOMPILE);'; + END; + ELSE + IF @param_name = 'include_plan_hashes' + OR @param_name = 'ignore_plan_hashes' + BEGIN + SELECT @secondary_sql = N' + SELECT DISTINCT + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE EXISTS + ( + SELECT + 1/0 + FROM #' + + CASE + WHEN @is_include = 1 + THEN N'include' + ELSE N'ignore' + END + N'_plan_hashes AS iph + WHERE iph.plan_hash = qsp.query_plan_hash + ) + OPTION(RECOMPILE);'; + END; + ELSE + IF + @param_name = 'include_sql_handles' + OR @param_name = 'ignore_sql_handles' + BEGIN + SELECT @secondary_sql = N' + SELECT DISTINCT + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + WHERE qsp.query_id = qsq.query_id + AND EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt + WHERE qsqt.query_text_id = qsq.query_text_id + AND EXISTS + ( + SELECT + 1/0 + FROM #' + + CASE + WHEN @is_include = 1 + THEN N'include' + ELSE N'ignore' + END + N'_sql_handles AS ish + WHERE ish.sql_handle = qsqt.statement_sql_handle + ) + ) + ) + OPTION(RECOMPILE);'; + END; + + /* Process secondary sql if defined */ + IF @secondary_sql IS NOT NULL + BEGIN + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; + + SET STATISTICS XML ON; + END; + + INSERT INTO + #include_plan_ids + WITH + (TABLOCK) + ( + plan_id + ) + EXECUTE sys.sp_executesql + @secondary_sql; + + IF @troubleshoot_performance = 1 + BEGIN + SET STATISTICS XML OFF; + + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; + + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), @current_table nvarchar(100)', + @secondary_sql, + @current_table; + END; + END; + + /* Update where clause if needed */ + DECLARE + @temp_target_table nvarchar(100) = + CASE + WHEN @is_include = 1 + THEN N'#include_plan_ids' + ELSE N'#ignore_plan_ids' + END, + @exist_or_not_exist nvarchar(20) = + CASE + WHEN @is_include = 1 + THEN N'EXISTS' + ELSE N'NOT EXISTS' + END; + + SELECT + @where_clause += + N'AND ' + + @exist_or_not_exist + + N' + ( + SELECT + 1/0 + FROM ' + @temp_target_table + N' AS idi + WHERE idi.plan_id = qsrs.plan_id + )' + @nc10; + END; + + FETCH NEXT + FROM @filter_cursor + INTO + @param_name, + @param_value, + @temp_table, + @column_name, + @data_type, + @is_include, + @requires_secondary_processing; + END; +END; /*End hash and handle filtering*/ - IF @include_query_ids IS NOT NULL +IF @sql_2022_views = 1 +BEGIN + IF @only_queries_with_hints = 1 BEGIN SELECT - @include_query_ids = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@include_query_ids)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); - SELECT - @current_table = 'inserting #include_query_ids', + @current_table = 'inserting #only_queries_with_hints', @sql = @isolation_level; - INSERT - #include_query_ids WITH(TABLOCK) - ( - query_id - ) - EXECUTE sys.sp_executesql - @string_split_ints, - N'@ids nvarchar(4000)', - @include_query_ids; - - SELECT - @current_table = 'inserting #include_plan_ids for included query ids'; - IF @troubleshoot_performance = 1 BEGIN EXECUTE sys.sp_executesql @@ -3435,17 +4369,20 @@ BEGIN SELECT @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS - ( - SELECT - 1/0 - FROM #include_query_ids AS iqi - WHERE iqi.query_id = qsp.query_id - ) -OPTION(RECOMPILE);' + @nc10; + SELECT DISTINCT + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_query_hints AS qsqh + WHERE qsqh.query_id = qsp.query_id + )'; + + SELECT + @sql += N' + OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN @@ -3454,7 +4391,9 @@ OPTION(RECOMPILE);' + @nc10; END; INSERT - #include_plan_ids + #only_queries_with_hints + WITH + (TABLOCK) ( plan_id ) @@ -3478,54 +4417,22 @@ OPTION(RECOMPILE);' + @nc10; @current_table; END; - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: - - If @include_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. - - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @include_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #include_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End include query ids*/ + SELECT + @where_clause += N'AND EXISTS + ( + SELECT + 1/0 + FROM #only_queries_with_hints AS qst + WHERE qst.plan_id = qsrs.plan_id + )' + @nc10; + END; - IF @ignore_query_ids IS NOT NULL + IF @only_queries_with_feedback = 1 BEGIN SELECT - @ignore_query_ids = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@ignore_query_ids)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); - SELECT - @current_table = 'inserting #ignore_query_ids', + @current_table = 'inserting #only_queries_with_feedback', @sql = @isolation_level; - INSERT - #ignore_query_ids WITH(TABLOCK) - ( - query_id - ) - EXECUTE sys.sp_executesql - @string_split_ints, - N'@ids nvarchar(4000)', - @ignore_query_ids; - - SELECT - @current_table = 'inserting #ignore_plan_ids for ignored query ids'; - IF @troubleshoot_performance = 1 BEGIN EXECUTE sys.sp_executesql @@ -3538,17 +4445,20 @@ OPTION(RECOMPILE);' + @nc10; SELECT @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS - ( - SELECT - 1/0 - FROM #ignore_query_ids AS iqi - WHERE iqi.query_id = qsp.query_id - ) -OPTION(RECOMPILE);' + @nc10; + SELECT DISTINCT + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_plan_feedback AS qsqf + WHERE qsqf.plan_id = qsp.plan_id + )'; + + SELECT + @sql += N' + OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN @@ -3557,7 +4467,9 @@ OPTION(RECOMPILE);' + @nc10; END; INSERT - #ignore_plan_ids + #only_queries_with_feedback + WITH + (TABLOCK) ( plan_id ) @@ -3581,69 +4493,22 @@ OPTION(RECOMPILE);' + @nc10; @current_table; END; - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: - - If @ignore_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. - - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @ignore_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND NOT EXISTS - ( - SELECT - 1/0 - FROM #ignore_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End ignore query ids*/ -END; /*End query and plan id filtering*/ - -/* -This section filters query or plan hashes -*/ -IF -( - @include_query_hashes IS NOT NULL -OR @include_plan_hashes IS NOT NULL -OR @include_sql_handles IS NOT NULL -OR @ignore_query_hashes IS NOT NULL -OR @ignore_plan_hashes IS NOT NULL -OR @ignore_sql_handles IS NOT NULL -) -BEGIN - IF @include_query_hashes IS NOT NULL - BEGIN SELECT - @include_query_hashes = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@include_query_hashes)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); + @where_clause += N'AND EXISTS + ( + SELECT + 1/0 + FROM #only_queries_with_feedback AS qst + WHERE qst.plan_id = qsrs.plan_id + )' + @nc10; + END; + IF @only_queries_with_variants = 1 + BEGIN SELECT - @current_table = 'inserting #include_query_hashes', + @current_table = 'inserting #only_queries_with_variants', @sql = @isolation_level; - INSERT - #include_query_hashes WITH(TABLOCK) - ( - query_hash_s - ) - EXECUTE sys.sp_executesql - @string_split_strings, - N'@ids nvarchar(4000)', - @include_query_hashes; - - SELECT - @current_table = 'inserting #include_plan_ids for included query hashes'; - IF @troubleshoot_performance = 1 BEGIN EXECUTE sys.sp_executesql @@ -3656,24 +4521,20 @@ BEGIN SELECT @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - WHERE qsq.query_id = qsp.query_id - AND EXISTS - ( - SELECT - 1/0 - FROM #include_query_hashes AS iqh - WHERE iqh.query_hash = qsq.query_hash - ) - ) -OPTION(RECOMPILE);' + @nc10; + SELECT DISTINCT + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_query_variant AS qsqv + WHERE qsqv.query_variant_query_id = qsp.query_id + )'; + + SELECT + @sql += N' + OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN @@ -3682,7 +4543,9 @@ OPTION(RECOMPILE);' + @nc10; END; INSERT - #include_plan_ids + #only_queries_with_variants + WITH + (TABLOCK) ( plan_id ) @@ -3706,178 +4569,159 @@ OPTION(RECOMPILE);' + @nc10; @current_table; END; - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: - - If @include_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. - - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @include_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #include_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End include query hashes*/ - - IF @ignore_query_hashes IS NOT NULL - BEGIN SELECT - @ignore_query_hashes = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@ignore_query_hashes)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); + @where_clause += N'AND EXISTS + ( + SELECT + 1/0 + FROM #only_queries_with_variants AS qst + WHERE qst.plan_id = qsrs.plan_id + )' + @nc10; + END; +END; - SELECT - @current_table = 'inserting #ignore_query_hashes', - @sql = @isolation_level; +IF +( + @only_queries_with_forced_plans = 1 + OR @only_queries_with_forced_plan_failures = 1 +) +BEGIN + SELECT + @current_table = 'inserting #forced_plans_failures', + @sql = @isolation_level; - INSERT - #ignore_query_hashes WITH(TABLOCK) - ( - query_hash_s - ) + IF @troubleshoot_performance = 1 + BEGIN EXECUTE sys.sp_executesql - @string_split_strings, - N'@ids nvarchar(4000)', - @ignore_query_hashes; + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - SELECT - @current_table = 'inserting #ignore_plan_ids for ignored query hashes'; + SET STATISTICS XML ON; + END; - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' + SELECT + @sql += N' SELECT DISTINCT qsp.plan_id FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - WHERE qsq.query_id = qsp.query_id - AND EXISTS - ( - SELECT - 1/0 - FROM #ignore_query_hashes AS iqh - WHERE iqh.query_hash = qsq.query_hash - ) - ) -OPTION(RECOMPILE);' + @nc10; +WHERE qsp.is_forced_plan = 1'; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; +IF @only_queries_with_forced_plan_failures = 1 +BEGIN + SELECT + @sql += N' +AND qsp.last_force_failure_reason > 0'; +END; - INSERT - #ignore_plan_ids - ( - plan_id - ) - EXECUTE sys.sp_executesql - @sql; + SELECT + @sql += N' +OPTION(RECOMPILE);' + @nc10; - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; + INSERT + #forced_plans_failures + WITH + (TABLOCK) + ( + plan_id + ) + EXECUTE sys.sp_executesql + @sql; - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; + IF @troubleshoot_performance = 1 + BEGIN + SET STATISTICS XML OFF; - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - If @ignore_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; + END; - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @ignore_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND NOT EXISTS - ( - SELECT - 1/0 - FROM #ignore_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End ignore query hashes*/ + SELECT + @where_clause += N'AND EXISTS + ( + SELECT + 1/0 + FROM #forced_plans_failures AS fpf + WHERE fpf.plan_id = qsrs.plan_id + )' + @nc10; +END; - IF @include_plan_hashes IS NOT NULL +IF @query_text_search IS NOT NULL +BEGIN + IF + ( + LEFT + ( + @query_text_search, + 1 + ) <> N'%' + ) BEGIN SELECT - @include_plan_hashes = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@include_plan_hashes)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); - - SELECT - @current_table = 'inserting #include_plan_hashes', - @sql = @isolation_level; + @query_text_search = + N'%' + @query_text_search; + END; - INSERT - #include_plan_hashes WITH(TABLOCK) + IF + ( + LEFT ( - plan_hash_s - ) - EXECUTE sys.sp_executesql - @string_split_strings, - N'@ids nvarchar(4000)', - @include_plan_hashes; + REVERSE + ( + @query_text_search + ), + 1 + ) <> N'%' + ) + BEGIN + SELECT + @query_text_search = + @query_text_search + N'%'; + END; + /* If our query texts contains square brackets (common in Entity Framework queries), add a leading escape character to each bracket character */ + IF @escape_brackets = 1 + BEGIN SELECT - @current_table = 'inserting #include_plan_ids for included plan hashes'; + @query_text_search = + REPLACE(REPLACE(REPLACE( + @query_text_search, + N'[', @escape_character + N'['), + N']', @escape_character + N']'), + N'_', @escape_character + N'_'); + END; - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; + SELECT + @current_table = 'inserting #query_text_search', + @sql = @isolation_level; - SET STATISTICS XML ON; - END; + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - SELECT - @sql += N' + SET STATISTICS XML ON; + END; + + SELECT + @sql += N' SELECT DISTINCT qsp.plan_id FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp @@ -3885,103 +4729,159 @@ WHERE EXISTS ( SELECT 1/0 - FROM #include_plan_hashes AS iph - WHERE iph.plan_hash = qsp.query_plan_hash - ) -OPTION(RECOMPILE);' + @nc10; + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + WHERE qsp.query_id = qsq.query_id + AND EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt + WHERE qsqt.query_text_id = qsq.query_text_id + AND qsqt.query_sql_text LIKE @query_text_search + ) + )'; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; + /* If we are escaping bracket character in our query text search, add the ESCAPE clause and character to the LIKE subquery*/ + IF @escape_brackets = 1 + BEGIN + SELECT + @sql = + REPLACE + ( + @sql, + N'@query_text_search', + N'@query_text_search ESCAPE ''' + @escape_character + N'''' + ); + END; - INSERT - #include_plan_ids - ( - plan_id - ) - EXECUTE sys.sp_executesql - @sql; +/*If we're searching by a procedure name, limit the text search to it */ +IF +( + @procedure_name IS NOT NULL +AND @procedure_exists = 1 +) +BEGIN + SELECT + @sql += N' +AND EXISTS + ( + SELECT + 1/0 + FROM #procedure_plans AS pp + WHERE pp.plan_id = qsp.plan_id + )'; +END; - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; + SELECT + @sql += N' + OPTION(RECOMPILE);' + @nc10; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; + INSERT + #query_text_search + WITH + (TABLOCK) + ( + plan_id + ) + EXECUTE sys.sp_executesql + @sql, + N'@query_text_search nvarchar(4000)', + @query_text_search; - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: + IF @troubleshoot_performance = 1 + BEGIN + SET STATISTICS XML OFF; - If @include_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @include_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #include_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End include plan hashes*/ + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; + END; - IF @ignore_plan_hashes IS NOT NULL - BEGIN - SELECT - @ignore_plan_hashes = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@ignore_plan_hashes)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); + SELECT + @where_clause += N'AND EXISTS + ( + SELECT + 1/0 + FROM #query_text_search AS qst + WHERE qst.plan_id = qsrs.plan_id + )' + @nc10; +END; +IF @query_text_search_not IS NOT NULL +BEGIN + IF + ( + LEFT + ( + @query_text_search_not, + 1 + ) <> N'%' + ) + BEGIN SELECT - @current_table = 'inserting #ignore_plan_hashes', - @sql = @isolation_level; + @query_text_search_not = + N'%' + @query_text_search_not; + END; - INSERT - #ignore_plan_hashes WITH(TABLOCK) + IF + ( + LEFT ( - plan_hash_s - ) - EXECUTE sys.sp_executesql - @string_split_strings, - N'@ids nvarchar(4000)', - @ignore_plan_hashes; + REVERSE + ( + @query_text_search_not + ), + 1 + ) <> N'%' + ) + BEGIN + SELECT + @query_text_search_not = + @query_text_search_not + N'%'; + END; + /* If our query texts contains square brackets (common in Entity Framework queries), add a leading escape character to each bracket character */ + IF @escape_brackets = 1 + BEGIN SELECT - @current_table = 'inserting #ignore_plan_ids for ignored query hashes'; + @query_text_search_not = + REPLACE(REPLACE(REPLACE( + @query_text_search_not, + N'[', @escape_character + N'['), + N']', @escape_character + N']'), + N'_', @escape_character + N'_'); + END; - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; + SELECT + @current_table = 'inserting #query_text_search_not', + @sql = @isolation_level; - SET STATISTICS XML ON; - END; + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - SELECT - @sql += N' + SET STATISTICS XML ON; + END; + + SELECT + @sql += N' SELECT DISTINCT qsp.plan_id FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp @@ -3989,94 +4889,110 @@ WHERE EXISTS ( SELECT 1/0 - FROM #ignore_plan_hashes AS iph - WHERE iph.plan_hash = qsp.query_plan_hash - ) -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #ignore_plan_ids - ( - plan_id - ) - EXECUTE sys.sp_executesql - @sql; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + WHERE qsp.query_id = qsq.query_id + AND EXISTS + ( + SELECT + 1/0 + FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt + WHERE qsqt.query_text_id = qsq.query_text_id + AND qsqt.query_sql_text LIKE @query_text_search_not + ) + )'; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; + /* If we are escaping bracket character in our query text search, add the ESCAPE clause and character to the LIKE subquery*/ + IF @escape_brackets = 1 + BEGIN + SELECT + @sql = + REPLACE + ( + @sql, + N'@query_text_search_not', + N'@query_text_search_not ESCAPE ''' + @escape_character + N'''' + ); + END; - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; +/*If we're searching by a procedure name, limit the text search to it */ +IF +( + @procedure_name IS NOT NULL +AND @procedure_exists = 1 +) +BEGIN + SELECT + @sql += N' +AND EXISTS + ( + SELECT + 1/0 + FROM #procedure_plans AS pp + WHERE pp.plan_id = qsp.plan_id + )'; +END; - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: + SELECT + @sql += N' + OPTION(RECOMPILE);' + @nc10; - If @ignore_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @ignore_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND NOT EXISTS - ( - SELECT - 1/0 - FROM #ignore_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End ignore plan hashes*/ + INSERT + #query_text_search_not + WITH + (TABLOCK) + ( + plan_id + ) + EXECUTE sys.sp_executesql + @sql, + N'@query_text_search_not nvarchar(4000)', + @query_text_search_not; - IF @include_sql_handles IS NOT NULL + IF @troubleshoot_performance = 1 BEGIN - SELECT - @include_sql_handles = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@include_sql_handles)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); + SET STATISTICS XML OFF; - SELECT - @current_table = 'inserting #include_sql_handles', - @sql = @isolation_level; + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - INSERT - #include_sql_handles WITH(TABLOCK) - ( - sql_handle_s - ) EXECUTE sys.sp_executesql - @string_split_strings, - N'@ids nvarchar(4000)', - @include_sql_handles; + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; + END; + + SELECT + @where_clause += N'AND NOT EXISTS + ( + SELECT + 1/0 + FROM #query_text_search_not AS qst + WHERE qst.plan_id = qsrs.plan_id + )' + @nc10; +END; +/* +Validate wait stats stuff +*/ +IF @wait_filter IS NOT NULL +BEGIN + BEGIN SELECT - @current_table = 'inserting #include_sql_handles for included sql handles'; + @current_table = 'inserting #wait_filter', + @sql = @isolation_level; IF @troubleshoot_performance = 1 BEGIN - EXECUTE sys.sp_executesql @troubleshoot_insert, N'@current_table nvarchar(100)', @@ -4087,438 +5003,378 @@ OPTION(RECOMPILE);' + @nc10; SELECT @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - WHERE qsp.query_id = qsq.query_id - AND EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt - WHERE qsqt.query_text_id = qsq.query_text_id - AND EXISTS - ( - SELECT - 1/0 - FROM #include_sql_handles AS ish - WHERE ish.sql_handle = qsqt.statement_sql_handle - ) - ) - ) -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #include_plan_ids - ( - plan_id - ) - EXECUTE sys.sp_executesql - @sql; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; - - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: +SELECT TOP (@top) + qsws.plan_id +FROM ' + @database_name_quoted + N'.sys.query_store_wait_stats AS qsws +WHERE 1 = 1 +AND qsws.wait_category = ' + +CASE @wait_filter + WHEN 'cpu' THEN N'1' + WHEN 'lock' THEN N'3' + WHEN 'locks' THEN N'3' + WHEN 'latch' THEN N'4' + WHEN 'latches' THEN N'4' + WHEN 'buffer latch' THEN N'5' + WHEN 'buffer latches' THEN N'5' + WHEN 'buffer io' THEN N'6' + WHEN 'log' THEN N'14' + WHEN 'log io' THEN N'14' + WHEN 'network' THEN N'15' + WHEN 'network io' THEN N'15' + WHEN 'parallel' THEN N'16' + WHEN 'parallelism' THEN N'16' + WHEN 'memory' THEN N'17' +END ++ N' +GROUP BY + qsws.plan_id +HAVING + SUM(qsws.avg_query_wait_time_ms) > 1000. +ORDER BY + SUM(qsws.avg_query_wait_time_ms) DESC +OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; + END; - If @include_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @include_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #include_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End include plan hashes*/ + INSERT + #wait_filter + WITH + (TABLOCK) + ( + plan_id + ) + EXECUTE sys.sp_executesql + @sql, + N'@top bigint', + @top; - IF @ignore_sql_handles IS NOT NULL + IF @troubleshoot_performance = 1 BEGIN - SELECT - @ignore_sql_handles = - REPLACE(REPLACE(REPLACE(REPLACE( - LTRIM(RTRIM(@ignore_sql_handles)), - CHAR(10), N''), CHAR(13), N''), - NCHAR(10), N''), NCHAR(13), N''); + SET STATISTICS XML OFF; - SELECT - @current_table = 'inserting #ignore_sql_handles', - @sql = @isolation_level; + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - INSERT - #ignore_sql_handles WITH(TABLOCK) - ( - sql_handle_s - ) EXECUTE sys.sp_executesql - @string_split_strings, - N'@ids nvarchar(4000)', - @ignore_sql_handles; + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; + END; - SELECT - @current_table = 'inserting #ignore_plan_ids for ignored sql handles'; + SELECT + @where_clause += N'AND EXISTS + ( + SELECT + 1/0 + FROM #wait_filter AS wf + WHERE wf.plan_id = qsrs.plan_id + )' + @nc10; +END; - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; +/* +This section screens out index create and alter statements because who cares +*/ - SET STATISTICS XML ON; - END; +SELECT + @current_table = 'inserting #maintenance_plans', + @sql = @isolation_level; - SELECT - @sql += N' +IF @troubleshoot_performance = 1 +BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; + + SET STATISTICS XML ON; +END; + +SELECT + @sql += N' SELECT DISTINCT - qsp.plan_id + qsp.plan_id FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS +WHERE NOT EXISTS ( SELECT - 1/0 + 1/0 FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - WHERE qsp.query_id = qsq.query_id - AND EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt - WHERE qsqt.query_text_id = qsq.query_text_id - AND EXISTS - ( - SELECT - 1/0 - FROM #ignore_sql_handles AS ish - WHERE ish.sql_handle = qsqt.statement_sql_handle - ) - ) + JOIN ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt + ON qsqt.query_text_id = qsq.query_text_id + WHERE qsq.query_id = qsp.query_id + AND qsqt.query_sql_text NOT LIKE N''ALTER INDEX%'' + AND qsqt.query_sql_text NOT LIKE N''ALTER TABLE%'' + AND qsqt.query_sql_text NOT LIKE N''CREATE%INDEX%'' + AND qsqt.query_sql_text NOT LIKE N''CREATE STATISTICS%'' + AND qsqt.query_sql_text NOT LIKE N''UPDATE STATISTICS%'' + AND qsqt.query_sql_text NOT LIKE N''%SELECT StatMan%'' + AND qsqt.query_sql_text NOT LIKE N''DBCC%'' + AND qsqt.query_sql_text NOT LIKE N''(@[_]msparam%'' ) OPTION(RECOMPILE);' + @nc10; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #ignore_plan_ids - ( - plan_id - ) - EXECUTE sys.sp_executesql - @sql; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; +IF @debug = 1 +BEGIN + PRINT LEN(@sql); + PRINT @sql; +END; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; +INSERT + #maintenance_plans +WITH + (TABLOCK) +( + plan_id +) +EXECUTE sys.sp_executesql + @sql; - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; +IF @troubleshoot_performance = 1 +BEGIN + SET STATISTICS XML OFF; - /* - This section of code confused me when I came back to it, - so I'm going to add a note here about why I do this: + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - If @ignore_plan_ids is NULL at this point, it's because - the user didn't populate the parameter. + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; +END; - We need to do this because it's how we figure - out which plans to keep in the main query - */ - IF @ignore_plan_ids IS NULL - BEGIN - SELECT - @where_clause += N'AND NOT EXISTS - ( - SELECT - 1/0 - FROM #ignore_plan_ids AS idi - WHERE idi.plan_id = qsrs.plan_id - )' + @nc10; - END; - END; /*End ignore plan hashes*/ -END; /*End hash and handle filtering*/ +SELECT + @where_clause += N'AND NOT EXISTS + ( + SELECT + 1/0 + FROM #maintenance_plans AS mp + WHERE mp.plan_id = qsrs.plan_id + )' + @nc10; -IF @sql_2022_views = 1 -BEGIN - IF @only_queries_with_hints = 1 - BEGIN - SELECT - @current_table = 'inserting #only_queries_with_hints', - @sql = @isolation_level; +/* +Tidy up the where clause a bit +*/ +SELECT + @where_clause = + SUBSTRING + ( + @where_clause, + 1, + LEN(@where_clause) - 1 + ); - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' - SELECT DISTINCT - qsp.plan_id - FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query_hints AS qsqh - WHERE qsqh.query_id = qsp.query_id - )'; +/* +Regression mode differs significantly from our defaults. +In this mode, we measure every query hash in the time period +specified by @regression_baseline_start_date and +@regression_baseline_end_date ("the baseline time period"). +Our measurements are taken based on the metric given +by @sort_order. +For all of the hashes we have taken measurements for, we +make the same measurement for the time period specified +by @start_date and @end_date ("the current time period"). +We then compare each hashes' measurement across the two +time periods, by the means specified by +@regression_comparator and take the @top results ordered by +@regression_direction. +We then get every plan_id in both time periods for those +query hashes and carry on as normal. - SELECT - @sql += N' - OPTION(RECOMPILE);' + @nc10; +This gives us three immediate concerns. We: + 1) Need to adjust our @where_clause to refer to the + baseline time period. + 2) Need all of the queries from the baseline time + period (rather than just the @top whatever). + 3) Are interested in the query hashes rather than + just plan_ids. - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; +We address part of the first concern immediately. +Later, we will do some wicked and foul things to +modify our dynamic SQL's usages of @where_clause +to use @regression_where_clause. +*/ +IF @regression_mode = 1 +BEGIN - INSERT - #only_queries_with_hints WITH(TABLOCK) +SELECT + @regression_where_clause = + REPLACE ( - plan_id - ) - EXECUTE sys.sp_executesql - @sql - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; + REPLACE + ( + @where_clause, + '@start_date', + '@regression_baseline_start_date' + ), + '@end_date', + '@regression_baseline_end_date' + ); +END; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; +/* +Populate sort-helping tables, if needed. - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; +In theory, these exist just to put in scope +columns that wouldn't normally be in scope. +However, they're also quite helpful for the next +temp table, #distinct_plans. - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #only_queries_with_hints AS qst - WHERE qst.plan_id = qsrs.plan_id - )' + @nc10; - END; +Note that this block must come after #maintenance_plans +because that edits @where_clause and we want to use +that here. - IF @only_queries_with_feedback = 1 - BEGIN - SELECT - @current_table = 'inserting #only_queries_with_feedback', - @sql = @isolation_level; +Regression mode complicates this process considerably. +It forces us to use different dates. +We also have to adjust @top. - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; +Luckily, the 'plan count by hashes' sort +order is not supported in regression mode. +Earlier on, we throw an error if somebody +tries (it just doesn't make sense). +*/ +IF @sort_order = 'plan count by hashes' +BEGIN + SELECT + @current_table = 'inserting #plan_ids_with_query_hashes', + @sql = @isolation_level; - SET STATISTICS XML ON; - END; + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - SELECT - @sql += N' - SELECT DISTINCT - qsp.plan_id - FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_plan_feedback AS qsqf - WHERE qsqf.plan_id = qsp.plan_id - )'; + SET STATISTICS XML ON; + END; + SELECT + /* + This sort order is useless if we don't show the + ties, so only DENSE_RANK() makes sense to use. + This is why this is not SELECT TOP. + */ + @sql += N' + SELECT + @database_id, + ranked_plans.plan_id, + ranked_plans.query_hash, + ranked_plans.plan_hash_count_for_query_hash + FROM + ( SELECT - @sql += N' - OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #only_queries_with_feedback WITH(TABLOCK) + QueryHashesWithIds.plan_id, + QueryHashesWithCounts.query_hash, + QueryHashesWithCounts.plan_hash_count_for_query_hash, + ranking = + DENSE_RANK() OVER + ( + ORDER BY + QueryHashesWithCounts.plan_hash_count_for_query_hash DESC, + QueryHashesWithCounts.query_hash DESC + ) + FROM ( - plan_id - ) - EXECUTE sys.sp_executesql - @sql - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; + SELECT + qsq.query_hash, + plan_hash_count_for_query_hash = + COUNT(DISTINCT qsp.query_plan_hash) + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + ON qsq.query_id = qsp.query_id + JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + ON qsp.plan_id = qsrs.plan_id + WHERE 1 = 1 + ' + @where_clause + + N' + GROUP + BY qsq.query_hash + ) AS QueryHashesWithCounts + JOIN + ( + SELECT DISTINCT + qsq.query_hash, + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + ON qsq.query_id = qsp.query_id + JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + ON qsp.plan_id = qsrs.plan_id + WHERE 1 = 1 + ' + @where_clause + + N' + ) AS QueryHashesWithIds + ON QueryHashesWithCounts.query_hash = QueryHashesWithIds.query_hash + ) AS ranked_plans + WHERE ranked_plans.ranking <= @top + OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #only_queries_with_feedback AS qst - WHERE qst.plan_id = qsrs.plan_id - )' + @nc10; + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; END; - IF @only_queries_with_variants = 1 - BEGIN - SELECT - @current_table = 'inserting #only_queries_with_variants', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' - SELECT DISTINCT - qsp.plan_id - FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query_variant AS qsqv - WHERE qsqv.query_variant_query_id = qsp.query_id - )'; - - SELECT - @sql += N' - OPTION(RECOMPILE);' + @nc10; + INSERT + #plan_ids_with_query_hashes + WITH + (TABLOCK) + ( + database_id, + plan_id, + query_hash, + plan_hash_count_for_query_hash + ) + EXECUTE sys.sp_executesql + @sql, + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; + IF @troubleshoot_performance = 1 + BEGIN + SET STATISTICS XML OFF; - INSERT - #only_queries_with_variants WITH(TABLOCK) - ( - plan_id - ) EXECUTE sys.sp_executesql - @sql - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #only_queries_with_variants AS qst - WHERE qst.plan_id = qsrs.plan_id - )' + @nc10; + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; END; END; - -IF -( - @only_queries_with_forced_plans = 1 - OR @only_queries_with_forced_plan_failures = 1 -) +IF @sort_order = 'total waits' BEGIN SELECT - @current_table = 'inserting #forced_plans_failures', + @current_table = 'inserting #plan_ids_with_total_waits', @sql = @isolation_level; IF @troubleshoot_performance = 1 @@ -4533,21 +5389,59 @@ BEGIN SELECT @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE qsp.is_forced_plan = 1'; + SELECT TOP (@top) + @database_id, + qsrs.plan_id, + from_regression_baseline = + CASE + WHEN qsrs.last_execution_time >= @start_date + AND qsrs.last_execution_time < @end_date + THEN ''No'' + ELSE ''Yes'' + END, + total_query_wait_time_ms = + SUM(qsws.total_query_wait_time_ms) + FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + JOIN ' + @database_name_quoted + N'.sys.query_store_wait_stats AS qsws + ON qsrs.plan_id = qsws.plan_id + WHERE 1 = 1 + ' + + CASE WHEN @regression_mode = 1 + THEN N' AND ( 1 = 1 + ' + @regression_where_clause + + N' ) +OR + ( 1 = 1 + ' + + @where_clause + + N' ) ' + ELSE @where_clause + END + + N' + GROUP + BY qsrs.plan_id, + CASE + WHEN qsrs.last_execution_time >= @start_date + AND qsrs.last_execution_time < @end_date + THEN ''No'' + ELSE ''Yes'' + END + ORDER BY + SUM(qsws.total_query_wait_time_ms) DESC + OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; -IF @only_queries_with_forced_plan_failures = 1 -BEGIN - SELECT - @sql += N' -AND qsp.last_force_failure_reason > 0' -END + IF @regression_mode = 1 + BEGIN - SELECT - @sql += N' -OPTION(RECOMPILE);' + @nc10; + /* Very stupid way to stop us repeating the above code. */ + SELECT + @sql = REPLACE + ( + @sql, + 'TOP (@top)', + 'TOP (2147483647 + (0 * @top))' + ); + END; IF @debug = 1 BEGIN @@ -4556,12 +5450,30 @@ OPTION(RECOMPILE);' + @nc10; END; INSERT - #forced_plans_failures WITH(TABLOCK) + #plan_ids_with_total_waits + WITH + (TABLOCK) ( - plan_id + database_id, + plan_id, + from_regression_baseline, + total_query_wait_time_ms ) EXECUTE sys.sp_executesql - @sql + @sql, + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; IF @troubleshoot_performance = 1 BEGIN @@ -4579,64 +5491,17 @@ OPTION(RECOMPILE);' + @nc10; @sql, @current_table; END; - - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #forced_plans_failures AS fpf - WHERE fpf.plan_id = qsrs.plan_id - )' + @nc10; END; +/* +'total waits' is special. It's a sum, not a max, so +we cover it above rather than here. +*/ -IF @query_text_search IS NOT NULL +IF @sort_order_is_a_wait = 1 +AND @sort_order <> 'total waits' BEGIN - IF - ( - LEFT - ( - @query_text_search, - 1 - ) <> N'%' - ) - BEGIN - SELECT - @query_text_search = - N'%' + @query_text_search; - END; - - IF - ( - LEFT - ( - REVERSE - ( - @query_text_search - ), - 1 - ) <> N'%' - ) - BEGIN - SELECT - @query_text_search = - @query_text_search + N'%'; - END; - - /* If our query texts contains square brackets (common in Entity Framework queries), add a leading escape character to each bracket character */ - IF @escape_brackets = 1 - BEGIN - SELECT - @query_text_search = - REPLACE(REPLACE(REPLACE( - @query_text_search, - N'[', @escape_character + N'['), - N']', @escape_character + N']'), - N'_', @escape_character + N'_'); - END; - SELECT - @current_table = 'inserting #query_text_search', + @current_table = 'inserting #plan_ids_with_total_waits', @sql = @isolation_level; IF @troubleshoot_performance = 1 @@ -4651,60 +5516,79 @@ BEGIN SELECT @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - WHERE qsp.query_id = qsq.query_id - AND EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt - WHERE qsqt.query_text_id = qsq.query_text_id - AND qsqt.query_sql_text LIKE @query_text_search - ) - )'; + SELECT TOP (@top) + @database_id, + qsrs.plan_id, + from_regression_baseline = + CASE + WHEN qsrs.last_execution_time >= @start_date + AND qsrs.last_execution_time < @end_date + THEN ''No'' + ELSE ''Yes'' + END, + total_query_wait_time_ms = + MAX(qsws.total_query_wait_time_ms) + FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + JOIN ' + @database_name_quoted + N'.sys.query_store_wait_stats AS qsws + ON qsrs.plan_id = qsws.plan_id + WHERE 1 = 1 + AND qsws.wait_category = ' + + CASE @sort_order + WHEN 'cpu waits' THEN N'1' + WHEN 'lock waits' THEN N'3' + WHEN 'locks waits' THEN N'3' + WHEN 'latch waits' THEN N'4' + WHEN 'latches waits' THEN N'4' + WHEN 'buffer latch waits' THEN N'5' + WHEN 'buffer latches waits' THEN N'5' + WHEN 'buffer io waits' THEN N'6' + WHEN 'log waits' THEN N'14' + WHEN 'log io waits' THEN N'14' + WHEN 'network waits' THEN N'15' + WHEN 'network io waits' THEN N'15' + WHEN 'parallel waits' THEN N'16' + WHEN 'parallelism waits' THEN N'16' + WHEN 'memory waits' THEN N'17' + END + + N' + ' + + CASE WHEN @regression_mode = 1 + THEN N' AND ( 1 = 1 + ' + @regression_where_clause + + N' ) + OR + ( 1 = 1 + ' + + @where_clause + + N' ) ' + ELSE @where_clause + END + + N' + GROUP + BY qsrs.plan_id, + CASE + WHEN qsrs.last_execution_time >= @start_date + AND qsrs.last_execution_time < @end_date + THEN ''No'' + ELSE ''Yes'' + END + ORDER BY + MAX(qsws.total_query_wait_time_ms) DESC + OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; - /* If we are escaping bracket character in our query text search, add the ESCAPE clause and character to the LIKE subquery*/ - IF @escape_brackets = 1 + IF @regression_mode = 1 BEGIN + + /* Very stupid way to stop us repeating the above code. */ SELECT - @sql = - REPLACE - ( - @sql, - N'@query_text_search', - N'@query_text_search ESCAPE ''' + @escape_character + N'''' - ); + @sql = REPLACE + ( + @sql, + 'TOP (@top)', + 'TOP (2147483647 + (0 * @top))' + ); END; -/*If we're searching by a procedure name, limit the text search to it */ -IF -( - @procedure_name IS NOT NULL -AND @procedure_exists = 1 -) -BEGIN - SELECT - @sql += N' -AND EXISTS - ( - SELECT - 1/0 - FROM #procedure_plans AS pp - WHERE pp.plan_id = qsp.plan_id - )'; -END; - - SELECT - @sql += N' - OPTION(RECOMPILE);' + @nc10; - IF @debug = 1 BEGIN PRINT LEN(@sql); @@ -4712,14 +5596,30 @@ END; END; INSERT - #query_text_search WITH(TABLOCK) + #plan_ids_with_total_waits + WITH + (TABLOCK) ( - plan_id + database_id, + plan_id, + from_regression_baseline, + total_query_wait_time_ms ) EXECUTE sys.sp_executesql @sql, - N'@query_text_search nvarchar(4000)', - @query_text_search; + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; IF @troubleshoot_performance = 1 BEGIN @@ -4737,64 +5637,23 @@ END; @sql, @current_table; END; - - SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #query_text_search AS qst - WHERE qst.plan_id = qsrs.plan_id - )' + @nc10; END; +/*End populating sort-helping tables*/ -IF @query_text_search_not IS NOT NULL +/* +This is where the bulk of the regression mode +work is done. We grab the metrics for both time +periods for each query hash, compare them, +and get the @top. +*/ +IF @regression_mode = 1 BEGIN - IF - ( - LEFT - ( - @query_text_search_not, - 1 - ) <> N'%' - ) - BEGIN - SELECT - @query_text_search_not = - N'%' + @query_text_search_not; - END; - - IF - ( - LEFT - ( - REVERSE - ( - @query_text_search_not - ), - 1 - ) <> N'%' - ) - BEGIN - SELECT - @query_text_search_not = - @query_text_search_not + N'%'; - END; - - /* If our query texts contains square brackets (common in Entity Framework queries), add a leading escape character to each bracket character */ - IF @escape_brackets = 1 - BEGIN - SELECT - @query_text_search_not = - REPLACE(REPLACE(REPLACE( - @query_text_search_not, - N'[', @escape_character + N'['), - N']', @escape_character + N']'), - N'_', @escape_character + N'_'); - END; - + /* + We begin by getting the metrics per query hash + in the time period. + */ SELECT - @current_table = 'inserting #query_text_search_not', + @current_table = 'inserting #regression_baseline_runtime_stats', @sql = @isolation_level; IF @troubleshoot_performance = 1 @@ -4809,75 +5668,74 @@ BEGIN SELECT @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - WHERE qsp.query_id = qsq.query_id - AND EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt - WHERE qsqt.query_text_id = qsq.query_text_id - AND qsqt.query_sql_text LIKE @query_text_search_not - ) - )'; + SELECT + qsq.query_hash, + /* All of these but count_executions are already floats. */ + regression_metric_average = + CONVERT + ( + float, + AVG + (' + + CASE @sort_order + WHEN 'cpu' THEN N'qsrs.avg_cpu_time' + WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' + WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' + WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' + WHEN 'duration' THEN N'qsrs.avg_duration' + WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' + WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END + WHEN 'executions' THEN N'qsrs.count_executions' + WHEN 'rows' THEN N'qsrs.avg_rowcount' + ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'waits.total_query_wait_time_ms' ELSE N'qsrs.avg_cpu_time' END + END + + N' + ) + ) + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + ON qsq.query_id = qsp.query_id + JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + ON qsp.plan_id = qsrs.plan_id + LEFT JOIN #plan_ids_with_total_waits AS waits + ON qsp.plan_id = waits.plan_id + AND waits.from_regression_baseline = ''Yes'' + WHERE 1 = 1 + ' + @regression_where_clause + + N' + GROUP + BY qsq.query_hash + OPTION(RECOMPILE);' + @nc10; - /* If we are escaping bracket character in our query text search, add the ESCAPE clause and character to the LIKE subquery*/ - IF @escape_brackets = 1 - BEGIN - SELECT - @sql = - REPLACE - ( - @sql, - N'@query_text_search_not', - N'@query_text_search_not ESCAPE ''' + @escape_character + N'''' - ); - END; - -/*If we're searching by a procedure name, limit the text search to it */ -IF -( - @procedure_name IS NOT NULL -AND @procedure_exists = 1 -) -BEGIN - SELECT - @sql += N' -AND EXISTS - ( - SELECT - 1/0 - FROM #procedure_plans AS pp - WHERE pp.plan_id = qsp.plan_id - )'; -END; - - SELECT - @sql += N' - OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 + IF @debug = 1 BEGIN PRINT LEN(@sql); PRINT @sql; END; INSERT - #query_text_search_not WITH(TABLOCK) + #regression_baseline_runtime_stats + WITH + (TABLOCK) ( - plan_id + query_hash, + regression_metric_average ) EXECUTE sys.sp_executesql @sql, - N'@query_text_search_not nvarchar(4000)', - @query_text_search_not; + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; IF @troubleshoot_performance = 1 BEGIN @@ -4896,69 +5754,71 @@ END; @current_table; END; + /* + We now take the same measurement for all of the same query hashes, + but in the @where_clause time period. + */ SELECT - @where_clause += N'AND NOT EXISTS - ( - SELECT - 1/0 - FROM #query_text_search_not AS qst - WHERE qst.plan_id = qsrs.plan_id - )' + @nc10; -END; + @current_table = 'inserting #regression_current_runtime_stats', + @sql = @isolation_level; -/* -Validate wait stats stuff -*/ -IF @wait_filter IS NOT NULL -BEGIN + IF @troubleshoot_performance = 1 BEGIN - SELECT - @current_table = 'inserting #wait_filter', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - SET STATISTICS XML ON; - END; + SET STATISTICS XML ON; + END; + SELECT + @sql += N' + SELECT + qsq.query_hash, + /* All of these but count_executions are already floats. */ + current_metric_average = + CONVERT + ( + float, + AVG + (' + + CASE @sort_order + WHEN 'cpu' THEN N'qsrs.avg_cpu_time' + WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' + WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' + WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' + WHEN 'duration' THEN N'qsrs.avg_duration' + WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' + WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END + WHEN 'executions' THEN N'qsrs.count_executions' + WHEN 'rows' THEN N'qsrs.avg_rowcount' + ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'waits.total_query_wait_time_ms' ELSE N'qsrs.avg_cpu_time' END + END + + N' + ) + ) + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + ON qsq.query_id = qsp.query_id + JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + ON qsp.plan_id = qsrs.plan_id + LEFT JOIN #plan_ids_with_total_waits AS waits + ON qsp.plan_id = waits.plan_id + AND waits.from_regression_baseline = ''No'' + WHERE 1 = 1 + AND EXISTS + ( SELECT - @sql += N' -SELECT TOP (@top) - qsws.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_wait_stats AS qsws -WHERE 1 = 1 -AND qsws.wait_category = ' + -CASE @wait_filter - WHEN 'cpu' THEN N'1' - WHEN 'lock' THEN N'3' - WHEN 'locks' THEN N'3' - WHEN 'latch' THEN N'4' - WHEN 'latches' THEN N'4' - WHEN 'buffer latch' THEN N'5' - WHEN 'buffer latches' THEN N'5' - WHEN 'buffer io' THEN N'6' - WHEN 'log' THEN N'14' - WHEN 'log io' THEN N'14' - WHEN 'network' THEN N'15' - WHEN 'network io' THEN N'15' - WHEN 'parallel' THEN N'16' - WHEN 'parallelism' THEN N'16' - WHEN 'memory' THEN N'17' -END -+ N' -GROUP BY - qsws.plan_id -HAVING - SUM(qsws.avg_query_wait_time_ms) > 1000. -ORDER BY - SUM(qsws.avg_query_wait_time_ms) DESC -OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; - END; + 1/0 + FROM #regression_baseline_runtime_stats AS base + WHERE base.query_hash = qsq.query_hash + ) + ' + @where_clause + + N' + GROUP + BY qsq.query_hash + OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN @@ -4967,14 +5827,28 @@ OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; END; INSERT - #wait_filter WITH(TABLOCK) + #regression_current_runtime_stats + WITH + (TABLOCK) ( - plan_id + query_hash, + current_metric_average ) EXECUTE sys.sp_executesql @sql, - N'@top bigint', - @top; + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; IF @troubleshoot_performance = 1 BEGIN @@ -4994,910 +5868,1151 @@ OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; END; SELECT - @where_clause += N'AND EXISTS - ( - SELECT - 1/0 - FROM #wait_filter AS wf - WHERE wf.plan_id = qsrs.plan_id - )' + @nc10; -END; - -/* -This section screens out index create and alter statements because who cares -*/ - -SELECT - @current_table = 'inserting #maintenance_plans', - @sql = @isolation_level; - -IF @troubleshoot_performance = 1 -BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; -END; + @current_table = 'inserting #regression_changes', + @sql = @isolation_level; -SELECT - @sql += N' -SELECT DISTINCT - qsp.plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp -WHERE NOT EXISTS - ( - SELECT - 1/0 - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - JOIN ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt - ON qsqt.query_text_id = qsq.query_text_id - WHERE qsq.query_id = qsp.query_id - AND qsqt.query_sql_text NOT LIKE N''ALTER INDEX%'' - AND qsqt.query_sql_text NOT LIKE N''ALTER TABLE%'' - AND qsqt.query_sql_text NOT LIKE N''CREATE%INDEX%'' - AND qsqt.query_sql_text NOT LIKE N''CREATE STATISTICS%'' - AND qsqt.query_sql_text NOT LIKE N''UPDATE STATISTICS%'' - AND qsqt.query_sql_text NOT LIKE N''%SELECT StatMan%'' - AND qsqt.query_sql_text NOT LIKE N''DBCC%'' - AND qsqt.query_sql_text NOT LIKE N''(@[_]msparam%'' - ) -OPTION(RECOMPILE);' + @nc10; + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; -IF @debug = 1 -BEGIN - PRINT LEN(@sql); - PRINT @sql; -END; + SET STATISTICS XML ON; + END; -INSERT - #maintenance_plans WITH(TABLOCK) -( - plan_id -) -EXECUTE sys.sp_executesql - @sql; + /* + Now that we have the data from + both time periods, we must + compare them as @regression_comparator + demands and order them as + @regression_direction demands. -IF @troubleshoot_performance = 1 -BEGIN - SET STATISTICS XML OFF; + However, we care about query_hashes + here despite everything after this + wanting plan_ids. This means we + must repeat some of the tricks + we used for #plan_ids_with_query_hashes. + */ + SELECT + @sql += N' + SELECT + @database_id, + plans_for_hashes.plan_id, + hashes_with_changes.query_hash, + change_since_regression_time_period = + ' + + /* + If we are returning differences that are not percentages, + then we need the units we show for any given metric to be + the same as anywhere else that gives the same metric. + If we do not, then our final output will look wrong. + For example, our CPU time will be 1,000 times bigger + here than it is in any other column. + To avoid this problem, we need to replicate the calculations + later used to populate #query_store_runtime_stats. + */ + CASE @regression_comparator + WHEN 'absolute' THEN + CASE @sort_order + WHEN 'cpu' THEN N'hashes_with_changes.change_since_regression_time_period / 1000.' + WHEN 'logical reads' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' + WHEN 'physical reads' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' + WHEN 'writes' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' + WHEN 'duration' THEN N'hashes_with_changes.change_since_regression_time_period / 1000.' + WHEN 'memory' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' + WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' ELSE N'hashes_with_changes.change_since_regression_time_period / 1000.' END + WHEN 'executions' THEN N'hashes_with_changes.change_since_regression_time_period' + WHEN 'rows' THEN N'hashes_with_changes.change_since_regression_time_period' + ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'hashes_with_changes.change_since_regression_time_period / 1000.' ELSE N'hashes_with_changes.change_since_regression_time_period / 1000.' END + END + ELSE N'hashes_with_changes.change_since_regression_time_period' END + + N' + FROM + ( + SELECT TOP (@top) + compared_stats.query_hash, + compared_stats.change_since_regression_time_period + FROM + ( + SELECT + current_stats.query_hash, + change_since_regression_time_period = + ' + + CASE @regression_comparator + WHEN 'relative' THEN N'((current_stats.current_metric_average / NULLIF(baseline.regression_metric_average, 0.0)) - 1.0)' + WHEN 'absolute' THEN N'(current_stats.current_metric_average - baseline.regression_metric_average)' + END + + N' + FROM #regression_current_runtime_stats AS current_stats + JOIN #regression_baseline_runtime_stats AS baseline + ON current_stats.query_hash = baseline.query_hash + ) AS compared_stats + ORDER BY + ' + /* + Current metrics that are better than that of the baseline period, + will give change_since_regression_time_period values that + are smaller than metrics that are worse. + In other words, ORDER BY change_since_regression_time_period DESC + gives us the regressed queries first. + This is true regardless of @regression_comparator. + To make @regression_direction behave as intended, we + need to account for this. We could use dynamic SQL, + but mathematics has given us better tools. + */ + + CASE @regression_direction + WHEN 'regressed' THEN N'change_since_regression_time_period' + WHEN 'worse' THEN N'change_since_regression_time_period' + WHEN 'improved' THEN N'change_since_regression_time_period * -1.0' + WHEN 'better' THEN N'change_since_regression_time_period * -1.0' + /* + The following two branches cannot be hit if + @regression_comparator is 'relative'. + We have made errors be thrown if somebody tries + to mix the two. + If you can figure out a way to make the two make + sense together, then feel free to add it in. + */ + WHEN 'magnitude' THEN N'ABS(change_since_regression_time_period)' + WHEN 'absolute' THEN N'ABS(change_since_regression_time_period)' + END + + N' DESC + ) AS hashes_with_changes + JOIN + ( + SELECT DISTINCT + qsq.query_hash, + qsp.plan_id + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + ON qsq.query_id = qsp.query_id + JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + ON qsp.plan_id = qsrs.plan_id + WHERE + ( 1 = 1 + ' + /* We want each time period's plan_ids for these query hashes. */ + + @regression_where_clause + + N' + ) + OR + ( 1 = 1 + ' + + @where_clause + + N' + ) + ) AS plans_for_hashes + ON hashes_with_changes.query_hash = plans_for_hashes.query_hash + OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + INSERT + #regression_changes + WITH + (TABLOCK) + ( + database_id, + plan_id, + query_hash, + change_since_regression_time_period + ) EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', @sql, - @current_table; -END; + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; -SELECT - @where_clause += N'AND NOT EXISTS - ( - SELECT - 1/0 - FROM #maintenance_plans AS mp - WHERE mp.plan_id = qsrs.plan_id - )' + @nc10; + IF @troubleshoot_performance = 1 + BEGIN + SET STATISTICS XML OFF; + + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; + + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; + END; + +END; /* -Tidy up the where clause a bit +This gets the plan_ids we care about. + +We unfortunately need an ELSE IF chain here +because the final branch contains defaults +that we only want to hit if we did not hit +any others. */ SELECT - @where_clause = - SUBSTRING - ( - @where_clause, - 1, - LEN(@where_clause) - 1 - ); + @current_table = 'inserting #distinct_plans', + @sql = @isolation_level; -/* -Regression mode differs significantly from our defaults. -In this mode, we measure every query hash in the time period -specified by @regression_baseline_start_date and -@regression_baseline_end_date ("the baseline time period"). -Our measurements are taken based on the metric given -by @sort_order. -For all of the hashes we have taken measurements for, we -make the same measurement for the time period specified -by @start_date and @end_date ("the current time period"). -We then compare each hashes' measurement across the two -time periods, by the means specified by -@regression_comparator and take the @top results ordered by -@regression_direction. -We then get every plan_id in both time periods for those -query hashes and carry on as normal. +IF @troubleshoot_performance = 1 +BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; -This gives us three immediate concerns. We: - 1) Need to adjust our @where_clause to refer to the - baseline time period. - 2) Need all of the queries from the baseline time - period (rather than just the @top whatever). - 3) Are interested in the query hashes rather than - just plan_ids. + SET STATISTICS XML ON; +END; -We address part of the first concern immediately. -Later, we will do some wicked and foul things to -modify our dynamic SQL's usages of @where_clause -to use @regression_where_clause. -*/ IF @regression_mode = 1 BEGIN - -SELECT - @regression_where_clause = - REPLACE - ( - REPLACE - ( - @where_clause, - '@start_date', - '@regression_baseline_start_date' - ), - '@end_date', - '@regression_baseline_end_date' - ); + SELECT + @sql += N' + SELECT DISTINCT + plan_id + FROM #regression_changes + WHERE database_id = @database_id + OPTION(RECOMPILE);' + @nc10; END; - -/* -Populate sort-helping tables, if needed. - -In theory, these exist just to put in scope -columns that wouldn't normally be in scope. -However, they're also quite helpful for the next -temp table, #distinct_plans. - -Note that this block must come after #maintenance_plans -because that edits @where_clause and we want to use -that here. - -Regression mode complicates this process considerably. -It forces us to use different dates. -We also have to adjust @top. - -Luckily, the 'plan count by hashes' sort -order is not supported in regression mode. -Earlier on, we throw an error if somebody -tries (it just doesn't make sense). -*/ -IF @sort_order = 'plan count by hashes' +ELSE IF @sort_order = 'plan count by hashes' BEGIN SELECT - @current_table = 'inserting #plan_ids_with_query_hashes', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - /* - This sort order is useless if we don't show the - ties, so only DENSE_RANK() makes sense to use. - This is why this is not SELECT TOP. - */ @sql += N' - SELECT - @database_id, - ranked_plans.plan_id, - ranked_plans.query_hash, - ranked_plans.plan_hash_count_for_query_hash - FROM - ( - SELECT - QueryHashesWithIds.plan_id, - QueryHashesWithCounts.query_hash, - QueryHashesWithCounts.plan_hash_count_for_query_hash, - ranking = - DENSE_RANK() OVER - ( - ORDER BY - QueryHashesWithCounts.plan_hash_count_for_query_hash DESC, - QueryHashesWithCounts.query_hash DESC - ) - FROM - ( - SELECT - qsq.query_hash, - plan_hash_count_for_query_hash = - COUNT(DISTINCT qsp.query_plan_hash) - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - ON qsq.query_id = qsp.query_id - JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - ON qsp.plan_id = qsrs.plan_id - WHERE 1 = 1 - ' + @where_clause - + N' - GROUP - BY qsq.query_hash - ) AS QueryHashesWithCounts - JOIN - ( - SELECT DISTINCT - qsq.query_hash, - qsp.plan_id - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - ON qsq.query_id = qsp.query_id - JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - ON qsp.plan_id = qsrs.plan_id - WHERE 1 = 1 - ' + @where_clause - + N' - ) AS QueryHashesWithIds - ON QueryHashesWithCounts.query_hash = QueryHashesWithIds.query_hash - ) AS ranked_plans - WHERE ranked_plans.ranking <= @top - OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #plan_ids_with_query_hashes WITH(TABLOCK) - ( - database_id, - plan_id, - query_hash, - plan_hash_count_for_query_hash - ) - EXECUTE sys.sp_executesql - @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; + SELECT DISTINCT + plan_id + FROM #plan_ids_with_query_hashes + WHERE database_id = @database_id + OPTION(RECOMPILE);' + @nc10; END; -IF @sort_order = 'total waits' +ELSE IF @sort_order_is_a_wait = 1 BEGIN SELECT - @current_table = 'inserting #plan_ids_with_total_waits', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - + @sql += N' + SELECT DISTINCT + plan_id + FROM #plan_ids_with_total_waits + WHERE database_id = @database_id + OPTION(RECOMPILE);' + @nc10; +END; +ELSE +BEGIN SELECT @sql += N' SELECT TOP (@top) - @database_id, - qsrs.plan_id, - from_regression_baseline = - CASE - WHEN qsrs.last_execution_time >= @start_date - AND qsrs.last_execution_time < @end_date - THEN ''No'' - ELSE ''Yes'' - END, - total_query_wait_time_ms = - SUM(qsws.total_query_wait_time_ms) + qsrs.plan_id FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - JOIN ' + @database_name_quoted + N'.sys.query_store_wait_stats AS qsws - ON qsrs.plan_id = qsws.plan_id WHERE 1 = 1 - ' - + CASE WHEN @regression_mode = 1 - THEN N' AND ( 1 = 1 - ' + @regression_where_clause - + N' ) -OR - ( 1 = 1 - ' - + @where_clause - + N' ) ' - ELSE @where_clause - END + ' + @where_clause + N' GROUP - BY qsrs.plan_id, - CASE - WHEN qsrs.last_execution_time >= @start_date - AND qsrs.last_execution_time < @end_date - THEN ''No'' - ELSE ''Yes'' - END + BY qsrs.plan_id ORDER BY - SUM(qsws.total_query_wait_time_ms) DESC + MAX(' + + CASE @sort_order + WHEN 'cpu' THEN N'qsrs.avg_cpu_time' + WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' + WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' + WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' + WHEN 'duration' THEN N'qsrs.avg_duration' + WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' + WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END + WHEN 'executions' THEN N'qsrs.count_executions' + WHEN 'recent' THEN N'qsrs.last_execution_time' + WHEN 'rows' THEN N'qsrs.avg_rowcount' + ELSE N'qsrs.avg_cpu_time' + END + + N') DESC OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; +END; - IF @regression_mode = 1 - BEGIN +IF @debug = 1 +BEGIN + PRINT LEN(@sql); + PRINT @sql; +END; - /* Very stupid way to stop us repeating the above code. */ - SELECT - @sql = REPLACE - ( - @sql, - 'TOP (@top)', - 'TOP (2147483647 + (0 * @top))' - ); - END; +INSERT + #distinct_plans +WITH + (TABLOCK) +( + plan_id +) +EXECUTE sys.sp_executesql + @sql, + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; +IF @troubleshoot_performance = 1 +BEGIN + SET STATISTICS XML OFF; - INSERT - #plan_ids_with_total_waits WITH(TABLOCK) - ( - database_id, - plan_id, - from_regression_baseline, - total_query_wait_time_ms - ) EXECUTE sys.sp_executesql - @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; +END; /*End gathering plan ids*/ - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; -END; /* -'total waits' is special. It's a sum, not a max, so -we cover it above rather than here. +This gets the runtime stats for the plans we care about. +It is notably the last usage of @where_clause. */ +SELECT + @current_table = 'inserting #query_store_runtime_stats', + @sql = @isolation_level; -IF @sort_order_is_a_wait = 1 -AND @sort_order <> 'total waits' +IF @troubleshoot_performance = 1 BEGIN - SELECT - @current_table = 'inserting #plan_ids_with_total_waits', - @sql = @isolation_level; + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; + SET STATISTICS XML ON; +END; - SET STATISTICS XML ON; - END; +SELECT + @sql += N' +SELECT + @database_id, + MAX(qsrs_with_lasts.runtime_stats_id), + qsrs_with_lasts.plan_id, + MAX(qsrs_with_lasts.runtime_stats_interval_id), + MAX(qsrs_with_lasts.execution_type_desc), + MIN(qsrs_with_lasts.first_execution_time), + MAX(qsrs_with_lasts.partitioned_last_execution_time), + SUM(qsrs_with_lasts.count_executions), + AVG((qsrs_with_lasts.avg_duration / 1000.)), + MAX((qsrs_with_lasts.partitioned_last_duration / 1000.)), + MIN((qsrs_with_lasts.min_duration / 1000.)), + MAX((qsrs_with_lasts.max_duration / 1000.)), + AVG((qsrs_with_lasts.avg_cpu_time / 1000.)), + MAX((qsrs_with_lasts.partitioned_last_cpu_time / 1000.)), + MIN((qsrs_with_lasts.min_cpu_time / 1000.)), + MAX((qsrs_with_lasts.max_cpu_time / 1000.)), + AVG(((qsrs_with_lasts.avg_logical_io_reads * 8.) / 1024.)), + MAX(((qsrs_with_lasts.partitioned_last_logical_io_reads * 8.) / 1024.)), + MIN(((qsrs_with_lasts.min_logical_io_reads * 8.) / 1024.)), + MAX(((qsrs_with_lasts.max_logical_io_reads * 8.) / 1024.)), + AVG(((qsrs_with_lasts.avg_logical_io_writes * 8.) / 1024.)), + MAX(((qsrs_with_lasts.partitioned_last_logical_io_writes * 8.) / 1024.)), + MIN(((qsrs_with_lasts.min_logical_io_writes * 8.) / 1024.)), + MAX(((qsrs_with_lasts.max_logical_io_writes * 8.) / 1024.)), + AVG(((qsrs_with_lasts.avg_physical_io_reads * 8.) / 1024.)), + MAX(((qsrs_with_lasts.partitioned_last_physical_io_reads * 8.) / 1024.)), + MIN(((qsrs_with_lasts.min_physical_io_reads * 8.) / 1024.)), + MAX(((qsrs_with_lasts.max_physical_io_reads * 8.) / 1024.)), + AVG((qsrs_with_lasts.avg_clr_time / 1000.)), + MAX((qsrs_with_lasts.partitioned_last_clr_time / 1000.)), + MIN((qsrs_with_lasts.min_clr_time / 1000.)), + MAX((qsrs_with_lasts.max_clr_time / 1000.)), + MAX(qsrs_with_lasts.partitioned_last_dop), + MIN(qsrs_with_lasts.min_dop), + MAX(qsrs_with_lasts.max_dop), + AVG(((qsrs_with_lasts.avg_query_max_used_memory * 8.) / 1024.)), + MAX(((qsrs_with_lasts.partitioned_last_query_max_used_memory * 8.) / 1024.)), + MIN(((qsrs_with_lasts.min_query_max_used_memory * 8.) / 1024.)), + MAX(((qsrs_with_lasts.max_query_max_used_memory * 8.) / 1024.)), + AVG(qsrs_with_lasts.avg_rowcount), + MAX(qsrs_with_lasts.partitioned_last_rowcount), + MIN(qsrs_with_lasts.min_rowcount), + MAX(qsrs_with_lasts.max_rowcount),'; +IF @new = 1 +BEGIN SELECT @sql += N' - SELECT TOP (@top) - @database_id, - qsrs.plan_id, - from_regression_baseline = - CASE - WHEN qsrs.last_execution_time >= @start_date - AND qsrs.last_execution_time < @end_date - THEN ''No'' - ELSE ''Yes'' - END, - total_query_wait_time_ms = - MAX(qsws.total_query_wait_time_ms) - FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - JOIN ' + @database_name_quoted + N'.sys.query_store_wait_stats AS qsws - ON qsrs.plan_id = qsws.plan_id - WHERE 1 = 1 - AND qsws.wait_category = ' + - CASE @sort_order - WHEN 'cpu waits' THEN N'1' - WHEN 'lock waits' THEN N'3' - WHEN 'locks waits' THEN N'3' - WHEN 'latch waits' THEN N'4' - WHEN 'latches waits' THEN N'4' - WHEN 'buffer latch waits' THEN N'5' - WHEN 'buffer latches waits' THEN N'5' - WHEN 'buffer io waits' THEN N'6' - WHEN 'log waits' THEN N'14' - WHEN 'log io waits' THEN N'14' - WHEN 'network waits' THEN N'15' - WHEN 'network io waits' THEN N'15' - WHEN 'parallel waits' THEN N'16' - WHEN 'parallelism waits' THEN N'16' - WHEN 'memory waits' THEN N'17' - END - + N' - ' - + CASE WHEN @regression_mode = 1 - THEN N' AND ( 1 = 1 - ' + @regression_where_clause - + N' ) - OR - ( 1 = 1 - ' - + @where_clause - + N' ) ' - ELSE @where_clause - END - + N' - GROUP - BY qsrs.plan_id, - CASE - WHEN qsrs.last_execution_time >= @start_date - AND qsrs.last_execution_time < @end_date - THEN ''No'' - ELSE ''Yes'' - END - ORDER BY - MAX(qsws.total_query_wait_time_ms) DESC - OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; - - IF @regression_mode = 1 - BEGIN + AVG(((qsrs_with_lasts.avg_num_physical_io_reads * 8.) / 1024.)), + MAX(((qsrs_with_lasts.partitioned_last_num_physical_io_reads * 8.) / 1024.)), + MIN(((qsrs_with_lasts.min_num_physical_io_reads * 8.) / 1024.)), + MAX(((qsrs_with_lasts.max_num_physical_io_reads * 8.) / 1024.)), + AVG((qsrs_with_lasts.avg_log_bytes_used / 100000000.)), + MAX((qsrs_with_lasts.partitioned_last_log_bytes_used / 100000000.)), + MIN((qsrs_with_lasts.min_log_bytes_used / 100000000.)), + MAX((qsrs_with_lasts.max_log_bytes_used / 100000000.)), + AVG(((qsrs_with_lasts.avg_tempdb_space_used * 8) / 1024.)), + MAX(((qsrs_with_lasts.partitioned_last_tempdb_space_used * 8) / 1024.)), + MIN(((qsrs_with_lasts.min_tempdb_space_used * 8) / 1024.)), + MAX(((qsrs_with_lasts.max_tempdb_space_used * 8) / 1024.)),'; +END; - /* Very stupid way to stop us repeating the above code. */ - SELECT - @sql = REPLACE - ( - @sql, - 'TOP (@top)', - 'TOP (2147483647 + (0 * @top))' - ); - END; +IF @new = 0 +BEGIN + SELECT + @sql += N' + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL,'; +END; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; +/* +In regression mode, we do not mind seeing the +same plan_id twice. We need the below to make +the two time periods under consideration +distinct. +*/ +IF @regression_mode = 1 +BEGIN + SELECT + @sql += N' + CASE + WHEN qsrs_with_lasts.last_execution_time >= @start_date + AND qsrs_with_lasts.last_execution_time < @end_date + THEN ''No'' + ELSE ''Yes'' + END,'; +END; +ELSE +BEGIN + SELECT + @sql += N' + NULL,'; +END; - INSERT - #plan_ids_with_total_waits WITH(TABLOCK) - ( - database_id, - plan_id, - from_regression_baseline, - total_query_wait_time_ms - ) - EXECUTE sys.sp_executesql - @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; +SELECT + @sql += N' + context_settings = NULL +FROM +( + SELECT + qsrs.*, + /* + We need this here to make sure that PARTITION BY runs before GROUP BY but after CROSS APPLY. + If it were after GROUP BY, then we would be dealing with already aggregated data. + If it were inside the CROSS APPLY, then we would be dealing with windows of size one. + Both are very wrong, so we need this. + */ + partitioned_last_execution_time = + LAST_VALUE(qsrs.last_execution_time) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_duration = + LAST_VALUE(qsrs.last_duration) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_cpu_time = + LAST_VALUE(qsrs.last_cpu_time) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_logical_io_reads = + LAST_VALUE(qsrs.last_logical_io_reads) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_logical_io_writes = + LAST_VALUE(qsrs.last_logical_io_writes) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_physical_io_reads = + LAST_VALUE(qsrs.last_physical_io_reads) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_clr_time = + LAST_VALUE(qsrs.last_clr_time) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_dop = + LAST_VALUE(qsrs.last_dop) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_query_max_used_memory = + LAST_VALUE(qsrs.last_query_max_used_memory) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_rowcount = + LAST_VALUE(qsrs.last_rowcount) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ),'; - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; +IF @new = 1 +BEGIN + SELECT + @sql += N' + partitioned_last_num_physical_io_reads = + LAST_VALUE(qsrs.last_num_physical_io_reads) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_log_bytes_used = + LAST_VALUE(qsrs.last_log_bytes_used) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_tempdb_space_used = + LAST_VALUE(qsrs.last_tempdb_space_used) OVER + ( + PARTITION BY + qsrs.plan_id, + qsrs.execution_type + ORDER BY + qsrs.runtime_stats_interval_id DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + )'; END; -/*End populating sort-helping tables*/ -/* -This is where the bulk of the regression mode -work is done. We grab the metrics for both time -periods for each query hash, compare them, -and get the @top. -*/ -IF @regression_mode = 1 +IF @new = 0 BEGIN - /* - We begin by getting the metrics per query hash - in the time period. - */ SELECT - @current_table = 'inserting #regression_baseline_runtime_stats', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; + @sql += N' + not_used = NULL'; +END; - SET STATISTICS XML ON; - END; +SELECT + @sql += N' + FROM #distinct_plans AS dp + CROSS APPLY + ( + SELECT TOP (@queries_top) + qsrs.*'; SELECT @sql += N' + FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs'; + IF @regression_mode = 1 + BEGIN + SELECT + @sql += N' + JOIN #regression_changes AS regression + ON qsrs.plan_id = regression.plan_id + AND regression.database_id = @database_id'; + END; + ELSE IF @sort_order = 'plan count by hashes' + BEGIN + SELECT + @sql += N' + JOIN #plan_ids_with_query_hashes AS hashes + ON qsrs.plan_id = hashes.plan_id + AND hashes.database_id = @database_id'; + END; + ELSE IF @sort_order_is_a_wait = 1 + BEGIN + /* + Note that we do not need this join in + regression mode, even if we are looking + at a wait. The tables here are only for + sorting. In regression mode, we sort + by columns found in #regression_changes. + */ + SELECT + @sql += N' + JOIN #plan_ids_with_total_waits AS waits + ON qsrs.plan_id = waits.plan_id + AND waits.database_id = @database_id'; + END; + SELECT - qsq.query_hash, - /* All of these but count_executions are already floats. */ - regression_metric_average = - CONVERT(float, AVG(' + - CASE @sort_order - WHEN 'cpu' THEN N'qsrs.avg_cpu_time' - WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' - WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' - WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' - WHEN 'duration' THEN N'qsrs.avg_duration' - WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' - WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END - WHEN 'executions' THEN N'qsrs.count_executions' - WHEN 'rows' THEN N'qsrs.avg_rowcount' - ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'waits.total_query_wait_time_ms' ELSE N'qsrs.avg_cpu_time' END - END - + N')) - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - ON qsq.query_id = qsp.query_id - JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - ON qsp.plan_id = qsrs.plan_id - LEFT JOIN #plan_ids_with_total_waits AS waits - ON qsp.plan_id = waits.plan_id - AND waits.from_regression_baseline = ''Yes'' - WHERE 1 = 1 - ' + @regression_where_clause + @sql += N' + WHERE qsrs.plan_id = dp.plan_id + AND 1 = 1 + ' + + CASE + WHEN @regression_mode = 1 + THEN N' AND ( 1 = 1 + ' + + @regression_where_clause + + N' ) + OR + ( 1 = 1 + ' + @where_clause + + N' ) ' + ELSE @where_clause + END + N' - GROUP - BY qsq.query_hash - OPTION(RECOMPILE);' + @nc10; + ORDER BY + ' + + CASE @regression_mode + WHEN 1 THEN + /* As seen when populating #regression_changes. */ + CASE @regression_direction + WHEN 'regressed' THEN N'regression.change_since_regression_time_period' + WHEN 'worse' THEN N'regression.change_since_regression_time_period' + WHEN 'improved' THEN N'regression.change_since_regression_time_period * -1.0' + WHEN 'better' THEN N'regression.change_since_regression_time_period * -1.0' + WHEN 'magnitude' THEN N'ABS(regression.change_since_regression_time_period)' + WHEN 'absolute' THEN N'ABS(regression.change_since_regression_time_period)' + END + ELSE + CASE @sort_order + WHEN 'cpu' THEN N'qsrs.avg_cpu_time' + WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' + WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' + WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' + WHEN 'duration' THEN N'qsrs.avg_duration' + WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' + WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END + WHEN 'executions' THEN N'qsrs.count_executions' + WHEN 'recent' THEN N'qsrs.last_execution_time' + WHEN 'rows' THEN N'qsrs.avg_rowcount' + WHEN 'plan count by hashes' THEN N'hashes.plan_hash_count_for_query_hash DESC, + hashes.query_hash' + ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'waits.total_query_wait_time_ms' ELSE N'qsrs.avg_cpu_time' END + END + END + N' DESC + ) AS qsrs +) AS qsrs_with_lasts +GROUP BY + qsrs_with_lasts.plan_id ' + +/* +In regression mode, we do not mind seeing the +same plan_id twice. We need the below to make +the two time periods under consideration +distinct. +*/ +CASE @regression_mode + WHEN 1 THEN N', + CASE + WHEN qsrs_with_lasts.last_execution_time >= @start_date AND qsrs_with_lasts.last_execution_time < @end_date + THEN ''No'' + ELSE ''Yes'' + END' + ELSE N' ' +END ++ N' +OPTION(RECOMPILE, OPTIMIZE FOR (@queries_top = 9223372036854775807));' + @nc10; - IF @debug = 1 +IF @debug = 1 +BEGIN + PRINT LEN(@sql); + + IF LEN(@sql) > 4000 + BEGIN + SELECT + query = + ( + SELECT + [processing-instruction(_)] = + @sql + FOR XML + PATH(''), + TYPE + ); + END; + ELSE BEGIN - PRINT LEN(@sql); PRINT @sql; END; +END; + +INSERT + #query_store_runtime_stats +WITH + (TABLOCK) +( + database_id, runtime_stats_id, plan_id, runtime_stats_interval_id, execution_type_desc, + first_execution_time, last_execution_time, count_executions, + avg_duration_ms, last_duration_ms, min_duration_ms, max_duration_ms, + avg_cpu_time_ms, last_cpu_time_ms, min_cpu_time_ms, max_cpu_time_ms, + avg_logical_io_reads_mb, last_logical_io_reads_mb, min_logical_io_reads_mb, max_logical_io_reads_mb, + avg_logical_io_writes_mb, last_logical_io_writes_mb, min_logical_io_writes_mb, max_logical_io_writes_mb, + avg_physical_io_reads_mb, last_physical_io_reads_mb, min_physical_io_reads_mb, max_physical_io_reads_mb, + avg_clr_time_ms, last_clr_time_ms, min_clr_time_ms, max_clr_time_ms, + last_dop, min_dop, max_dop, + avg_query_max_used_memory_mb, last_query_max_used_memory_mb, min_query_max_used_memory_mb, max_query_max_used_memory_mb, + avg_rowcount, last_rowcount, min_rowcount, max_rowcount, + avg_num_physical_io_reads_mb, last_num_physical_io_reads_mb, min_num_physical_io_reads_mb, max_num_physical_io_reads_mb, + avg_log_bytes_used_mb, last_log_bytes_used_mb, min_log_bytes_used_mb, max_log_bytes_used_mb, + avg_tempdb_space_used_mb, last_tempdb_space_used_mb, min_tempdb_space_used_mb, max_tempdb_space_used_mb, + from_regression_baseline, + context_settings +) +EXECUTE sys.sp_executesql + @sql, + @parameters, + @top, + @start_date, + @end_date, + @execution_count, + @duration_ms, + @execution_type_desc, + @database_id, + @queries_top, + @work_start_utc, + @work_end_utc, + @regression_baseline_start_date, + @regression_baseline_end_date; + +IF @troubleshoot_performance = 1 +BEGIN + SET STATISTICS XML OFF; + + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - INSERT - #regression_baseline_runtime_stats WITH(TABLOCK) - ( - query_hash, - regression_metric_average - ) EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; + @current_table; +END; /*End getting runtime stats*/ - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; +/* +This gets the query plans we're after +*/ +SELECT + @current_table = 'inserting #query_store_plan', + @sql = @isolation_level; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; +IF @troubleshoot_performance = 1 +BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; + SET STATISTICS XML ON; +END; - /* - We now take the same measurement for all of the same query hashes, - but in the @where_clause time period. - */ +SELECT + @sql += N' +SELECT + @database_id, + qsp.plan_id, + qsp.query_id, + all_plan_ids = + STUFF + ( + ( + SELECT DISTINCT + '', '' + + RTRIM + (qsp_plans.plan_id) + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp_plans + WHERE qsp_plans.query_id = qsp.query_id + FOR XML + PATH(''''), + TYPE + ).value(''./text()[1]'', ''varchar(max)''), + 1, + 2, + '''' + ), + qsp.plan_group_id, + qsp.engine_version, + qsp.compatibility_level, + qsp.query_plan_hash, + qsp.query_plan, + qsp.is_online_index_plan, + qsp.is_trivial_plan, + qsp.is_parallel_plan, + qsp.is_forced_plan, + qsp.is_natively_compiled, + qsp.force_failure_count, + qsp.last_force_failure_reason_desc, + qsp.count_compiles, + qsp.initial_compile_start_time, + qsp.last_compile_start_time, + qsp.last_execution_time, + (qsp.avg_compile_duration / 1000.), + (qsp.last_compile_duration / 1000.),'; + +IF +( + @new = 0 + AND @sql_2022_views = 0 +) +BEGIN SELECT - @current_table = 'inserting #regression_current_runtime_stats', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; + @sql += N' + NULL, + NULL, + NULL, + NULL'; +END; +IF +( + @new = 1 + AND @sql_2022_views = 0 +) +BEGIN SELECT @sql += N' + qsp.plan_forcing_type_desc, + NULL, + NULL, + NULL'; +END; + +IF +( + @new = 1 + AND @sql_2022_views = 1 +) +BEGIN SELECT - qsq.query_hash, - /* All of these but count_executions are already floats. */ - current_metric_average = - CONVERT(float, AVG(' + - CASE @sort_order - WHEN 'cpu' THEN N'qsrs.avg_cpu_time' - WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' - WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' - WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' - WHEN 'duration' THEN N'qsrs.avg_duration' - WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' - WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END - WHEN 'executions' THEN N'qsrs.count_executions' - WHEN 'rows' THEN N'qsrs.avg_rowcount' - ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'waits.total_query_wait_time_ms' ELSE N'qsrs.avg_cpu_time' END - END - + N')) - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - ON qsq.query_id = qsp.query_id - JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - ON qsp.plan_id = qsrs.plan_id - LEFT JOIN #plan_ids_with_total_waits AS waits - ON qsp.plan_id = waits.plan_id - AND waits.from_regression_baseline = ''No'' - WHERE 1 = 1 - AND qsq.query_hash IN (SELECT base.query_hash FROM #regression_baseline_runtime_stats AS base) - ' + @where_clause - + N' - GROUP - BY qsq.query_hash - OPTION(RECOMPILE);' + @nc10; + @sql += N' + qsp.plan_forcing_type_desc, + qsp.has_compile_replay_script, + qsp.is_optimized_plan_forcing_disabled, + qsp.plan_type_desc'; +END; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; +SELECT + @sql += N' +FROM #query_store_runtime_stats AS qsrs +CROSS APPLY +( + SELECT TOP (@plans_top) + qsp.* + FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + WHERE qsp.plan_id = qsrs.plan_id + AND qsp.is_online_index_plan = 0 + ORDER BY + qsp.last_execution_time DESC +) AS qsp +WHERE qsrs.database_id = @database_id +OPTION(RECOMPILE, OPTIMIZE FOR (@plans_top = 9223372036854775807));' + @nc10; - INSERT - #regression_current_runtime_stats WITH(TABLOCK) - ( - query_hash, - current_metric_average - ) - EXECUTE sys.sp_executesql - @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; +IF @debug = 1 +BEGIN + PRINT LEN(@sql); + PRINT @sql; +END; - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; +INSERT + #query_store_plan +WITH + (TABLOCK) +( + database_id, + plan_id, + query_id, + all_plan_ids, + plan_group_id, + engine_version, + compatibility_level, + query_plan_hash, + query_plan, + is_online_index_plan, + is_trivial_plan, + is_parallel_plan, + is_forced_plan, + is_natively_compiled, + force_failure_count, + last_force_failure_reason_desc, + count_compiles, + initial_compile_start_time, + last_compile_start_time, + last_execution_time, + avg_compile_duration_ms, + last_compile_duration_ms, + plan_forcing_type_desc, + has_compile_replay_script, + is_optimized_plan_forcing_disabled, + plan_type_desc +) +EXECUTE sys.sp_executesql + @sql, + N'@plans_top bigint, + @database_id int', + @plans_top, + @database_id; - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; +IF @troubleshoot_performance = 1 +BEGIN + SET STATISTICS XML OFF; - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; - SELECT - @current_table = 'inserting #regression_changes', - @sql = @isolation_level; + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; +END; /*End getting query plans*/ - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; +/* +This gets some query information +*/ +SELECT + @current_table = 'inserting #query_store_query', + @sql = @isolation_level; - SET STATISTICS XML ON; - END; +IF @troubleshoot_performance = 1 +BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - /* - Now that we have the data from - both time periods, we must - compare them as @regression_comparator - demands and order them as - @regression_direction demands. + SET STATISTICS XML ON; +END; - However, we care about query_hashes - here despite everything after this - wanting plan_ids. This means we - must repeat some of the tricks - we used for #plan_ids_with_query_hashes. - */ - SELECT - @sql += N' - SELECT - @database_id, - plans_for_hashes.plan_id, - hashes_with_changes.query_hash, - change_since_regression_time_period = - ' + - /* - If we are returning differences that are not percentages, - then we need the units we show for any given metric to be - the same as anywhere else that gives the same metric. - If we do not, then our final output will look wrong. - For example, our CPU time will be 1,000 times bigger - here than it is in any other column. - To avoid this problem, we need to replicate the calculations - later used to populate #query_store_runtime_stats. - */ - CASE @regression_comparator - WHEN 'absolute' THEN - CASE @sort_order - WHEN 'cpu' THEN N'hashes_with_changes.change_since_regression_time_period / 1000.' - WHEN 'logical reads' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' - WHEN 'physical reads' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' - WHEN 'writes' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' - WHEN 'duration' THEN N'hashes_with_changes.change_since_regression_time_period / 1000.' - WHEN 'memory' THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' - WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'(hashes_with_changes.change_since_regression_time_period * 8.) / 1024.' ELSE N'hashes_with_changes.change_since_regression_time_period / 1000.' END - WHEN 'executions' THEN N'hashes_with_changes.change_since_regression_time_period' - WHEN 'rows' THEN N'hashes_with_changes.change_since_regression_time_period' - ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'hashes_with_changes.change_since_regression_time_period / 1000.' ELSE N'hashes_with_changes.change_since_regression_time_period / 1000.' END - END - ELSE N'hashes_with_changes.change_since_regression_time_period' END - + N' - FROM - ( - SELECT TOP (@top) - compared_stats.query_hash, - compared_stats.change_since_regression_time_period - FROM - ( - SELECT - current_stats.query_hash, - change_since_regression_time_period = - ' - + CASE @regression_comparator - WHEN 'relative' THEN N'((current_stats.current_metric_average / NULLIF(baseline.regression_metric_average, 0.0)) - 1.0)' - WHEN 'absolute' THEN N'(current_stats.current_metric_average - baseline.regression_metric_average)' - END - + N' - FROM #regression_current_runtime_stats AS current_stats - JOIN #regression_baseline_runtime_stats AS baseline - ON current_stats.query_hash = baseline.query_hash - ) AS compared_stats - ORDER BY - ' - /* - Current metrics that are better than that of the baseline period, - will give change_since_regression_time_period values that - are smaller than metrics that are worse. - In other words, ORDER BY change_since_regression_time_period DESC - gives us the regressed queries first. - This is true regardless of @regression_comparator. - To make @regression_direction behave as intended, we - need to account for this. We could use dynamic SQL, - but mathematics has given us better tools. - */ - + CASE @regression_direction - WHEN 'regressed' THEN N'change_since_regression_time_period' - WHEN 'worse' THEN N'change_since_regression_time_period' - WHEN 'improved' THEN N'change_since_regression_time_period * -1.0' - WHEN 'better' THEN N'change_since_regression_time_period * -1.0' - /* - The following two branches cannot be hit if - @regression_comparator is 'relative'. - We have made errors be thrown if somebody tries - to mix the two. - If you can figure out a way to make the two make - sense together, then feel free to add it in. - */ - WHEN 'magnitude' THEN N'ABS(change_since_regression_time_period)' - WHEN 'absolute' THEN N'ABS(change_since_regression_time_period)' - END - + N' DESC - ) AS hashes_with_changes - JOIN +SELECT + @sql += N' +SELECT + @database_id, + qsq.query_id, + qsq.query_text_id, + qsq.context_settings_id, + qsq.object_id, + qsq.batch_sql_handle, + qsq.query_hash, + qsq.is_internal_query, + qsq.query_parameterization_type_desc, + qsq.initial_compile_start_time, + qsq.last_compile_start_time, + qsq.last_execution_time, + qsq.last_compile_batch_sql_handle, + qsq.last_compile_batch_offset_start, + qsq.last_compile_batch_offset_end, + qsq.count_compiles, + (qsq.avg_compile_duration / 1000.), + (qsq.last_compile_duration / 1000.), + (qsq.avg_bind_duration / 1000.), + (qsq.last_bind_duration / 1000.), + (qsq.avg_bind_cpu_time / 1000.), + (qsq.last_bind_cpu_time / 1000.), + (qsq.avg_optimize_duration / 1000.), + (qsq.last_optimize_duration / 1000.), + (qsq.avg_optimize_cpu_time / 1000.), + (qsq.last_optimize_cpu_time / 1000.), + ((qsq.avg_compile_memory_kb * 8) / 1024.), + ((qsq.last_compile_memory_kb * 8) / 1024.), + ((qsq.max_compile_memory_kb * 8) / 1024.), + qsq.is_clouddb_internal_query +FROM #query_store_plan AS qsp +CROSS APPLY +( + SELECT TOP (1) + qsq.* + FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq + WHERE qsq.query_id = qsp.query_id + ORDER BY + qsq.last_execution_time DESC +) AS qsq +WHERE qsp.database_id = @database_id +OPTION(RECOMPILE);' + @nc10; + +IF @debug = 1 +BEGIN + PRINT LEN(@sql); + PRINT @sql; +END; + +INSERT + #query_store_query +WITH + (TABLOCK) +( + database_id, + query_id, + query_text_id, + context_settings_id, + object_id, + batch_sql_handle, + query_hash, + is_internal_query, + query_parameterization_type_desc, + initial_compile_start_time, + last_compile_start_time, + last_execution_time, + last_compile_batch_sql_handle, + last_compile_batch_offset_start, + last_compile_batch_offset_end, + count_compiles, + avg_compile_duration_ms, + last_compile_duration_ms, + avg_bind_duration_ms, + last_bind_duration_ms, + avg_bind_cpu_time_ms, + last_bind_cpu_time_ms, + avg_optimize_duration_ms, + last_optimize_duration_ms, + avg_optimize_cpu_time_ms, + last_optimize_cpu_time_ms, + avg_compile_memory_mb, + last_compile_memory_mb, + max_compile_memory_mb, + is_clouddb_internal_query +) +EXECUTE sys.sp_executesql + @sql, + N'@database_id int', + @database_id; + +IF @troubleshoot_performance = 1 +BEGIN + SET STATISTICS XML OFF; + + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; + + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; +END; /*End getting query details*/ + + +IF @include_query_hash_totals = 1 +BEGIN + SELECT + @current_table = 'inserting #query_hash_totals for @include_query_hash_totals', + @sql = @isolation_level; + + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; + + SET STATISTICS XML ON; + END; + + SELECT + @sql += N' + SELECT + @database_id, + qsq.query_hash, + SUM(qsrs.count_executions), + SUM(qsrs.count_executions * qsrs.avg_duration) / 1000., + SUM(qsrs.count_executions * qsrs.avg_cpu_time) / 1000., + SUM(qsrs.count_executions * (qsrs.avg_logical_io_reads * 8.)) / 1024., + SUM(qsrs.count_executions * (qsrs.avg_physical_io_reads * 8.)) / 1024., + SUM(qsrs.count_executions * (qsrs.avg_logical_io_writes * 8.)) / 1024., + SUM(qsrs.count_executions * qsrs.avg_clr_time) / 1000., + SUM(qsrs.count_executions * (qsrs.avg_query_max_used_memory * 8.)) / 1024., + SUM(qsrs.count_executions * qsrs.avg_rowcount) + FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs + JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp + ON qsrs.plan_id = qsp.plan_id + JOIN ' + @database_name_quoted + N'.sys.query_store_query AS qsq + ON qsp.query_id = qsq.query_id + WHERE EXISTS ( - SELECT DISTINCT - qsq.query_hash, - qsp.plan_id - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - JOIN ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - ON qsq.query_id = qsp.query_id - JOIN ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - ON qsp.plan_id = qsrs.plan_id - WHERE - ( 1 = 1 - ' - /* We want each time period's plan_ids for these query hashes. */ - + @regression_where_clause - + N' - ) - OR - ( 1 = 1 - ' - + @where_clause - + N' - ) - ) AS plans_for_hashes - ON hashes_with_changes.query_hash = plans_for_hashes.query_hash - OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; + SELECT + 1/0 + FROM #query_store_query AS qsq2 + WHERE qsq2.query_hash = qsq.query_hash + ) + GROUP BY + qsq.query_hash + OPTION(RECOMPILE); +'; IF @debug = 1 BEGIN @@ -5905,39 +7020,37 @@ BEGIN PRINT @sql; END; - INSERT - #regression_changes WITH(TABLOCK) + INSERT INTO + #query_hash_totals + WITH + (TABLOCK) ( database_id, - plan_id, query_hash, - change_since_regression_time_period + total_executions, + total_duration_ms, + total_cpu_time_ms, + total_logical_reads_mb, + total_physical_reads_mb, + total_logical_writes_mb, + total_clr_time_ms, + total_memory_mb, + total_rowcount ) EXECUTE sys.sp_executesql @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; - + N'@database_id int', + @database_id; + IF @troubleshoot_performance = 1 BEGIN SET STATISTICS XML OFF; - + EXECUTE sys.sp_executesql @troubleshoot_update, N'@current_table nvarchar(100)', @current_table; - + EXECUTE sys.sp_executesql @troubleshoot_info, N'@sql nvarchar(max), @@ -5945,19 +7058,14 @@ BEGIN @sql, @current_table; END; - END; -/* -This gets the plan_ids we care about. -We unfortunately need an ELSE IF chain here -because the final branch contains defaults -that we only want to hit if we did not hit -any others. +/* +This gets the query text for them! */ SELECT - @current_table = 'inserting #distinct_plans', + @current_table = 'inserting #query_store_query_text', @sql = @isolation_level; IF @troubleshoot_performance = 1 @@ -5970,66 +7078,40 @@ BEGIN SET STATISTICS XML ON; END; -IF @regression_mode = 1 -BEGIN - SELECT - @sql += N' - SELECT DISTINCT - plan_id - FROM #regression_changes - WHERE database_id = @database_id - OPTION(RECOMPILE);' + @nc10; -END -ELSE IF @sort_order = 'plan count by hashes' -BEGIN - SELECT - @sql += N' - SELECT DISTINCT - plan_id - FROM #plan_ids_with_query_hashes - WHERE database_id = @database_id - OPTION(RECOMPILE);' + @nc10; -END -ELSE IF @sort_order_is_a_wait = 1 -BEGIN - SELECT - @sql += N' - SELECT DISTINCT - plan_id - FROM #plan_ids_with_total_waits - WHERE database_id = @database_id - OPTION(RECOMPILE);' + @nc10; -END -ELSE -BEGIN - SELECT - @sql += N' - SELECT TOP (@top) - qsrs.plan_id - FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs - WHERE 1 = 1 - ' + @where_clause - + N' - GROUP - BY qsrs.plan_id - ORDER BY - MAX(' + - CASE @sort_order - WHEN 'cpu' THEN N'qsrs.avg_cpu_time' - WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' - WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' - WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' - WHEN 'duration' THEN N'qsrs.avg_duration' - WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' - WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END - WHEN 'executions' THEN N'qsrs.count_executions' - WHEN 'recent' THEN N'qsrs.last_execution_time' - WHEN 'rows' THEN N'qsrs.avg_rowcount' - ELSE N'qsrs.avg_cpu_time' - END + - N') DESC - OPTION(RECOMPILE, OPTIMIZE FOR (@top = 9223372036854775807));' + @nc10; -END; + +SELECT + @sql += N' +SELECT + @database_id, + qsqt.query_text_id, + query_sql_text = + ( + SELECT + [processing-instruction(query)] = + REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( + REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( + REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( + qsqt.query_sql_text COLLATE Latin1_General_BIN2, + NCHAR(31),N''?''),NCHAR(30),N''?''),NCHAR(29),N''?''),NCHAR(28),N''?''),NCHAR(27),N''?''),NCHAR(26),N''?''),NCHAR(25),N''?''),NCHAR(24),N''?''),NCHAR(23),N''?''),NCHAR(22),N''?''), + NCHAR(21),N''?''),NCHAR(20),N''?''),NCHAR(19),N''?''),NCHAR(18),N''?''),NCHAR(17),N''?''),NCHAR(16),N''?''),NCHAR(15),N''?''),NCHAR(14),N''?''),NCHAR(12),N''?''), + NCHAR(11),N''?''),NCHAR(8),N''?''),NCHAR(7),N''?''),NCHAR(6),N''?''),NCHAR(5),N''?''),NCHAR(4),N''?''),NCHAR(3),N''?''),NCHAR(2),N''?''),NCHAR(1),N''?''),NCHAR(0),N'''') + FOR XML + PATH(''''), + TYPE + ), + qsqt.statement_sql_handle, + qsqt.is_part_of_encrypted_module, + qsqt.has_restricted_text +FROM #query_store_query AS qsq +CROSS APPLY +( + SELECT TOP (1) + qsqt.* + FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt + WHERE qsqt.query_text_id = qsq.query_text_id +) AS qsqt +WHERE qsq.database_id = @database_id +OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN @@ -6038,25 +7120,21 @@ BEGIN END; INSERT - #distinct_plans WITH(TABLOCK) + #query_store_query_text +WITH + (TABLOCK) ( - plan_id + database_id, + query_text_id, + query_sql_text, + statement_sql_handle, + is_part_of_encrypted_module, + has_restricted_text ) EXECUTE sys.sp_executesql @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; + N'@database_id int', + @database_id; IF @troubleshoot_performance = 1 BEGIN @@ -6073,15 +7151,15 @@ BEGIN @current_table nvarchar(100)', @sql, @current_table; -END; /*End gathering plan ids*/ +END; /*End getting query text*/ /* -This gets the runtime stats for the plans we care about. -It is notably the last usage of @where_clause. +Here we try to get some data from the "plan cache" +that isn't available in Query Store :( */ SELECT - @current_table = 'inserting #query_store_runtime_stats', - @sql = @isolation_level; + @sql = N'', + @current_table = 'inserting #dm_exec_query_stats'; IF @troubleshoot_performance = 1 BEGIN @@ -6093,440 +7171,338 @@ BEGIN SET STATISTICS XML ON; END; +INSERT + #dm_exec_query_stats +WITH + (TABLOCK) +( + statement_sql_handle, + total_grant_mb, + last_grant_mb, + min_grant_mb, + max_grant_mb, + total_used_grant_mb, + last_used_grant_mb, + min_used_grant_mb, + max_used_grant_mb, + total_ideal_grant_mb, + last_ideal_grant_mb, + min_ideal_grant_mb, + max_ideal_grant_mb, + total_reserved_threads, + last_reserved_threads, + min_reserved_threads, + max_reserved_threads, + total_used_threads, + last_used_threads, + min_used_threads, + max_used_threads +) SELECT - @sql += N' -SELECT - @database_id, - MAX(qsrs_with_lasts.runtime_stats_id), - qsrs_with_lasts.plan_id, - MAX(qsrs_with_lasts.runtime_stats_interval_id), - MAX(qsrs_with_lasts.execution_type_desc), - MIN(qsrs_with_lasts.first_execution_time), - MAX(qsrs_with_lasts.partitioned_last_execution_time), - SUM(qsrs_with_lasts.count_executions), - AVG((qsrs_with_lasts.avg_duration / 1000.)), - MAX((qsrs_with_lasts.partitioned_last_duration / 1000.)), - MIN((qsrs_with_lasts.min_duration / 1000.)), - MAX((qsrs_with_lasts.max_duration / 1000.)), - AVG((qsrs_with_lasts.avg_cpu_time / 1000.)), - MAX((qsrs_with_lasts.partitioned_last_cpu_time / 1000.)), - MIN((qsrs_with_lasts.min_cpu_time / 1000.)), - MAX((qsrs_with_lasts.max_cpu_time / 1000.)), - AVG(((qsrs_with_lasts.avg_logical_io_reads * 8.) / 1024.)), - MAX(((qsrs_with_lasts.partitioned_last_logical_io_reads * 8.) / 1024.)), - MIN(((qsrs_with_lasts.min_logical_io_reads * 8.) / 1024.)), - MAX(((qsrs_with_lasts.max_logical_io_reads * 8.) / 1024.)), - AVG(((qsrs_with_lasts.avg_logical_io_writes * 8.) / 1024.)), - MAX(((qsrs_with_lasts.partitioned_last_logical_io_writes * 8.) / 1024.)), - MIN(((qsrs_with_lasts.min_logical_io_writes * 8.) / 1024.)), - MAX(((qsrs_with_lasts.max_logical_io_writes * 8.) / 1024.)), - AVG(((qsrs_with_lasts.avg_physical_io_reads * 8.) / 1024.)), - MAX(((qsrs_with_lasts.partitioned_last_physical_io_reads * 8.) / 1024.)), - MIN(((qsrs_with_lasts.min_physical_io_reads * 8.) / 1024.)), - MAX(((qsrs_with_lasts.max_physical_io_reads * 8.) / 1024.)), - AVG((qsrs_with_lasts.avg_clr_time / 1000.)), - MAX((qsrs_with_lasts.partitioned_last_clr_time / 1000.)), - MIN((qsrs_with_lasts.min_clr_time / 1000.)), - MAX((qsrs_with_lasts.max_clr_time / 1000.)), - MAX(qsrs_with_lasts.partitioned_last_dop), - MIN(qsrs_with_lasts.min_dop), - MAX(qsrs_with_lasts.max_dop), - AVG(((qsrs_with_lasts.avg_query_max_used_memory * 8.) / 1024.)), - MAX(((qsrs_with_lasts.partitioned_last_query_max_used_memory * 8.) / 1024.)), - MIN(((qsrs_with_lasts.min_query_max_used_memory * 8.) / 1024.)), - MAX(((qsrs_with_lasts.max_query_max_used_memory * 8.) / 1024.)), - AVG(qsrs_with_lasts.avg_rowcount), - MAX(qsrs_with_lasts.partitioned_last_rowcount), - MIN(qsrs_with_lasts.min_rowcount), - MAX(qsrs_with_lasts.max_rowcount),'; - -IF @new = 1 -BEGIN - SELECT - @sql += N' - AVG(((qsrs_with_lasts.avg_num_physical_io_reads * 8.) / 1024.)), - MAX(((qsrs_with_lasts.partitioned_last_num_physical_io_reads * 8.) / 1024.)), - MIN(((qsrs_with_lasts.min_num_physical_io_reads * 8.) / 1024.)), - MAX(((qsrs_with_lasts.max_num_physical_io_reads * 8.) / 1024.)), - AVG((qsrs_with_lasts.avg_log_bytes_used / 100000000.)), - MAX((qsrs_with_lasts.partitioned_last_log_bytes_used / 100000000.)), - MIN((qsrs_with_lasts.min_log_bytes_used / 100000000.)), - MAX((qsrs_with_lasts.max_log_bytes_used / 100000000.)), - AVG(((qsrs_with_lasts.avg_tempdb_space_used * 8) / 1024.)), - MAX(((qsrs_with_lasts.partitioned_last_tempdb_space_used * 8) / 1024.)), - MIN(((qsrs_with_lasts.min_tempdb_space_used * 8) / 1024.)), - MAX(((qsrs_with_lasts.max_tempdb_space_used * 8) / 1024.)),'; -END; - -IF @new = 0 -BEGIN - SELECT - @sql += N' - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL,'; -END; - -/* -In regression mode, we do not mind seeing the -same plan_id twice. We need the below to make -the two time periods under consideration -distinct. -*/ -IF @regression_mode = 1 -BEGIN - SELECT - @sql += N' - CASE - WHEN qsrs_with_lasts.last_execution_time >= @start_date - AND qsrs_with_lasts.last_execution_time < @end_date - THEN ''No'' - ELSE ''Yes'' - END,'; -END -ELSE -BEGIN - SELECT - @sql += N' - NULL,'; -END; - -SELECT - @sql += N' - context_settings = NULL + deqs_with_lasts.statement_sql_handle, + MAX(deqs_with_lasts.total_grant_kb) / 1024., + MAX(deqs_with_lasts.partitioned_last_grant_kb) / 1024., + MAX(deqs_with_lasts.min_grant_kb) / 1024., + MAX(deqs_with_lasts.max_grant_kb) / 1024., + MAX(deqs_with_lasts.total_used_grant_kb) / 1024., + MAX(deqs_with_lasts.partitioned_last_used_grant_kb) / 1024., + MAX(deqs_with_lasts.min_used_grant_kb) / 1024., + MAX(deqs_with_lasts.max_used_grant_kb) / 1024., + MAX(deqs_with_lasts.total_ideal_grant_kb) / 1024., + MAX(deqs_with_lasts.partitioned_last_ideal_grant_kb) / 1024., + MAX(deqs_with_lasts.min_ideal_grant_kb) / 1024., + MAX(deqs_with_lasts.max_ideal_grant_kb) / 1024., + MAX(deqs_with_lasts.total_reserved_threads), + MAX(deqs_with_lasts.partitioned_last_reserved_threads), + MAX(deqs_with_lasts.min_reserved_threads), + MAX(deqs_with_lasts.max_reserved_threads), + MAX(deqs_with_lasts.total_used_threads), + MAX(deqs_with_lasts.partitioned_last_used_threads), + MAX(deqs_with_lasts.min_used_threads), + MAX(deqs_with_lasts.max_used_threads) FROM ( SELECT - qsrs.*, - /* - We need this here to make sure that PARTITION BY runs before GROUP BY but after CROSS APPLY. - If it were after GROUP BY, then we would be dealing with already aggregated data. - If it were inside the CROSS APPLY, then we would be dealing with windows of size one. - Both are very wrong, so we need this. - */ - partitioned_last_execution_time = - LAST_VALUE(qsrs.last_execution_time) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_duration = - LAST_VALUE(qsrs.last_duration) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_cpu_time = - LAST_VALUE(qsrs.last_cpu_time) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_logical_io_reads = - LAST_VALUE(qsrs.last_logical_io_reads) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_logical_io_writes = - LAST_VALUE(qsrs.last_logical_io_writes) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_physical_io_reads = - LAST_VALUE(qsrs.last_physical_io_reads) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_clr_time = - LAST_VALUE(qsrs.last_clr_time) OVER + deqs.statement_sql_handle, + deqs.total_grant_kb, + deqs.min_grant_kb, + deqs.max_grant_kb, + deqs.total_used_grant_kb, + deqs.min_used_grant_kb, + deqs.max_used_grant_kb, + deqs.total_ideal_grant_kb, + deqs.min_ideal_grant_kb, + deqs.max_ideal_grant_kb, + deqs.total_reserved_threads, + deqs.min_reserved_threads, + deqs.max_reserved_threads, + deqs.total_used_threads, + deqs.min_used_threads, + deqs.max_used_threads, + partitioned_last_grant_kb = + LAST_VALUE(deqs.last_grant_kb) OVER ( PARTITION BY - qsrs.plan_id, - qsrs.execution_type + deqs.sql_handle ORDER BY - qsrs.runtime_stats_interval_id DESC + deqs.last_execution_time DESC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ), - partitioned_last_dop = - LAST_VALUE(qsrs.last_dop) OVER + partitioned_last_used_grant_kb = + LAST_VALUE(deqs.last_used_grant_kb) OVER ( PARTITION BY - qsrs.plan_id, - qsrs.execution_type + deqs.sql_handle ORDER BY - qsrs.runtime_stats_interval_id DESC + deqs.last_execution_time DESC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ), - partitioned_last_query_max_used_memory = - LAST_VALUE(qsrs.last_query_max_used_memory) OVER + partitioned_last_ideal_grant_kb = + LAST_VALUE(deqs.last_ideal_grant_kb) OVER ( PARTITION BY - qsrs.plan_id, - qsrs.execution_type + deqs.sql_handle ORDER BY - qsrs.runtime_stats_interval_id DESC + deqs.last_execution_time DESC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ), - partitioned_last_rowcount = - LAST_VALUE(qsrs.last_rowcount) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ),'; + partitioned_last_reserved_threads = + LAST_VALUE(deqs.last_reserved_threads) OVER + ( + PARTITION BY + deqs.sql_handle + ORDER BY + deqs.last_execution_time DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ), + partitioned_last_used_threads = + LAST_VALUE(deqs.last_used_threads) OVER + ( + PARTITION BY + deqs.sql_handle + ORDER BY + deqs.last_execution_time DESC + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ) + FROM sys.dm_exec_query_stats AS deqs + WHERE EXISTS + ( + SELECT + 1/0 + FROM #query_store_query_text AS qsqt + WHERE qsqt.statement_sql_handle = deqs.statement_sql_handle + ) +) AS deqs_with_lasts +GROUP BY + deqs_with_lasts.statement_sql_handle +OPTION(RECOMPILE); -IF @new = 1 +SELECT + @rc = ROWCOUNT_BIG(); + +IF @troubleshoot_performance = 1 BEGIN - SELECT - @sql += N' - partitioned_last_num_physical_io_reads = - LAST_VALUE(qsrs.last_num_physical_io_reads) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_log_bytes_used = - LAST_VALUE(qsrs.last_log_bytes_used) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_tempdb_space_used = - LAST_VALUE(qsrs.last_tempdb_space_used) OVER - ( - PARTITION BY - qsrs.plan_id, - qsrs.execution_type - ORDER BY - qsrs.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - )'; -END + SET STATISTICS XML OFF; -IF @new = 0 + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; + + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; +END; /*End getting runtime stats*/ + +/*Only update if we got anything*/ +IF @rc > 0 BEGIN SELECT - @sql += N' - not_used = NULL' -END + @current_table = 'updating #dm_exec_query_stats'; -SELECT - @sql += N' - FROM #distinct_plans AS dp - CROSS APPLY - ( - SELECT TOP (@queries_top) - qsrs.*' + IF @troubleshoot_performance = 1 + BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; - SELECT - @sql += N' - FROM ' + @database_name_quoted + N'.sys.query_store_runtime_stats AS qsrs' - IF @regression_mode = 1 - BEGIN - SELECT - @sql += N' - JOIN #regression_changes AS regression - ON qsrs.plan_id = regression.plan_id - AND regression.database_id = @database_id' - END - ELSE IF @sort_order = 'plan count by hashes' - BEGIN - SELECT - @sql += N' - JOIN #plan_ids_with_query_hashes AS hashes - ON qsrs.plan_id = hashes.plan_id - AND hashes.database_id = @database_id' - END - ELSE IF @sort_order_is_a_wait = 1 - BEGIN - /* - Note that we do not need this join in - regression mode, even if we are looking - at a wait. The tables here are only for - sorting. In regression mode, we sort - by columns found in #regression_changes. - */ - SELECT - @sql += N' - JOIN #plan_ids_with_total_waits AS waits - ON qsrs.plan_id = waits.plan_id - AND waits.database_id = @database_id' - END; + SET STATISTICS XML ON; + END; + + UPDATE + qsqt + SET + qsqt.total_grant_mb = deqs.total_grant_mb, + qsqt.last_grant_mb = deqs.last_grant_mb, + qsqt.min_grant_mb = deqs.min_grant_mb, + qsqt.max_grant_mb = deqs.max_grant_mb, + qsqt.total_used_grant_mb = deqs.total_used_grant_mb, + qsqt.last_used_grant_mb = deqs.last_used_grant_mb, + qsqt.min_used_grant_mb = deqs.min_used_grant_mb, + qsqt.max_used_grant_mb = deqs.max_used_grant_mb, + qsqt.total_ideal_grant_mb = deqs.total_ideal_grant_mb, + qsqt.last_ideal_grant_mb = deqs.last_ideal_grant_mb, + qsqt.min_ideal_grant_mb = deqs.min_ideal_grant_mb, + qsqt.max_ideal_grant_mb = deqs.max_ideal_grant_mb, + qsqt.total_reserved_threads = deqs.total_reserved_threads, + qsqt.last_reserved_threads = deqs.last_reserved_threads, + qsqt.min_reserved_threads = deqs.min_reserved_threads, + qsqt.max_reserved_threads = deqs.max_reserved_threads, + qsqt.total_used_threads = deqs.total_used_threads, + qsqt.last_used_threads = deqs.last_used_threads, + qsqt.min_used_threads = deqs.min_used_threads, + qsqt.max_used_threads = deqs.max_used_threads + FROM #query_store_query_text AS qsqt + JOIN #dm_exec_query_stats AS deqs + ON qsqt.statement_sql_handle = deqs.statement_sql_handle + OPTION(RECOMPILE); + + IF @troubleshoot_performance = 1 + BEGIN + SET STATISTICS XML OFF; + + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; + + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; + END; +END; /*End updating runtime stats*/ - SELECT - @sql += N' - WHERE qsrs.plan_id = dp.plan_id - AND 1 = 1 - ' - + CASE WHEN @regression_mode = 1 - THEN N' AND ( 1 = 1 - ' + @regression_where_clause - + N' ) - OR - ( 1 = 1 - ' - + @where_clause - + N' ) ' - ELSE @where_clause - END - + N' - ORDER BY - ' + - CASE @regression_mode - WHEN 1 THEN - /* As seen when populating #regression_changes. */ - CASE @regression_direction - WHEN 'regressed' THEN N'regression.change_since_regression_time_period' - WHEN 'worse' THEN N'regression.change_since_regression_time_period' - WHEN 'improved' THEN N'regression.change_since_regression_time_period * -1.0' - WHEN 'better' THEN N'regression.change_since_regression_time_period * -1.0' - WHEN 'magnitude' THEN N'ABS(regression.change_since_regression_time_period)' - WHEN 'absolute' THEN N'ABS(regression.change_since_regression_time_period)' - END - ELSE - CASE @sort_order - WHEN 'cpu' THEN N'qsrs.avg_cpu_time' - WHEN 'logical reads' THEN N'qsrs.avg_logical_io_reads' - WHEN 'physical reads' THEN N'qsrs.avg_physical_io_reads' - WHEN 'writes' THEN N'qsrs.avg_logical_io_writes' - WHEN 'duration' THEN N'qsrs.avg_duration' - WHEN 'memory' THEN N'qsrs.avg_query_max_used_memory' - WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'qsrs.avg_tempdb_space_used' ELSE N'qsrs.avg_cpu_time' END - WHEN 'executions' THEN N'qsrs.count_executions' - WHEN 'recent' THEN N'qsrs.last_execution_time' - WHEN 'rows' THEN N'qsrs.avg_rowcount' - WHEN 'plan count by hashes' THEN N'hashes.plan_hash_count_for_query_hash DESC, - hashes.query_hash' - ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'waits.total_query_wait_time_ms' ELSE N'qsrs.avg_cpu_time' END - END - END + N' DESC - ) AS qsrs -) AS qsrs_with_lasts -GROUP BY - qsrs_with_lasts.plan_id ' + /* -In regression mode, we do not mind seeing the -same plan_id twice. We need the below to make -the two time periods under consideration -distinct. +Let's check on settings, etc. +We do this first so we can see if wait stats capture mode is true more easily */ -CASE @regression_mode - WHEN 1 THEN N', - CASE - WHEN qsrs_with_lasts.last_execution_time >= @start_date AND qsrs_with_lasts.last_execution_time < @end_date - THEN ''No'' - ELSE ''Yes'' - END' - ELSE N' ' -END -+ N' -OPTION(RECOMPILE, OPTIMIZE FOR (@queries_top = 9223372036854775807));' + @nc10; +SELECT + @current_table = 'inserting #database_query_store_options', + @sql = @isolation_level; + +IF @troubleshoot_performance = 1 +BEGIN + EXECUTE sys.sp_executesql + @troubleshoot_insert, + N'@current_table nvarchar(100)', + @current_table; + + SET STATISTICS XML ON; +END; + +SELECT + @sql += N' +SELECT + @database_id, + dqso.desired_state_desc, + dqso.actual_state_desc, + readonly_reason = + CASE dqso.readonly_reason + WHEN 0 + THEN ''None'' + WHEN 2 + THEN ''Database in single user mode'' + WHEN 4 + THEN ''Database is in emergency mode'' + WHEN 8 + THEN ''Database is AG secondary'' + WHEN 65536 + THEN ''Reached max size: '' + + FORMAT(dqso.current_storage_size_mb, ''N0'') + + '' of '' + + FORMAT(dqso.max_storage_size_mb, ''N0'') + + ''.'' + WHEN 131072 + THEN ''The number of different statements in Query Store has reached the internal memory limit'' + WHEN 262144 + THEN ''Size of in-memory items waiting to be persisted on disk has reached the internal memory limit'' + WHEN 524288 + THEN ''Database has reached disk size limit'' + ELSE ''WOAH'' + END, + dqso.current_storage_size_mb, + dqso.flush_interval_seconds, + dqso.interval_length_minutes, + dqso.max_storage_size_mb, + dqso.stale_query_threshold_days, + dqso.max_plans_per_query, + dqso.query_capture_mode_desc,' + + + CASE + WHEN + ( + @product_version > 14 + OR @azure = 1 + ) + THEN N' + dqso.capture_policy_execution_count, + dqso.capture_policy_total_compile_cpu_time_ms, + dqso.capture_policy_total_execution_cpu_time_ms, + dqso.capture_policy_stale_threshold_hours,' + ELSE N' + NULL, + NULL, + NULL, + NULL,' + END + + N' + dqso.size_based_cleanup_mode_desc,' + + + CASE + WHEN + ( + @product_version = 13 + AND @azure = 0 + ) + THEN N' + NULL' + ELSE N' + dqso.wait_stats_capture_mode_desc' + END + + N' +FROM ' + @database_name_quoted + N'.sys.database_query_store_options AS dqso +OPTION(RECOMPILE);' + @nc10; IF @debug = 1 BEGIN PRINT LEN(@sql); - - IF LEN(@sql) > 7999 - BEGIN - SELECT - query = - ( - SELECT - [processing-instruction(_)] = - @sql - FOR XML - PATH(''), - TYPE - ); - END; - ELSE - BEGIN - PRINT @sql; - END; + PRINT @sql; END; INSERT - #query_store_runtime_stats WITH(TABLOCK) + #database_query_store_options +WITH + (TABLOCK) ( - database_id, runtime_stats_id, plan_id, runtime_stats_interval_id, execution_type_desc, - first_execution_time, last_execution_time, count_executions, - avg_duration_ms, last_duration_ms, min_duration_ms, max_duration_ms, - avg_cpu_time_ms, last_cpu_time_ms, min_cpu_time_ms, max_cpu_time_ms, - avg_logical_io_reads_mb, last_logical_io_reads_mb, min_logical_io_reads_mb, max_logical_io_reads_mb, - avg_logical_io_writes_mb, last_logical_io_writes_mb, min_logical_io_writes_mb, max_logical_io_writes_mb, - avg_physical_io_reads_mb, last_physical_io_reads_mb, min_physical_io_reads_mb, max_physical_io_reads_mb, - avg_clr_time_ms, last_clr_time_ms, min_clr_time_ms, max_clr_time_ms, - last_dop, min_dop, max_dop, - avg_query_max_used_memory_mb, last_query_max_used_memory_mb, min_query_max_used_memory_mb, max_query_max_used_memory_mb, - avg_rowcount, last_rowcount, min_rowcount, max_rowcount, - avg_num_physical_io_reads_mb, last_num_physical_io_reads_mb, min_num_physical_io_reads_mb, max_num_physical_io_reads_mb, - avg_log_bytes_used_mb, last_log_bytes_used_mb, min_log_bytes_used_mb, max_log_bytes_used_mb, - avg_tempdb_space_used_mb, last_tempdb_space_used_mb, min_tempdb_space_used_mb, max_tempdb_space_used_mb, - from_regression_baseline, - context_settings + database_id, + desired_state_desc, + actual_state_desc, + readonly_reason, + current_storage_size_mb, + flush_interval_seconds, + interval_length_minutes, + max_storage_size_mb, + stale_query_threshold_days, + max_plans_per_query, + query_capture_mode_desc, + capture_policy_execution_count, + capture_policy_total_compile_cpu_time_ms, + capture_policy_total_execution_cpu_time_ms, + capture_policy_stale_threshold_hours, + size_based_cleanup_mode_desc, + wait_stats_capture_mode_desc ) EXECUTE sys.sp_executesql @sql, - @parameters, - @top, - @start_date, - @end_date, - @execution_count, - @duration_ms, - @execution_type_desc, - @database_id, - @queries_top, - @work_start_utc, - @work_end_utc, - @regression_baseline_start_date, - @regression_baseline_end_date; + N'@database_id int', + @database_id; IF @troubleshoot_performance = 1 BEGIN @@ -6543,3893 +7519,1179 @@ BEGIN @current_table nvarchar(100)', @sql, @current_table; -END; /*End getting runtime stats*/ +END; /*End getting query store settings*/ /* -This gets the query plans we're after +If wait stats are available, we'll grab them here */ -SELECT - @current_table = 'inserting #query_store_plan', - @sql = @isolation_level; - -IF @troubleshoot_performance = 1 -BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; -END; - -SELECT - @sql += N' -SELECT - @database_id, - qsp.plan_id, - qsp.query_id, - all_plan_ids = - STUFF - ( - ( - SELECT DISTINCT - '', '' + - RTRIM - (qsp_plans.plan_id) - FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp_plans - WHERE qsp_plans.query_id = qsp.query_id - FOR XML - PATH(''''), - TYPE - ).value(''./text()[1]'', ''varchar(max)''), - 1, - 2, - '''' - ), - qsp.plan_group_id, - qsp.engine_version, - qsp.compatibility_level, - qsp.query_plan_hash, - qsp.query_plan, - qsp.is_online_index_plan, - qsp.is_trivial_plan, - qsp.is_parallel_plan, - qsp.is_forced_plan, - qsp.is_natively_compiled, - qsp.force_failure_count, - qsp.last_force_failure_reason_desc, - qsp.count_compiles, - qsp.initial_compile_start_time, - qsp.last_compile_start_time, - qsp.last_execution_time, - (qsp.avg_compile_duration / 1000.), - (qsp.last_compile_duration / 1000.),'; - -IF -( - @new = 0 - AND @sql_2022_views = 0 -) -BEGIN - SELECT - @sql += N' - NULL, - NULL, - NULL, - NULL'; -END; - -IF -( - @new = 1 - AND @sql_2022_views = 0 -) -BEGIN - SELECT - @sql += N' - qsp.plan_forcing_type_desc, - NULL, - NULL, - NULL'; -END; - IF -( - @new = 1 - AND @sql_2022_views = 1 -) -BEGIN - SELECT - @sql += N' - qsp.plan_forcing_type_desc, - qsp.has_compile_replay_script, - qsp.is_optimized_plan_forcing_disabled, - qsp.plan_type_desc'; -END; - -SELECT - @sql += N' -FROM #query_store_runtime_stats AS qsrs -CROSS APPLY -( - SELECT TOP (@plans_top) - qsp.* - FROM ' + @database_name_quoted + N'.sys.query_store_plan AS qsp - WHERE qsp.plan_id = qsrs.plan_id - AND qsp.is_online_index_plan = 0 - ORDER BY - qsp.last_execution_time DESC -) AS qsp -WHERE qsrs.database_id = @database_id -OPTION(RECOMPILE, OPTIMIZE FOR (@plans_top = 9223372036854775807));' + @nc10; - -IF @debug = 1 -BEGIN - PRINT LEN(@sql); - PRINT @sql; -END; - -INSERT - #query_store_plan WITH(TABLOCK) -( - database_id, - plan_id, - query_id, - all_plan_ids, - plan_group_id, - engine_version, - compatibility_level, - query_plan_hash, - query_plan, - is_online_index_plan, - is_trivial_plan, - is_parallel_plan, - is_forced_plan, - is_natively_compiled, - force_failure_count, - last_force_failure_reason_desc, - count_compiles, - initial_compile_start_time, - last_compile_start_time, - last_execution_time, - avg_compile_duration_ms, - last_compile_duration_ms, - plan_forcing_type_desc, - has_compile_replay_script, - is_optimized_plan_forcing_disabled, - plan_type_desc -) -EXECUTE sys.sp_executesql - @sql, - N'@plans_top bigint, - @database_id int', - @plans_top, - @database_id; - -IF @troubleshoot_performance = 1 -BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; -END; /*End getting query plans*/ - -/* -This gets some query information -*/ -SELECT - @current_table = 'inserting #query_store_query', - @sql = @isolation_level; - -IF @troubleshoot_performance = 1 -BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; -END; - -SELECT - @sql += N' -SELECT - @database_id, - qsq.query_id, - qsq.query_text_id, - qsq.context_settings_id, - qsq.object_id, - qsq.batch_sql_handle, - qsq.query_hash, - qsq.is_internal_query, - qsq.query_parameterization_type_desc, - qsq.initial_compile_start_time, - qsq.last_compile_start_time, - qsq.last_execution_time, - qsq.last_compile_batch_sql_handle, - qsq.last_compile_batch_offset_start, - qsq.last_compile_batch_offset_end, - qsq.count_compiles, - (qsq.avg_compile_duration / 1000.), - (qsq.last_compile_duration / 1000.), - (qsq.avg_bind_duration / 1000.), - (qsq.last_bind_duration / 1000.), - (qsq.avg_bind_cpu_time / 1000.), - (qsq.last_bind_cpu_time / 1000.), - (qsq.avg_optimize_duration / 1000.), - (qsq.last_optimize_duration / 1000.), - (qsq.avg_optimize_cpu_time / 1000.), - (qsq.last_optimize_cpu_time / 1000.), - ((qsq.avg_compile_memory_kb * 8) / 1024.), - ((qsq.last_compile_memory_kb * 8) / 1024.), - ((qsq.max_compile_memory_kb * 8) / 1024.), - qsq.is_clouddb_internal_query -FROM #query_store_plan AS qsp -CROSS APPLY -( - SELECT TOP (1) - qsq.* - FROM ' + @database_name_quoted + N'.sys.query_store_query AS qsq - WHERE qsq.query_id = qsp.query_id - ORDER BY - qsq.last_execution_time DESC -) AS qsq -WHERE qsp.database_id = @database_id -OPTION(RECOMPILE);' + @nc10; - -IF @debug = 1 -BEGIN - PRINT LEN(@sql); - PRINT @sql; -END; - -INSERT - #query_store_query WITH(TABLOCK) -( - database_id, - query_id, - query_text_id, - context_settings_id, - object_id, - batch_sql_handle, - query_hash, - is_internal_query, - query_parameterization_type_desc, - initial_compile_start_time, - last_compile_start_time, - last_execution_time, - last_compile_batch_sql_handle, - last_compile_batch_offset_start, - last_compile_batch_offset_end, - count_compiles, - avg_compile_duration_ms, - last_compile_duration_ms, - avg_bind_duration_ms, - last_bind_duration_ms, - avg_bind_cpu_time_ms, - last_bind_cpu_time_ms, - avg_optimize_duration_ms, - last_optimize_duration_ms, - avg_optimize_cpu_time_ms, - last_optimize_cpu_time_ms, - avg_compile_memory_mb, - last_compile_memory_mb, - max_compile_memory_mb, - is_clouddb_internal_query -) -EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - -IF @troubleshoot_performance = 1 -BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; -END; /*End getting query details*/ - -/* -This gets the query text for them! -*/ -SELECT - @current_table = 'inserting #query_store_query_text', - @sql = @isolation_level; - -IF @troubleshoot_performance = 1 -BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; -END; - - -SELECT - @sql += N' -SELECT - @database_id, - qsqt.query_text_id, - query_sql_text = - ( - SELECT - [processing-instruction(query)] = - REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( - REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( - REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( - qsqt.query_sql_text COLLATE Latin1_General_BIN2, - NCHAR(31),N''?''),NCHAR(30),N''?''),NCHAR(29),N''?''),NCHAR(28),N''?''),NCHAR(27),N''?''),NCHAR(26),N''?''),NCHAR(25),N''?''),NCHAR(24),N''?''),NCHAR(23),N''?''),NCHAR(22),N''?''), - NCHAR(21),N''?''),NCHAR(20),N''?''),NCHAR(19),N''?''),NCHAR(18),N''?''),NCHAR(17),N''?''),NCHAR(16),N''?''),NCHAR(15),N''?''),NCHAR(14),N''?''),NCHAR(12),N''?''), - NCHAR(11),N''?''),NCHAR(8),N''?''),NCHAR(7),N''?''),NCHAR(6),N''?''),NCHAR(5),N''?''),NCHAR(4),N''?''),NCHAR(3),N''?''),NCHAR(2),N''?''),NCHAR(1),N''?''),NCHAR(0),N'''') - FOR XML - PATH(''''), - TYPE - ), - qsqt.statement_sql_handle, - qsqt.is_part_of_encrypted_module, - qsqt.has_restricted_text -FROM #query_store_query AS qsq -CROSS APPLY -( - SELECT TOP (1) - qsqt.* - FROM ' + @database_name_quoted + N'.sys.query_store_query_text AS qsqt - WHERE qsqt.query_text_id = qsq.query_text_id -) AS qsqt -WHERE qsq.database_id = @database_id -OPTION(RECOMPILE);' + @nc10; - -IF @debug = 1 -BEGIN - PRINT LEN(@sql); - PRINT @sql; -END; - -INSERT - #query_store_query_text WITH(TABLOCK) -( - database_id, - query_text_id, - query_sql_text, - statement_sql_handle, - is_part_of_encrypted_module, - has_restricted_text -) -EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - -IF @troubleshoot_performance = 1 -BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; -END; /*End getting query text*/ - -/* -Here we try to get some data from the "plan cache" -that isn't available in Query Store :( -*/ -SELECT - @sql = N'', - @current_table = 'inserting #dm_exec_query_stats'; - -IF @troubleshoot_performance = 1 -BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; -END; - -INSERT - #dm_exec_query_stats WITH(TABLOCK) -( - statement_sql_handle, - total_grant_mb, - last_grant_mb, - min_grant_mb, - max_grant_mb, - total_used_grant_mb, - last_used_grant_mb, - min_used_grant_mb, - max_used_grant_mb, - total_ideal_grant_mb, - last_ideal_grant_mb, - min_ideal_grant_mb, - max_ideal_grant_mb, - total_reserved_threads, - last_reserved_threads, - min_reserved_threads, - max_reserved_threads, - total_used_threads, - last_used_threads, - min_used_threads, - max_used_threads -) -SELECT - deqs_with_lasts.statement_sql_handle, - MAX(deqs_with_lasts.total_grant_kb) / 1024., - MAX(deqs_with_lasts.partitioned_last_grant_kb) / 1024., - MAX(deqs_with_lasts.min_grant_kb) / 1024., - MAX(deqs_with_lasts.max_grant_kb) / 1024., - MAX(deqs_with_lasts.total_used_grant_kb) / 1024., - MAX(deqs_with_lasts.partitioned_last_used_grant_kb) / 1024., - MAX(deqs_with_lasts.min_used_grant_kb) / 1024., - MAX(deqs_with_lasts.max_used_grant_kb) / 1024., - MAX(deqs_with_lasts.total_ideal_grant_kb) / 1024., - MAX(deqs_with_lasts.partitioned_last_ideal_grant_kb) / 1024., - MAX(deqs_with_lasts.min_ideal_grant_kb) / 1024., - MAX(deqs_with_lasts.max_ideal_grant_kb) / 1024., - MAX(deqs_with_lasts.total_reserved_threads), - MAX(deqs_with_lasts.partitioned_last_reserved_threads), - MAX(deqs_with_lasts.min_reserved_threads), - MAX(deqs_with_lasts.max_reserved_threads), - MAX(deqs_with_lasts.total_used_threads), - MAX(deqs_with_lasts.partitioned_last_used_threads), - MAX(deqs_with_lasts.min_used_threads), - MAX(deqs_with_lasts.max_used_threads) -FROM -( - SELECT - deqs.statement_sql_handle, - deqs.total_grant_kb, - deqs.min_grant_kb, - deqs.max_grant_kb, - deqs.total_used_grant_kb, - deqs.min_used_grant_kb, - deqs.max_used_grant_kb, - deqs.total_ideal_grant_kb, - deqs.min_ideal_grant_kb, - deqs.max_ideal_grant_kb, - deqs.total_reserved_threads, - deqs.min_reserved_threads, - deqs.max_reserved_threads, - deqs.total_used_threads, - deqs.min_used_threads, - deqs.max_used_threads, - partitioned_last_grant_kb = - LAST_VALUE(deqs.last_grant_kb) OVER - ( - PARTITION BY - deqs.sql_handle - ORDER BY - deqs.last_execution_time DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_used_grant_kb = - LAST_VALUE(deqs.last_used_grant_kb) OVER - ( - PARTITION BY - deqs.sql_handle - ORDER BY - deqs.last_execution_time DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_ideal_grant_kb = - LAST_VALUE(deqs.last_ideal_grant_kb) OVER - ( - PARTITION BY - deqs.sql_handle - ORDER BY - deqs.last_execution_time DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_reserved_threads = - LAST_VALUE(deqs.last_reserved_threads) OVER - ( - PARTITION BY - deqs.sql_handle - ORDER BY - deqs.last_execution_time DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ), - partitioned_last_used_threads = - LAST_VALUE(deqs.last_used_threads) OVER - ( - PARTITION BY - deqs.sql_handle - ORDER BY - deqs.last_execution_time DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ) - FROM sys.dm_exec_query_stats AS deqs - WHERE EXISTS - ( - SELECT - 1/0 - FROM #query_store_query_text AS qsqt - WHERE qsqt.statement_sql_handle = deqs.statement_sql_handle - ) -) AS deqs_with_lasts -GROUP BY - deqs_with_lasts.statement_sql_handle -OPTION(RECOMPILE); - -SELECT - @rc = ROWCOUNT_BIG(); - -IF @troubleshoot_performance = 1 -BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; -END; /*End getting runtime stats*/ - -/*Only update if we got anything*/ -IF @rc > 0 -BEGIN - SELECT - @current_table = 'updating #dm_exec_query_stats'; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - UPDATE - qsqt - SET - qsqt.total_grant_mb = deqs.total_grant_mb, - qsqt.last_grant_mb = deqs.last_grant_mb, - qsqt.min_grant_mb = deqs.min_grant_mb, - qsqt.max_grant_mb = deqs.max_grant_mb, - qsqt.total_used_grant_mb = deqs.total_used_grant_mb, - qsqt.last_used_grant_mb = deqs.last_used_grant_mb, - qsqt.min_used_grant_mb = deqs.min_used_grant_mb, - qsqt.max_used_grant_mb = deqs.max_used_grant_mb, - qsqt.total_ideal_grant_mb = deqs.total_ideal_grant_mb, - qsqt.last_ideal_grant_mb = deqs.last_ideal_grant_mb, - qsqt.min_ideal_grant_mb = deqs.min_ideal_grant_mb, - qsqt.max_ideal_grant_mb = deqs.max_ideal_grant_mb, - qsqt.total_reserved_threads = deqs.total_reserved_threads, - qsqt.last_reserved_threads = deqs.last_reserved_threads, - qsqt.min_reserved_threads = deqs.min_reserved_threads, - qsqt.max_reserved_threads = deqs.max_reserved_threads, - qsqt.total_used_threads = deqs.total_used_threads, - qsqt.last_used_threads = deqs.last_used_threads, - qsqt.min_used_threads = deqs.min_used_threads, - qsqt.max_used_threads = deqs.max_used_threads - FROM #query_store_query_text AS qsqt - JOIN #dm_exec_query_stats AS deqs - ON qsqt.statement_sql_handle = deqs.statement_sql_handle - OPTION(RECOMPILE); - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; -END; /*End updating runtime stats*/ - -/* -Let's check on settings, etc. -We do this first so we can see if wait stats capture mode is true more easily -*/ -SELECT - @current_table = 'inserting #database_query_store_options', - @sql = @isolation_level; - -IF @troubleshoot_performance = 1 -BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; -END; - -SELECT - @sql += N' -SELECT - @database_id, - dqso.desired_state_desc, - dqso.actual_state_desc, - readonly_reason = - CASE dqso.readonly_reason - WHEN 0 - THEN ''None'' - WHEN 2 - THEN ''Database in single user mode'' - WHEN 4 - THEN ''Database is in emergency mode'' - WHEN 8 - THEN ''Database is AG secondary'' - WHEN 65536 - THEN ''Reached max size: '' + - FORMAT(dqso.current_storage_size_mb, ''N0'') + - '' of '' + - FORMAT(dqso.max_storage_size_mb, ''N0'') + - ''.'' - WHEN 131072 - THEN ''The number of different statements in Query Store has reached the internal memory limit'' - WHEN 262144 - THEN ''Size of in-memory items waiting to be persisted on disk has reached the internal memory limit'' - WHEN 524288 - THEN ''Database has reached disk size limit'' - ELSE ''WOAH'' - END, - dqso.current_storage_size_mb, - dqso.flush_interval_seconds, - dqso.interval_length_minutes, - dqso.max_storage_size_mb, - dqso.stale_query_threshold_days, - dqso.max_plans_per_query, - dqso.query_capture_mode_desc,' - + - CASE - WHEN - ( - @product_version > 14 - OR @azure = 1 - ) - THEN N' - dqso.capture_policy_execution_count, - dqso.capture_policy_total_compile_cpu_time_ms, - dqso.capture_policy_total_execution_cpu_time_ms, - dqso.capture_policy_stale_threshold_hours,' - ELSE N' - NULL, - NULL, - NULL, - NULL,' - END - + N' - dqso.size_based_cleanup_mode_desc,' - + - CASE - WHEN - ( - @product_version = 13 - AND @azure = 0 - ) - THEN N' - NULL' - ELSE N' - dqso.wait_stats_capture_mode_desc' - END - + N' -FROM ' + @database_name_quoted + N'.sys.database_query_store_options AS dqso -OPTION(RECOMPILE);' + @nc10; - -IF @debug = 1 -BEGIN - PRINT LEN(@sql); - PRINT @sql; -END; - -INSERT - #database_query_store_options WITH(TABLOCK) -( - database_id, - desired_state_desc, - actual_state_desc, - readonly_reason, - current_storage_size_mb, - flush_interval_seconds, - interval_length_minutes, - max_storage_size_mb, - stale_query_threshold_days, - max_plans_per_query, - query_capture_mode_desc, - capture_policy_execution_count, - capture_policy_total_compile_cpu_time_ms, - capture_policy_total_execution_cpu_time_ms, - capture_policy_stale_threshold_hours, - size_based_cleanup_mode_desc, - wait_stats_capture_mode_desc -) -EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - -IF @troubleshoot_performance = 1 -BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; -END; /*End getting query store settings*/ - -/* -If wait stats are available, we'll grab them here -*/ -IF -( - @new = 1 - AND EXISTS - ( - SELECT - 1/0 - FROM #database_query_store_options AS dqso - WHERE dqso.wait_stats_capture_mode_desc = N'ON' - ) -) -BEGIN - SELECT - @current_table = 'inserting #query_store_wait_stats', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' -SELECT - @database_id, - qsws_with_lasts.plan_id, - qsws_with_lasts.wait_category_desc, - total_query_wait_time_ms = - SUM(qsws_with_lasts.total_query_wait_time_ms), - avg_query_wait_time_ms = - SUM(qsws_with_lasts.avg_query_wait_time_ms), - last_query_wait_time_ms = - MAX(qsws_with_lasts.partitioned_last_query_wait_time_ms), - min_query_wait_time_ms = - SUM(qsws_with_lasts.min_query_wait_time_ms), - max_query_wait_time_ms = - SUM(qsws_with_lasts.max_query_wait_time_ms) -FROM -( - SELECT - qsws.*, - /* - We need this here to make sure that PARTITION BY runs before GROUP BY but after CROSS APPLY. - If it were after GROUP BY, then we would be dealing with already aggregated data. - If it were inside the CROSS APPLY, then we would be dealing with windows of size one. - Both are very wrong, so we need this. - */ - partitioned_last_query_wait_time_ms = - LAST_VALUE(qsws.last_query_wait_time_ms) OVER - ( - PARTITION BY - qsws.plan_id, - qsws.execution_type, - qsws.wait_category_desc - ORDER BY - qsws.runtime_stats_interval_id DESC - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ) - FROM #query_store_runtime_stats AS qsrs - CROSS APPLY - ( - SELECT TOP (5) - qsws.* - FROM ' + @database_name_quoted + N'.sys.query_store_wait_stats AS qsws - WHERE qsws.runtime_stats_interval_id = qsrs.runtime_stats_interval_id - AND qsws.plan_id = qsrs.plan_id - AND qsws.wait_category > 0 - AND qsws.min_query_wait_time_ms > 0 - ORDER BY - qsws.avg_query_wait_time_ms DESC - ) AS qsws - WHERE qsrs.database_id = @database_id -) AS qsws_with_lasts -GROUP BY - qsws_with_lasts.plan_id, - qsws_with_lasts.wait_category_desc -HAVING - SUM(qsws_with_lasts.min_query_wait_time_ms) > 0. -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #query_store_wait_stats WITH(TABLOCK) - ( - database_id, - plan_id, - wait_category_desc, - total_query_wait_time_ms, - avg_query_wait_time_ms, - last_query_wait_time_ms, - min_query_wait_time_ms, - max_query_wait_time_ms - ) - EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; -END; /*End getting wait stats*/ - -/* -This gets context info and settings -*/ -SELECT - @current_table = 'inserting #query_context_settings', - @sql = @isolation_level; - -IF @troubleshoot_performance = 1 -BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; -END; - -SELECT - @sql += N' -SELECT - @database_id, - context_settings_id, - set_options, - language_id, - date_format, - date_first, - status, - required_cursor_options, - acceptable_cursor_options, - merge_action_type, - default_schema_id, - is_replication_specific, - is_contained -FROM ' + @database_name_quoted + N'.sys.query_context_settings AS qcs -WHERE EXISTS - ( - SELECT - 1/0 - FROM #query_store_runtime_stats AS qsrs - JOIN #query_store_plan AS qsp - ON qsrs.plan_id = qsp.plan_id - AND qsrs.database_id = qsp.database_id - JOIN #query_store_query AS qsq - ON qsp.query_id = qsq.query_id - AND qsp.database_id = qsq.database_id - WHERE qsq.context_settings_id = qcs.context_settings_id - ) -OPTION(RECOMPILE);'; - -INSERT - #query_context_settings WITH(TABLOCK) -( - database_id, - context_settings_id, - set_options, - language_id, - date_format, - date_first, - status, - required_cursor_options, - acceptable_cursor_options, - merge_action_type, - default_schema_id, - is_replication_specific, - is_contained -) -EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - -IF @troubleshoot_performance = 1 -BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; -END; /*End getting context settings*/ - -/* -Update things to get the context settings for each query -*/ -SELECT - @current_table = 'updating context_settings in #query_store_runtime_stats'; - -UPDATE - qsrs -SET - qsrs.context_settings = - SUBSTRING - ( - CASE - WHEN - CONVERT - ( - integer, - qcs.set_options - ) & 1 = 1 - THEN ', ANSI_PADDING' - ELSE '' - END + - CASE - WHEN - CONVERT - ( - integer, - qcs.set_options - ) & 8 = 8 - THEN ', CONCAT_NULL_YIELDS_NULL' - ELSE '' - END + - CASE - WHEN - CONVERT - ( - integer, - qcs.set_options - ) & 16 = 16 - THEN ', ANSI_WARNINGS' - ELSE '' - END + - CASE - WHEN - CONVERT - ( - integer, - qcs.set_options - ) & 32 = 32 - THEN ', ANSI_NULLS' - ELSE '' - END + - CASE - WHEN - CONVERT - ( - integer, - qcs.set_options - ) & 64 = 64 - THEN ', QUOTED_IDENTIFIER' - ELSE '' - END + - CASE - WHEN - CONVERT - ( - integer, - qcs.set_options - ) & 4096 = 4096 - THEN ', ARITH_ABORT' - ELSE '' - END + - CASE - WHEN - CONVERT - ( - integer, - qcs.set_options - ) & 8192 = 8192 - THEN ', NUMERIC_ROUNDABORT' - ELSE '' - END, - 2, - 256 - ) -FROM #query_store_runtime_stats AS qsrs -JOIN #query_store_plan AS qsp - ON qsrs.plan_id = qsp.plan_id - AND qsrs.database_id = qsp.database_id -JOIN #query_store_query AS qsq - ON qsp.query_id = qsq.query_id - AND qsp.database_id = qsq.database_id -JOIN #query_context_settings AS qcs - ON qsq.context_settings_id = qcs.context_settings_id - AND qsq.database_id = qcs.database_id -OPTION(RECOMPILE); - -IF @sql_2022_views = 1 -BEGIN - /*query_store_plan_feedback*/ - SELECT - @current_table = 'inserting #query_store_plan_feedback', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' -SELECT - @database_id, - qspf.plan_feedback_id, - qspf.plan_id, - qspf.feature_desc, - qspf.feedback_data, - qspf.state_desc, - qspf.create_time, - qspf.last_updated_time -FROM ' + @database_name_quoted + N'.sys.query_store_plan_feedback AS qspf -WHERE EXISTS - ( - SELECT - 1/0 - FROM #query_store_plan AS qsp - WHERE qspf.plan_id = qsp.plan_id - ) -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #query_store_plan_feedback WITH(TABLOCK) - ( - database_id, - plan_feedback_id, - plan_id, - feature_desc, - feedback_data, - state_desc, - create_time, - last_updated_time - ) - EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; - - /*query_store_query_variant*/ - SELECT - @current_table = 'inserting #query_store_query_variant', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' -SELECT - @database_id, - qsqv.query_variant_query_id, - qsqv.parent_query_id, - qsqv.dispatcher_plan_id -FROM ' + @database_name_quoted + N'.sys.query_store_query_variant AS qsqv -WHERE EXISTS - ( - SELECT - 1/0 - FROM #query_store_plan AS qsp - WHERE qsqv.query_variant_query_id = qsp.query_id - AND qsqv.dispatcher_plan_id = qsp.plan_id - ) -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #query_store_query_variant WITH(TABLOCK) - ( - database_id, - query_variant_query_id, - parent_query_id, - dispatcher_plan_id - ) - EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; - - /*query_store_query_hints*/ - SELECT - @current_table = 'inserting #query_store_query_hints', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' -SELECT - @database_id, - qsqh.query_hint_id, - qsqh.query_id, - qsqh.query_hint_text, - qsqh.last_query_hint_failure_reason_desc, - qsqh.query_hint_failure_count, - qsqh.source_desc -FROM ' + @database_name_quoted + N'.sys.query_store_query_hints AS qsqh -WHERE EXISTS - ( - SELECT - 1/0 - FROM #query_store_plan AS qsp - WHERE qsqh.query_id = qsp.query_id - ) -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #query_store_query_hints WITH(TABLOCK) - ( - database_id, - query_hint_id, - query_id, - query_hint_text, - last_query_hint_failure_reason_desc, - query_hint_failure_count, - source_desc - ) - EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; - - IF @ags_present = 1 - BEGIN - /*query_store_plan_forcing_locations*/ - SELECT - @current_table = 'inserting #query_store_plan_forcing_locations', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' -SELECT - @database_id, - qspfl.plan_forcing_location_id, - qspfl.query_id, - qspfl.plan_id, - qspfl.replica_group_id -FROM ' + @database_name_quoted + N'.sys.query_store_plan_forcing_locations AS qspfl -WHERE EXISTS - ( - SELECT - 1/0 - FROM #query_store_plan AS qsp - WHERE qspfl.query_id = qsp.query_id - AND qspfl.plan_id = qsp.plan_id - ) -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #query_store_plan_forcing_locations WITH(TABLOCK) - ( - database_id, - plan_forcing_location_id, - query_id, - plan_id, - replica_group_id - ) - EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; - - /*query_store_replicas*/ - SELECT - @current_table = 'inserting #query_store_replicas', - @sql = @isolation_level; - - IF @troubleshoot_performance = 1 - BEGIN - EXECUTE sys.sp_executesql - @troubleshoot_insert, - N'@current_table nvarchar(100)', - @current_table; - - SET STATISTICS XML ON; - END; - - SELECT - @sql += N' -SELECT - @database_id, - qsr.replica_group_id, - qsr.role_type, - qsr.replica_name -FROM ' + @database_name_quoted + N'.sys.query_store_replicas AS qsr -WHERE EXISTS - ( - SELECT - 1/0 - FROM #query_store_plan_forcing_locations AS qspfl - WHERE qspfl.replica_group_id = qsr.replica_group_id - ) -OPTION(RECOMPILE);' + @nc10; - - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; - - INSERT - #query_store_replicas WITH(TABLOCK) - ( - database_id, - replica_group_id, - role_type, - replica_name - ) - EXECUTE sys.sp_executesql - @sql, - N'@database_id int', - @database_id; - - IF @troubleshoot_performance = 1 - BEGIN - SET STATISTICS XML OFF; - - EXECUTE sys.sp_executesql - @troubleshoot_update, - N'@current_table nvarchar(100)', - @current_table; - - EXECUTE sys.sp_executesql - @troubleshoot_info, - N'@sql nvarchar(max), - @current_table nvarchar(100)', - @sql, - @current_table; - END; - END; /*End AG queries*/ -END; /*End SQL 2022 views*/ - -/* -These tables need to get cleared out -to avoid result pollution and -primary key violations -*/ -IF @get_all_databases = 1 -BEGIN - TRUNCATE TABLE - #regression_baseline_runtime_stats; - - TRUNCATE TABLE - #regression_current_runtime_stats; - - TRUNCATE TABLE - #distinct_plans; - - TRUNCATE TABLE - #procedure_plans; - - TRUNCATE TABLE - #procedure_object_ids; - - TRUNCATE TABLE - #maintenance_plans; - - TRUNCATE TABLE - #query_text_search; - - TRUNCATE TABLE - #query_text_search_not; - - TRUNCATE TABLE - #dm_exec_query_stats; - - TRUNCATE TABLE - #query_types; - - TRUNCATE TABLE - #wait_filter; - - TRUNCATE TABLE - #only_queries_with_hints; - - TRUNCATE TABLE - #only_queries_with_feedback; - - TRUNCATE TABLE - #only_queries_with_variants; - - TRUNCATE TABLE - #forced_plans_failures; -END; - -FETCH NEXT -FROM @database_cursor -INTO @database_name; -END; - -/* -This is where we start returning results -*/ -IF EXISTS - ( - SELECT - 1/0 - FROM #query_store_runtime_stats AS qsrs - ) -BEGIN - SELECT - @sql = @isolation_level, - @current_table = 'selecting final results'; - - SELECT - @sql += - CONVERT - ( - nvarchar(MAX), - N' -SELECT - x.* -FROM -(' - ); - - /* - Expert mode returns more columns from runtime stats - */ - IF - ( - @expert_mode = 1 - AND @format_output = 0 - ) - BEGIN - SELECT - @sql += - CONVERT - ( - nvarchar(MAX), - N' - SELECT - source = - ''runtime_stats'', - database_name = - DB_NAME(qsrs.database_id), - qsp.query_id, - qsrs.plan_id, - qsp.all_plan_ids,' - + - CASE - WHEN @include_plan_hashes IS NOT NULL - THEN - N' - qsp.query_plan_hash,' - WHEN @include_query_hashes IS NOT NULL - OR @sort_order = 'plan count by hashes' - OR @include_query_hash_totals = 1 - THEN - N' - qsq.query_hash,' - WHEN @include_sql_handles IS NOT NULL - THEN - N' - qsqt.statement_sql_handle,' - ELSE - N'' - END + N' - qsrs.execution_type_desc, - qsq.object_name, - qsqt.query_sql_text, - query_plan = - CASE - WHEN TRY_CAST(qsp.query_plan AS xml) IS NOT NULL - THEN TRY_CAST(qsp.query_plan AS xml) - WHEN TRY_CAST(qsp.query_plan AS xml) IS NULL - THEN - ( - SELECT - [processing-instruction(query_plan)] = - N''-- '' + NCHAR(13) + NCHAR(10) + - N''-- This is a huge query plan.'' + NCHAR(13) + NCHAR(10) + - N''-- Remove the headers and footers, save it as a .sqlplan file, and re-open it.'' + NCHAR(13) + NCHAR(10) + - NCHAR(13) + NCHAR(10) + - REPLACE(qsp.query_plan, N'' 0 + AND qsws.min_query_wait_time_ms > 0 + ORDER BY + qsws.avg_query_wait_time_ms DESC + ) AS qsws + WHERE qsrs.database_id = @database_id +) AS qsws_with_lasts +GROUP BY + qsws_with_lasts.plan_id, + qsws_with_lasts.wait_category_desc +HAVING + SUM(qsws_with_lasts.min_query_wait_time_ms) > 0. +OPTION(RECOMPILE);' + @nc10; - /* - For non-experts only! - */ - IF + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + + INSERT + #query_store_wait_stats + WITH + (TABLOCK) ( - @expert_mode = 0 - AND @format_output = 0 + database_id, + plan_id, + wait_category_desc, + total_query_wait_time_ms, + avg_query_wait_time_ms, + last_query_wait_time_ms, + min_query_wait_time_ms, + max_query_wait_time_ms ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id int', + @database_id; + + IF @troubleshoot_performance = 1 BEGIN - SELECT - @sql += - CONVERT - ( - nvarchar(MAX), - N' - SELECT - source = - ''runtime_stats'', - database_name = - DB_NAME(qsrs.database_id), - qsp.query_id, - qsrs.plan_id, - qsp.all_plan_ids,' - + - CASE - WHEN @include_plan_hashes IS NOT NULL - THEN - N' - qsp.query_plan_hash,' - WHEN @include_query_hashes IS NOT NULL - OR @sort_order = 'plan count by hashes' - OR @include_query_hash_totals = 1 - THEN - N' - qsq.query_hash,' - WHEN @include_sql_handles IS NOT NULL - THEN - N' - qsqt.statement_sql_handle,' - ELSE - N'' - END + N' - qsrs.execution_type_desc, - qsq.object_name, - qsqt.query_sql_text, - query_plan = - CASE - WHEN TRY_CAST(qsp.query_plan AS xml) IS NOT NULL - THEN TRY_CAST(qsp.query_plan AS xml) - WHEN TRY_CAST(qsp.query_plan AS xml) IS NULL - THEN - ( - SELECT - [processing-instruction(query_plan)] = - N''-- '' + NCHAR(13) + NCHAR(10) + - N''-- This is a huge query plan.'' + NCHAR(13) + NCHAR(10) + - N''-- Remove the headers and footers, save it as a .sqlplan file, and re-open it.'' + NCHAR(13) + NCHAR(10) + - NCHAR(13) + NCHAR(10) + - REPLACE(qsp.query_plan, N'' 0 + SELECT + @sql += N' +SELECT + @database_id, + qsr.replica_group_id, + qsr.role_type, + qsr.replica_name +FROM ' + @database_name_quoted + N'.sys.query_store_replicas AS qsr +WHERE EXISTS + ( + SELECT + 1/0 + FROM #query_store_plan_forcing_locations AS qspfl + WHERE qspfl.replica_group_id = qsr.replica_group_id + ) +OPTION(RECOMPILE);' + @nc10; + + IF @debug = 1 BEGIN - SELECT - @current_table = 'selecting resource stats'; + PRINT LEN(@sql); + PRINT @sql; + END; - SELECT - source = - 'resource_stats', - database_name = - DB_NAME(qsq.database_id), - qsq.query_id, - qsq.object_name, - qsqt.total_grant_mb, - qsqt.last_grant_mb, - qsqt.min_grant_mb, - qsqt.max_grant_mb, - qsqt.total_used_grant_mb, - qsqt.last_used_grant_mb, - qsqt.min_used_grant_mb, - qsqt.max_used_grant_mb, - qsqt.total_ideal_grant_mb, - qsqt.last_ideal_grant_mb, - qsqt.min_ideal_grant_mb, - qsqt.max_ideal_grant_mb, - qsqt.total_reserved_threads, - qsqt.last_reserved_threads, - qsqt.min_reserved_threads, - qsqt.max_reserved_threads, - qsqt.total_used_threads, - qsqt.last_used_threads, - qsqt.min_used_threads, - qsqt.max_used_threads - FROM #query_store_query AS qsq - JOIN #query_store_query_text AS qsqt - ON qsq.query_text_id = qsqt.query_text_id - AND qsq.database_id = qsqt.database_id - WHERE - ( - qsqt.total_grant_mb IS NOT NULL - OR qsqt.total_reserved_threads IS NOT NULL - ) - ORDER BY - qsq.query_id - OPTION(RECOMPILE); + INSERT + #query_store_replicas + WITH + (TABLOCK) + ( + database_id, + replica_group_id, + role_type, + replica_name + ) + EXECUTE sys.sp_executesql + @sql, + N'@database_id int', + @database_id; - END; /*End resource stats query*/ - ELSE + IF @troubleshoot_performance = 1 BEGIN - SELECT - result = - '#dm_exec_query_stats is empty'; + SET STATISTICS XML OFF; + + EXECUTE sys.sp_executesql + @troubleshoot_update, + N'@current_table nvarchar(100)', + @current_table; + + EXECUTE sys.sp_executesql + @troubleshoot_info, + N'@sql nvarchar(max), + @current_table nvarchar(100)', + @sql, + @current_table; END; - END; + END; /*End AG queries*/ +END; /*End SQL 2022 views*/ - IF @new = 1 - BEGIN - IF @expert_mode = 1 - BEGIN - IF EXISTS - ( - SELECT - 1/0 - FROM #query_store_wait_stats AS qsws - ) - BEGIN - SELECT - @current_table = 'selecting wait stats by query'; +FETCH NEXT +FROM @database_cursor +INTO @database_name; +END; - SELECT DISTINCT - source = - 'query_store_wait_stats_by_query', - database_name = - DB_NAME(qsws.database_id), - qsws.plan_id, - x.object_name, - qsws.wait_category_desc, - qsws.total_query_wait_time_ms, - total_query_duration_ms = - x.total_duration_ms, - qsws.avg_query_wait_time_ms, - avg_query_duration_ms = - x.avg_duration_ms, - qsws.last_query_wait_time_ms, - last_query_duration_ms = - x.last_duration_ms, - qsws.min_query_wait_time_ms, - min_query_duration_ms = - x.min_duration_ms, - qsws.max_query_wait_time_ms, - max_query_duration_ms = - x.max_duration_ms - FROM #query_store_wait_stats AS qsws - CROSS APPLY - ( - SELECT - qsrs.avg_duration_ms, - qsrs.last_duration_ms, - qsrs.min_duration_ms, - qsrs.max_duration_ms, - qsrs.total_duration_ms, - qsq.object_name - FROM #query_store_runtime_stats AS qsrs - JOIN #query_store_plan AS qsp - ON qsrs.plan_id = qsp.plan_id - AND qsrs.database_id = qsp.database_id - JOIN #query_store_query AS qsq - ON qsp.query_id = qsq.query_id - AND qsp.database_id = qsq.database_id - WHERE qsws.plan_id = qsrs.plan_id - AND qsws.database_id = qsrs.database_id - ) AS x - ORDER BY - qsws.plan_id, - qsws.total_query_wait_time_ms DESC - OPTION(RECOMPILE); +/* +This is where we start returning results +*/ +IF EXISTS + ( + SELECT + 1/0 + FROM #query_store_runtime_stats AS qsrs + ) +BEGIN + SELECT + @sql = @isolation_level, + @current_table = 'selecting final results'; - SELECT - @current_table = 'selecting wait stats in total'; + SELECT + @sql += + CONVERT + ( + nvarchar(MAX), + N' +SELECT + x.* +FROM +( + SELECT + source = ''runtime_stats'', + database_name = DB_NAME(qsrs.database_id), + qsp.query_id, + qsrs.plan_id, + qsp.all_plan_ids,' + + CASE + WHEN @include_plan_hashes IS NOT NULL + THEN N' + qsp.query_plan_hash,' + WHEN @include_query_hashes IS NOT NULL + OR @sort_order = 'plan count by hashes' + OR @include_query_hash_totals = 1 + THEN N' + qsq.query_hash,' + WHEN @include_sql_handles IS NOT NULL + THEN N' + qsqt.statement_sql_handle,' + ELSE N'' + END + N' + qsrs.execution_type_desc, + qsq.object_name, + qsqt.query_sql_text, + query_plan = + CASE + WHEN TRY_CAST(qsp.query_plan AS xml) IS NOT NULL + THEN TRY_CAST(qsp.query_plan AS xml) + WHEN TRY_CAST(qsp.query_plan AS xml) IS NULL + THEN + ( + SELECT + [processing-instruction(query_plan)] = + N''-- '' + NCHAR(13) + NCHAR(10) + + N''-- This is a huge query plan.'' + NCHAR(13) + NCHAR(10) + + N''-- Remove the headers and footers, save it as a .sqlplan file, and re-open it.'' + NCHAR(13) + NCHAR(10) + + NCHAR(13) + NCHAR(10) + + REPLACE(qsp.query_plan, N'' 0 + BEGIN + SET @column_sql = + LEFT + ( + @column_sql, + LEN(@column_sql) - 1 + ); + END; + + /* Append the column SQL to the main SQL */ + SELECT + @sql += @column_sql; - END; /*End unformatted wait stats*/ - ELSE - BEGIN - SELECT - result = - '#query_store_wait_stats is empty' + - CASE - WHEN - ( - @product_version = 13 - AND @azure = 0 - ) - THEN ' because it''s not available < 2017' - WHEN EXISTS - ( - SELECT - 1/0 - FROM #database_query_store_options AS dqso - WHERE dqso.wait_stats_capture_mode_desc <> 'ON' - ) - THEN ' because you have it disabled in your Query Store options' - ELSE ' for the queries in the results' - END; - END; - END; - END; /*End wait stats queries*/ + /* + Add on the from and stuff + */ + SELECT + @sql += + CONVERT + ( + nvarchar(MAX), + N' + FROM #query_store_runtime_stats AS qsrs' + ); + + /* + Bolt on any sort-helping tables. + */ + IF @regression_mode = 1 + BEGIN + SELECT + @sql += N' + JOIN #regression_changes AS regression + ON qsrs.plan_id = regression.plan_id + AND qsrs.database_id = regression.database_id'; + END; - IF - ( - @sql_2022_views = 1 - AND @ags_present = 1 - ) + IF @sort_order = 'plan count by hashes' BEGIN - IF @expert_mode = 1 - BEGIN - IF EXISTS - ( - SELECT - 1/0 - FROM #query_store_replicas AS qsr - JOIN #query_store_plan_forcing_locations AS qspfl - ON qsr.replica_group_id = qspfl.replica_group_id - AND qsr.database_id = qspfl.database_id - ) - BEGIN - SELECT - @current_table = 'selecting #query_store_replicas and #query_store_plan_forcing_locations'; + SELECT + @sql += N' + JOIN #plan_ids_with_query_hashes AS hashes + ON qsrs.plan_id = hashes.plan_id + AND qsrs.database_id = hashes.database_id'; + END; - SELECT - database_name = - DB_NAME(qsr.database_id), - qsr.replica_group_id, - qsr.role_type, - qsr.replica_name, - qspfl.plan_forcing_location_id, - qspfl.query_id, - qspfl.plan_id, - qspfl.replica_group_id - FROM #query_store_replicas AS qsr - JOIN #query_store_plan_forcing_locations AS qspfl - ON qsr.replica_group_id = qspfl.replica_group_id - ORDER BY - qsr.replica_group_id; - END; - ELSE - BEGIN - SELECT - result = 'Availability Group information is empty'; - END; + IF @sort_order_is_a_wait = 1 + BEGIN + SELECT + @sql += N' + JOIN #plan_ids_with_total_waits AS waits + ON qsrs.plan_id = waits.plan_id + AND qsrs.database_id = waits.database_id'; + + IF @regression_mode = 1 + BEGIN + SELECT + @sql += N' + AND qsrs.from_regression_baseline = waits.from_regression_baseline'; END; END; - IF @expert_mode = 1 - BEGIN +/*Get more stuff, like query plans and query text*/ +SELECT + @sql += + CONVERT + ( + nvarchar(MAX), + N' + CROSS APPLY + ( SELECT - @current_table = 'selecting query store options', - @sql = N''; + x.* + FROM + ( + SELECT + qsp.*, + pn = + ROW_NUMBER() OVER + ( + PARTITION BY + qsp.query_plan_hash + ORDER BY + qsp.last_execution_time DESC + ) + FROM #query_store_plan AS qsp + WHERE qsp.plan_id = qsrs.plan_id + AND qsp.database_id = qsrs.database_id + ) AS x + WHERE x.pn = 1 + ) AS qsp + CROSS APPLY + ( + SELECT TOP (1) + qsqt.* + FROM #query_store_query AS qsq + JOIN #query_store_query_text AS qsqt + ON qsqt.query_text_id = qsq.query_text_id + AND qsqt.database_id = qsq.database_id + WHERE qsq.query_id = qsp.query_id + AND qsq.database_id = qsp.database_id + ORDER BY + qsq.last_execution_time DESC + ) AS qsqt + CROSS APPLY + ( + SELECT TOP (1) + qsq.* + FROM #query_store_query AS qsq + WHERE qsq.query_id = qsp.query_id + AND qsq.database_id = qsp.database_id + ORDER BY + qsq.last_execution_time DESC + ) AS qsq' + ); + /* + Get wait stats if we can + */ + IF + ( + @new = 1 + ) + BEGIN SELECT @sql += CONVERT ( nvarchar(MAX), N' - SELECT - source = - ''query_store_options'', - database_name = - DB_NAME(dqso.database_id), - dqso.desired_state_desc, - dqso.actual_state_desc, - dqso.readonly_reason, - dqso.current_storage_size_mb, - dqso.flush_interval_seconds, - dqso.interval_length_minutes, - dqso.max_storage_size_mb, - dqso.stale_query_threshold_days, - dqso.max_plans_per_query, - dqso.query_capture_mode_desc,' - + - CASE - WHEN - ( - @azure = 1 - OR @product_version > 13 - ) - THEN N' - dqso.wait_stats_capture_mode_desc,' - ELSE N'' - END - + - CASE - WHEN + CROSS APPLY + ( + SELECT TOP (1) + top_waits = + STUFF ( - @azure = 1 - OR @product_version > 14 + ( + SELECT TOP (5) + '', '' + + qsws.wait_category_desc + + '' ('' + ' + + CASE + @format_output + WHEN 0 + THEN N' + CONVERT + ( + varchar(20), + SUM + ( + CONVERT + ( + bigint, + qsws.avg_query_wait_time_ms + ) + ) + ) + ' + ELSE N' + FORMAT + ( + SUM + ( + CONVERT + ( + bigint, + qsws.avg_query_wait_time_ms + ) + ), + ''N0'' + ) + ' + END + N' '' ms)'' + FROM #query_store_wait_stats AS qsws + WHERE qsws.plan_id = qsrs.plan_id + AND qsws.database_id = qsrs.database_id + GROUP BY + qsws.wait_category_desc + ORDER BY + SUM(qsws.avg_query_wait_time_ms) DESC + FOR XML + PATH(''''), + TYPE + ).value(''./text()[1]'', ''varchar(max)''), + 1, + 2, + '''' ) - THEN N' - dqso.capture_policy_execution_count, - dqso.capture_policy_total_compile_cpu_time_ms, - dqso.capture_policy_total_execution_cpu_time_ms, - dqso.capture_policy_stale_threshold_hours,' - ELSE N'' - END - ); + ) AS w' + ); + END; /*End wait stats query*/ + /*Strap on the query hash totals table*/ + IF @include_query_hash_totals = 1 + BEGIN SELECT - @sql += - CONVERT - ( - nvarchar(MAX), - N' - dqso.size_based_cleanup_mode_desc - FROM #database_query_store_options AS dqso - OPTION(RECOMPILE);' - ); + @sql += N' + JOIN #query_hash_totals AS qht + ON qsq.query_hash = qht.query_hash + AND qsq.database_id = qht.database_id'; + END; - IF @debug = 1 - BEGIN - PRINT LEN(@sql); - PRINT @sql; - END; + SELECT + @sql += + CONVERT + ( + nvarchar(MAX), + N' +) AS x +' + CASE WHEN @regression_mode = 1 THEN N'' ELSE N'WHERE x.n = 1 ' END ++ N' +ORDER BY + ' + + CASE @format_output + WHEN 0 + THEN + CASE WHEN @regression_mode = 1 + AND @regression_direction IN ('improved', 'better') + THEN 'x.change_in_average_for_query_hash_since_regression_time_period ASC, + x.query_hash_from_regression_checking, + x.from_regression_baseline_time_period' + WHEN @regression_mode = 1 + AND @regression_direction IN ('regressed', 'worse') + THEN 'x.change_in_average_for_query_hash_since_regression_time_period DESC, + x.query_hash_from_regression_checking, + x.from_regression_baseline_time_period' + WHEN @regression_mode = 1 + AND @regression_direction IN ('magnitude', 'absolute') + THEN 'ABS(x.change_in_average_for_query_hash_since_regression_time_period) DESC, + x.query_hash_from_regression_checking, + x.from_regression_baseline_time_period' + ELSE + CASE @sort_order + WHEN 'cpu' THEN N'x.avg_cpu_time_ms' + WHEN 'logical reads' THEN N'x.avg_logical_io_reads_mb' + WHEN 'physical reads' THEN N'x.avg_physical_io_reads_mb' + WHEN 'writes' THEN N'x.avg_logical_io_writes_mb' + WHEN 'duration' THEN N'x.avg_duration_ms' + WHEN 'memory' THEN N'x.avg_query_max_used_memory_mb' + WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'x.avg_tempdb_space_used_mb' ELSE N'x.avg_cpu_time' END + WHEN 'executions' THEN N'x.count_executions' + WHEN 'recent' THEN N'x.last_execution_time' + WHEN 'rows' THEN N'x.avg_rowcount' + WHEN 'plan count by hashes' THEN N'x.plan_hash_count_for_query_hash DESC, + x.query_hash_from_hash_counting' + ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'x.total_wait_time_from_sort_order_ms' ELSE N'x.avg_cpu_time' END + END END + /* + The ORDER BY is on the same level as the topmost SELECT, which is just SELECT x.*. + This means that to sort formatted output, we have to un-format it. + */ + WHEN 1 + THEN + CASE WHEN @regression_mode = 1 + AND @regression_direction IN ('improved', 'better') + THEN 'TRY_PARSE(replace(x.change_in_average_for_query_hash_since_regression_time_period, ''%'', '''') AS money) ASC, + x.query_hash_from_regression_checking, + x.from_regression_baseline_time_period' + WHEN @regression_mode = 1 + AND @regression_direction IN ('regressed', 'worse') + THEN 'TRY_PARSE(replace(x.change_in_average_for_query_hash_since_regression_time_period, ''%'', '''') AS money) DESC, + x.query_hash_from_regression_checking, + x.from_regression_baseline_time_period' + WHEN @regression_mode = 1 + AND @regression_direction IN ('magnitude', 'absolute') + THEN 'ABS(TRY_PARSE(replace(x.change_in_average_for_query_hash_since_regression_time_period, ''%'', '''') AS money)) DESC, + x.query_hash_from_regression_checking, + x.from_regression_baseline_time_period' + ELSE + CASE @sort_order + WHEN 'cpu' THEN N'TRY_PARSE(x.avg_cpu_time_ms AS money)' + WHEN 'logical reads' THEN N'TRY_PARSE(x.avg_logical_io_reads_mb AS money)' + WHEN 'physical reads' THEN N'TRY_PARSE(x.avg_physical_io_reads_mb AS money)' + WHEN 'writes' THEN N'TRY_PARSE(x.avg_logical_io_writes_mb AS money)' + WHEN 'duration' THEN N'TRY_PARSE(x.avg_duration_ms AS money)' + WHEN 'memory' THEN N'TRY_PARSE(x.avg_query_max_used_memory_mb AS money)' + WHEN 'tempdb' THEN CASE WHEN @new = 1 THEN N'TRY_PARSE(x.avg_tempdb_space_used_mb AS money)' ELSE N'TRY_PARSE(x.avg_cpu_time AS money)' END + WHEN 'executions' THEN N'TRY_PARSE(x.count_executions AS money)' + WHEN 'recent' THEN N'x.last_execution_time' + WHEN 'rows' THEN N'TRY_PARSE(x.avg_rowcount AS money)' + WHEN 'plan count by hashes' THEN N'TRY_PARSE(x.plan_hash_count_for_query_hash AS money) DESC, + x.query_hash_from_hash_counting' + ELSE CASE WHEN @sort_order_is_a_wait = 1 THEN N'TRY_PARSE(x.total_wait_time_from_sort_order_ms AS money)' ELSE N'TRY_PARSE(x.avg_cpu_time AS money)' END + END END + END + + N' DESC +OPTION(RECOMPILE);' + + @nc10 + ); - EXECUTE sys.sp_executesql - @sql; + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT SUBSTRING(@sql, 0, 4000); + PRINT SUBSTRING(@sql, 4001, 8000); + PRINT SUBSTRING(@sql, 8001, 12000); + PRINT SUBSTRING(@sql, 12001, 16000); END; -END; /*End expert mode format output = 0*/ + + EXECUTE sys.sp_executesql + @sql, + N'@utc_offset_string nvarchar(6), + @timezone sysname', + @utc_offset_string, + @timezone; +END; /*End runtime stats main query*/ +ELSE +BEGIN + SELECT + result = + '#query_store_runtime_stats is empty'; +END; /* -Return special things, formatted +Return special things: plan feedback, query hints, query variants, query text, wait stats, and query store options +This section handles all expert mode and special output formats +Format numeric values based on @format_output */ IF ( - ( - @expert_mode = 1 - OR - ( - @only_queries_with_hints = 1 - OR @only_queries_with_feedback = 1 - OR @only_queries_with_variants = 1 - ) - ) -AND @format_output = 1 + @expert_mode = 1 + OR + ( + @only_queries_with_hints = 1 + OR @only_queries_with_feedback = 1 + OR @only_queries_with_variants = 1 + ) ) BEGIN + /* + SQL 2022+ features: plan feedback, query hints, and query variants + */ IF @sql_2022_views = 1 BEGIN - IF @expert_mode = 1 + /* + Handle query_store_plan_feedback + */ + IF @expert_mode = 1 + OR @only_queries_with_feedback = 1 BEGIN IF EXISTS ( @@ -10440,7 +8702,15 @@ BEGIN BEGIN SELECT @current_table = 'selecting plan feedback'; - + + /* + Use dynamic SQL to handle formatting differences based on @format_output + */ + SELECT + @sql = @isolation_level; + + SELECT + @sql += N' SELECT database_name = DB_NAME(qspf.database_id), @@ -10480,46 +8750,84 @@ BEGIN FROM #query_store_plan_feedback AS qspf ORDER BY qspf.plan_id - OPTION(RECOMPILE); + OPTION(RECOMPILE);'; + + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql, + N'@timezone sysname, @utc_offset_string nvarchar(max)', + @timezone, @utc_offset_string; END; - ELSE + ELSE IF @only_queries_with_feedback = 1 BEGIN SELECT result = '#query_store_plan_feedback is empty'; END; - END - - IF EXISTS - ( - SELECT - 1/0 - FROM #query_store_query_hints AS qsqh - ) - BEGIN - SELECT - @current_table = 'selecting query hints'; + END; /*@only_queries_with_feedback*/ - SELECT - database_name = - DB_NAME(qsqh.database_id), - qsqh.query_hint_id, - qsqh.query_id, - qsqh.query_hint_text, - qsqh.last_query_hint_failure_reason_desc, - qsqh.query_hint_failure_count, - qsqh.source_desc - FROM #query_store_query_hints AS qsqh - ORDER BY - qsqh.query_id - OPTION(RECOMPILE); - END; - ELSE + IF @expert_mode = 1 + OR @only_queries_with_hints = 1 BEGIN - SELECT - result = '#query_store_query_hints is empty'; - END; - - IF @expert_mode = 1 + IF EXISTS + ( + SELECT + 1/0 + FROM #query_store_query_hints AS qsqh + ) + BEGIN + SELECT + @current_table = 'selecting query hints'; + + /* + Use dynamic SQL to handle formatting differences based on @format_output + */ + SELECT + @sql = @isolation_level; + + SELECT + @sql += N' + SELECT + database_name = + DB_NAME(qsqh.database_id), + qsqh.query_hint_id, + qsqh.query_id, + qsqh.query_hint_text, + qsqh.last_query_hint_failure_reason_desc, + query_hint_failure_count = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqh.query_hint_failure_count, ''N0'')' + ELSE N'qsqh.query_hint_failure_count' + END + N', + qsqh.source_desc + FROM #query_store_query_hints AS qsqh + ORDER BY + qsqh.query_id + OPTION(RECOMPILE);'; + + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql; + END; + ELSE IF @only_queries_with_hints = 1 + BEGIN + SELECT + result = '#query_store_query_hints is empty'; + END; + END; /*@only_queries_with_hints*/ + + IF @expert_mode = 1 + OR @only_queries_with_variants = 1 BEGIN IF EXISTS ( @@ -10530,7 +8838,15 @@ BEGIN BEGIN SELECT @current_table = 'selecting query variants'; - + + /* + Use dynamic SQL to handle formatting differences based on @format_output + */ + SELECT + @sql = @isolation_level; + + SELECT + @sql += N' SELECT database_name = DB_NAME(qsqv.database_id), @@ -10540,15 +8856,70 @@ BEGIN FROM #query_store_query_variant AS qsqv ORDER BY qsqv.parent_query_id - OPTION(RECOMPILE); + OPTION(RECOMPILE);'; + + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql; END; - ELSE + ELSE IF @only_queries_with_variants = 1 BEGIN SELECT result = '#query_store_query_variant is empty'; END; - END; - END; + END; /*@only_queries_with_variants*/ + + IF + ( + @sql_2022_views = 1 + AND @ags_present = 1 + ) + BEGIN + IF @expert_mode = 1 + BEGIN + IF EXISTS + ( + SELECT + 1/0 + FROM #query_store_replicas AS qsr + JOIN #query_store_plan_forcing_locations AS qspfl + ON qsr.replica_group_id = qspfl.replica_group_id + AND qsr.database_id = qspfl.database_id + ) + BEGIN + SELECT + @current_table = 'selecting #query_store_replicas and #query_store_plan_forcing_locations'; + + SELECT + database_name = + DB_NAME(qsr.database_id), + qsr.replica_group_id, + qsr.role_type, + qsr.replica_name, + qspfl.plan_forcing_location_id, + qspfl.query_id, + qspfl.plan_id, + qspfl.replica_group_id + FROM #query_store_replicas AS qsr + JOIN #query_store_plan_forcing_locations AS qspfl + ON qsr.replica_group_id = qspfl.replica_group_id + ORDER BY + qsr.replica_group_id + OPTION(RECOMPILE); + END; + ELSE + BEGIN + SELECT + result = 'Availability Group information is empty'; + END; + END; + END; /*@ags_present*/ + END; /*End 2022 views*/ IF @expert_mode = 1 BEGIN @@ -10561,14 +8932,22 @@ BEGIN BEGIN SELECT @current_table = 'selecting compilation stats'; - + + /* + Use dynamic SQL to handle formatting differences based on @format_output + */ + SELECT + @sql = @isolation_level; + + SELECT + @sql += N' SELECT x.* FROM ( SELECT source = - 'compilation_stats', + ''compilation_stats'', database_name = DB_NAME(qsq.database_id), qsq.query_id, @@ -10617,46 +8996,130 @@ BEGIN END, last_execution_time_utc = qsq.last_execution_time, - count_compiles = - FORMAT(qsq.count_compiles, 'N0'), - avg_compile_duration_ms = - FORMAT(qsq.avg_compile_duration_ms, 'N0'), - total_compile_duration_ms = - FORMAT(qsq.total_compile_duration_ms, 'N0'), - last_compile_duration_ms = - FORMAT(qsq.last_compile_duration_ms, 'N0'), - avg_bind_duration_ms = - FORMAT(qsq.avg_bind_duration_ms, 'N0'), - total_bind_duration_ms = - FORMAT(qsq.total_bind_duration_ms, 'N0'), - last_bind_duration_ms = - FORMAT(qsq.last_bind_duration_ms, 'N0'), - avg_bind_cpu_time_ms = - FORMAT(qsq.avg_bind_cpu_time_ms, 'N0'), - total_bind_cpu_time_ms = - FORMAT(qsq.total_bind_cpu_time_ms, 'N0'), - last_bind_cpu_time_ms = - FORMAT(qsq.last_bind_cpu_time_ms, 'N0'), - avg_optimize_duration_ms = - FORMAT(qsq.avg_optimize_duration_ms, 'N0'), - total_optimize_duration_ms = - FORMAT(qsq.total_optimize_duration_ms, 'N0'), - last_optimize_duration_ms = - FORMAT(qsq.last_optimize_duration_ms, 'N0'), - avg_optimize_cpu_time_ms = - FORMAT(qsq.avg_optimize_cpu_time_ms, 'N0'), - total_optimize_cpu_time_ms = - FORMAT(qsq.total_optimize_cpu_time_ms, 'N0'), - last_optimize_cpu_time_ms = - FORMAT(qsq.last_optimize_cpu_time_ms, 'N0'), - avg_compile_memory_mb = - FORMAT(qsq.avg_compile_memory_mb, 'N0'), - total_compile_memory_mb = - FORMAT(qsq.total_compile_memory_mb, 'N0'), - last_compile_memory_mb = - FORMAT(qsq.last_compile_memory_mb, 'N0'), - max_compile_memory_mb = - FORMAT(qsq.max_compile_memory_mb, 'N0'), + count_compiles = ' + + CONVERT + ( + nvarchar(max), + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.count_compiles, ''N0'')' + ELSE N'qsq.count_compiles' + END + N', + avg_compile_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.avg_compile_duration_ms, ''N0'')' + ELSE N'qsq.avg_compile_duration_ms' + END + N', + total_compile_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.total_compile_duration_ms, ''N0'')' + ELSE N'qsq.total_compile_duration_ms' + END + N', + last_compile_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.last_compile_duration_ms, ''N0'')' + ELSE N'qsq.last_compile_duration_ms' + END + N', + avg_bind_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.avg_bind_duration_ms, ''N0'')' + ELSE N'qsq.avg_bind_duration_ms' + END + N', + total_bind_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.total_bind_duration_ms, ''N0'')' + ELSE N'qsq.total_bind_duration_ms' + END + N', + last_bind_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.last_bind_duration_ms, ''N0'')' + ELSE N'qsq.last_bind_duration_ms' + END + N', + avg_bind_cpu_time_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.avg_bind_cpu_time_ms, ''N0'')' + ELSE N'qsq.avg_bind_cpu_time_ms' + END + N', + total_bind_cpu_time_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.total_bind_cpu_time_ms, ''N0'')' + ELSE N'qsq.total_bind_cpu_time_ms' + END + N', + last_bind_cpu_time_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.last_bind_cpu_time_ms, ''N0'')' + ELSE N'qsq.last_bind_cpu_time_ms' + END + N', + avg_optimize_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.avg_optimize_duration_ms, ''N0'')' + ELSE N'qsq.avg_optimize_duration_ms' + END + N', + total_optimize_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.total_optimize_duration_ms, ''N0'')' + ELSE N'qsq.total_optimize_duration_ms' + END + N', + last_optimize_duration_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.last_optimize_duration_ms, ''N0'')' + ELSE N'qsq.last_optimize_duration_ms' + END + N', + avg_optimize_cpu_time_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.avg_optimize_cpu_time_ms, ''N0'')' + ELSE N'qsq.avg_optimize_cpu_time_ms' + END + N', + total_optimize_cpu_time_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.total_optimize_cpu_time_ms, ''N0'')' + ELSE N'qsq.total_optimize_cpu_time_ms' + END + N', + last_optimize_cpu_time_ms = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.last_optimize_cpu_time_ms, ''N0'')' + ELSE N'qsq.last_optimize_cpu_time_ms' + END + N', + avg_compile_memory_mb = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.avg_compile_memory_mb, ''N0'')' + ELSE N'qsq.avg_compile_memory_mb' + END + N', + total_compile_memory_mb = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.total_compile_memory_mb, ''N0'')' + ELSE N'qsq.total_compile_memory_mb' + END + N', + last_compile_memory_mb = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.last_compile_memory_mb, ''N0'')' + ELSE N'qsq.last_compile_memory_mb' + END + N', + max_compile_memory_mb = ' + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsq.max_compile_memory_mb, ''N0'')' + ELSE N'qsq.max_compile_memory_mb' + END + ) + N', qsq.query_hash, qsq.batch_sql_handle, qsqt.statement_sql_handle, @@ -10684,238 +9147,243 @@ BEGIN WHERE x.n = 1 ORDER BY x.query_id - OPTION(RECOMPILE); + OPTION(RECOMPILE);'; + + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql, + N'@timezone sysname, @utc_offset_string nvarchar(max)', + @timezone, @utc_offset_string; - END; /*End query store query, format output = 1*/ + END; /*End compilation query section*/ ELSE BEGIN SELECT result = '#query_store_query is empty'; END; - END; - - IF @expert_mode = 1 - BEGIN - IF @rc > 0 - BEGIN - SELECT - @current_table = 'selecting resource stats'; - - SELECT - source = - 'resource_stats', - database_name = - DB_NAME(qsq.database_id), - qsq.query_id, - qsq.object_name, - total_grant_mb = - FORMAT(qsqt.total_grant_mb, 'N0'), - last_grant_mb = - FORMAT(qsqt.last_grant_mb, 'N0'), - min_grant_mb = - FORMAT(qsqt.min_grant_mb, 'N0'), - max_grant_mb = - FORMAT(qsqt.max_grant_mb, 'N0'), - total_used_grant_mb = - FORMAT(qsqt.total_used_grant_mb, 'N0'), - last_used_grant_mb = - FORMAT(qsqt.last_used_grant_mb, 'N0'), - min_used_grant_mb = - FORMAT(qsqt.min_used_grant_mb, 'N0'), - max_used_grant_mb = - FORMAT(qsqt.max_used_grant_mb, 'N0'), - total_ideal_grant_mb = - FORMAT(qsqt.total_ideal_grant_mb, 'N0'), - last_ideal_grant_mb = - FORMAT(qsqt.last_ideal_grant_mb, 'N0'), - min_ideal_grant_mb = - FORMAT(qsqt.min_ideal_grant_mb, 'N0'), - max_ideal_grant_mb = - FORMAT(qsqt.max_ideal_grant_mb, 'N0'), - qsqt.total_reserved_threads, - qsqt.last_reserved_threads, - qsqt.min_reserved_threads, - qsqt.max_reserved_threads, - qsqt.total_used_threads, - qsqt.last_used_threads, - qsqt.min_used_threads, - qsqt.max_used_threads - FROM #query_store_query AS qsq - JOIN #query_store_query_text AS qsqt - ON qsq.query_text_id = qsqt.query_text_id - AND qsq.database_id = qsqt.database_id - WHERE - ( - qsqt.total_grant_mb IS NOT NULL - OR qsqt.total_reserved_threads IS NOT NULL - ) - ORDER BY - qsq.query_id - OPTION(RECOMPILE); - - END; /*End resource stats, format output = 1*/ - ELSE - BEGIN - SELECT - result = - '#dm_exec_query_stats is empty'; - END; - END; - - IF @new = 1 + END; /*compilation stats*/ + + IF @rc > 0 BEGIN - IF EXISTS - ( - SELECT - 1/0 - FROM #query_store_wait_stats AS qsws - ) - AND @expert_mode = 1 - BEGIN - SELECT - @current_table = 'selecting wait stats by query'; - - SELECT - source = - 'query_store_wait_stats_by_query', - database_name = - DB_NAME(qsws.database_id), - qsws.plan_id, - x.object_name, - qsws.wait_category_desc, - total_query_wait_time_ms = - FORMAT(qsws.total_query_wait_time_ms, 'N0'), - total_query_duration_ms = - FORMAT(x.total_duration_ms, 'N0'), - avg_query_wait_time_ms = - FORMAT(qsws.avg_query_wait_time_ms, 'N0'), - avg_query_duration_ms = - FORMAT(x.avg_duration_ms, 'N0'), - last_query_wait_time_ms = - FORMAT(qsws.last_query_wait_time_ms, 'N0'), - last_query_duration_ms = - FORMAT(x.last_duration_ms, 'N0'), - min_query_wait_time_ms = - FORMAT(qsws.min_query_wait_time_ms, 'N0'), - min_query_duration_ms = - FORMAT(x.min_duration_ms, 'N0'), - max_query_wait_time_ms = - FORMAT(qsws.max_query_wait_time_ms, 'N0'), - max_query_duration_ms = - FORMAT(x.max_duration_ms, 'N0') - FROM #query_store_wait_stats AS qsws - CROSS APPLY - ( - SELECT - qsrs.avg_duration_ms, - qsrs.last_duration_ms, - qsrs.min_duration_ms, - qsrs.max_duration_ms, - qsrs.total_duration_ms, - qsq.object_name - FROM #query_store_runtime_stats AS qsrs - JOIN #query_store_plan AS qsp - ON qsrs.plan_id = qsp.plan_id - AND qsrs.database_id = qsp.database_id - JOIN #query_store_query AS qsq - ON qsp.query_id = qsq.query_id - AND qsp.database_id = qsq.database_id - WHERE qsws.plan_id = qsrs.plan_id - AND qsws.database_id = qsrs.database_id - ) AS x - ORDER BY - qsws.plan_id, - qsws.total_query_wait_time_ms DESC - OPTION(RECOMPILE); - - SELECT - @current_table = 'selecting wait stats in total'; - - SELECT - source = - 'query_store_wait_stats_total', - database_name = - DB_NAME(qsws.database_id), - qsws.wait_category_desc, - total_query_wait_time_ms = - FORMAT(SUM(qsws.total_query_wait_time_ms), 'N0'), - total_query_duration_ms = - FORMAT(SUM(x.total_duration_ms), 'N0'), - avg_query_wait_time_ms = - FORMAT(SUM(qsws.avg_query_wait_time_ms), 'N0'), - avg_query_duration_ms = - FORMAT(SUM(x.avg_duration_ms), 'N0'), - last_query_wait_time_ms = - FORMAT(SUM(qsws.last_query_wait_time_ms), 'N0'), - last_query_duration_ms = - FORMAT(SUM(x.last_duration_ms), 'N0'), - min_query_wait_time_ms = - FORMAT(SUM(qsws.min_query_wait_time_ms), 'N0'), - min_query_duration_ms = - FORMAT(SUM(x.min_duration_ms), 'N0'), - max_query_wait_time_ms = - FORMAT(SUM(qsws.max_query_wait_time_ms), 'N0'), - max_query_duration_ms = - FORMAT(SUM(x.max_duration_ms), 'N0') - FROM #query_store_wait_stats AS qsws - CROSS APPLY + SELECT + @current_table = 'selecting resource stats'; + + SET @sql = N''; + + SELECT + @sql = + CONVERT + ( + nvarchar(MAX), + N' + SELECT + source = + ''resource_stats'', + database_name = + DB_NAME(qsq.database_id), + qsq.query_id, + qsq.object_name, + total_grant_mb = ' + + + CONVERT ( - SELECT - qsrs.avg_duration_ms, - qsrs.last_duration_ms, - qsrs.min_duration_ms, - qsrs.max_duration_ms, - qsrs.total_duration_ms, - qsq.object_name - FROM #query_store_runtime_stats AS qsrs - JOIN #query_store_plan AS qsp - ON qsrs.plan_id = qsp.plan_id - AND qsrs.database_id = qsp.database_id - JOIN #query_store_query AS qsq - ON qsp.query_id = qsq.query_id - AND qsp.database_id = qsq.database_id - WHERE qsws.plan_id = qsrs.plan_id - AND qsws.database_id = qsrs.database_id - ) AS x - GROUP BY - qsws.wait_category_desc, - qsws.database_id - ORDER BY - SUM(qsws.total_query_wait_time_ms) DESC - OPTION(RECOMPILE); - + nvarchar(max), + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.total_grant_mb, ''N0'')' + ELSE N'qsqt.total_grant_mb' + END + + N', + last_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.last_grant_mb, ''N0'')' + ELSE N'qsqt.last_grant_mb' + END + + N', + min_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.min_grant_mb, ''N0'')' + ELSE N'qsqt.min_grant_mb' + END + + N', + max_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.max_grant_mb, ''N0'')' + ELSE N'qsqt.max_grant_mb' + END + + N', + total_used_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.total_used_grant_mb, ''N0'')' + ELSE N'qsqt.total_used_grant_mb' + END + + N', + last_used_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.last_used_grant_mb, ''N0'')' + ELSE N'qsqt.last_used_grant_mb' + END + + N', + min_used_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.min_used_grant_mb, ''N0'')' + ELSE N'qsqt.min_used_grant_mb' + END + + N', + max_used_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.max_used_grant_mb, ''N0'')' + ELSE N'qsqt.max_used_grant_mb' + END + + N', + total_ideal_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.total_ideal_grant_mb, ''N0'')' + ELSE N'qsqt.total_ideal_grant_mb' + END + + N', + last_ideal_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.last_ideal_grant_mb, ''N0'')' + ELSE N'qsqt.last_ideal_grant_mb' + END + + N', + min_ideal_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.min_ideal_grant_mb, ''N0'')' + ELSE N'qsqt.min_ideal_grant_mb' + END + + N', + max_ideal_grant_mb = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.max_ideal_grant_mb, ''N0'')' + ELSE N'qsqt.max_ideal_grant_mb' + END + + N', + total_reserved_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.total_reserved_threads, ''N0'')' + ELSE N'qsqt.total_reserved_threads' + END + + N', + last_reserved_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.last_reserved_threads, ''N0'')' + ELSE N'qsqt.last_reserved_threads' + END + + N', + min_reserved_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.min_reserved_threads, ''N0'')' + ELSE N'qsqt.min_reserved_threads' + END + + N', + max_reserved_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.max_reserved_threads, ''N0'')' + ELSE N'qsqt.max_reserved_threads' + END + + N', + total_used_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.total_used_threads, ''N0'')' + ELSE N'qsqt.total_used_threads' + END + + N', + last_used_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.last_used_threads, ''N0'')' + ELSE N'qsqt.last_used_threads' + END + + N', + min_used_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.min_used_threads, ''N0'')' + ELSE N'qsqt.min_used_threads' + END + + N', + max_used_threads = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsqt.max_used_threads, ''N0'')' + ELSE N'qsqt.max_used_threads' + END + ) + N' + FROM #query_store_query AS qsq + JOIN #query_store_query_text AS qsqt + ON qsq.query_text_id = qsqt.query_text_id + AND qsq.database_id = qsqt.database_id + WHERE + ( + qsqt.total_grant_mb IS NOT NULL + OR qsqt.total_reserved_threads IS NOT NULL + ) + ORDER BY + qsq.query_id + OPTION(RECOMPILE);' + ); + + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; END; + + EXECUTE sys.sp_executesql + @sql; - END; /*End wait stats, format output = 1*/ + END; /*End resource stats query*/ ELSE BEGIN SELECT result = - '#query_store_wait_stats is empty' + - CASE - WHEN ( - @product_version = 13 - AND @azure = 0 - ) - THEN ' because it''s not available < 2017' - WHEN EXISTS - ( - SELECT - 1/0 - FROM #database_query_store_options AS dqso - WHERE dqso.wait_stats_capture_mode_desc <> 'ON' - ) - THEN ' because you have it disabled in your Query Store options' - ELSE ' for the queries in the results' - END; + '#dm_exec_query_stats is empty'; END; - IF - ( - @sql_2022_views = 1 - AND @ags_present = 1 - ) + IF @new = 1 BEGIN IF @expert_mode = 1 BEGIN @@ -10923,40 +9391,313 @@ BEGIN ( SELECT 1/0 - FROM #query_store_replicas AS qsr - JOIN #query_store_plan_forcing_locations AS qspfl - ON qsr.replica_group_id = qspfl.replica_group_id - AND qsr.replica_group_id = qspfl.database_id + FROM #query_store_wait_stats AS qsws ) BEGIN + /* + Wait stats by query + */ SELECT - @current_table = '#query_store_replicas and #query_store_plan_forcing_locations'; + @current_table = 'selecting wait stats by query'; + SET @sql = N''; + SELECT + @sql = + CONVERT + ( + nvarchar(MAX), + N' + SELECT DISTINCT + source = + ''query_store_wait_stats_by_query'', database_name = - DB_NAME(qsr.database_id), - qsr.replica_group_id, - qsr.role_type, - qsr.replica_name, - qspfl.plan_forcing_location_id, - qspfl.query_id, - qspfl.plan_id, - qspfl.replica_group_id - FROM #query_store_replicas AS qsr - JOIN #query_store_plan_forcing_locations AS qspfl - ON qsr.replica_group_id = qspfl.replica_group_id - AND qsr.database_id = qspfl.database_id + DB_NAME(qsws.database_id), + qsws.plan_id, + x.object_name, + qsws.wait_category_desc, + total_query_wait_time_ms = ' + + + CONVERT + ( + nvarchar(max), + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsws.total_query_wait_time_ms, ''N0'')' + ELSE N'qsws.total_query_wait_time_ms' + END + + N', + total_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(x.total_duration_ms, ''N0'')' + ELSE N'x.total_duration_ms' + END + + N', + avg_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsws.avg_query_wait_time_ms, ''N0'')' + ELSE N'qsws.avg_query_wait_time_ms' + END + + N', + avg_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(x.avg_duration_ms, ''N0'')' + ELSE N'x.avg_duration_ms' + END + + N', + last_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsws.last_query_wait_time_ms, ''N0'')' + ELSE N'qsws.last_query_wait_time_ms' + END + + N', + last_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(x.last_duration_ms, ''N0'')' + ELSE N'x.last_duration_ms' + END + + N', + min_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsws.min_query_wait_time_ms, ''N0'')' + ELSE N'qsws.min_query_wait_time_ms' + END + + N', + min_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(x.min_duration_ms, ''N0'')' + ELSE N'x.min_duration_ms' + END + + N', + max_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(qsws.max_query_wait_time_ms, ''N0'')' + ELSE N'qsws.max_query_wait_time_ms' + END + + N', + max_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(x.max_duration_ms, ''N0'')' + ELSE N'x.max_duration_ms' + END + ) + N' + FROM #query_store_wait_stats AS qsws + CROSS APPLY + ( + SELECT + qsrs.avg_duration_ms, + qsrs.last_duration_ms, + qsrs.min_duration_ms, + qsrs.max_duration_ms, + qsrs.total_duration_ms, + qsq.object_name + FROM #query_store_runtime_stats AS qsrs + JOIN #query_store_plan AS qsp + ON qsrs.plan_id = qsp.plan_id + AND qsrs.database_id = qsp.database_id + JOIN #query_store_query AS qsq + ON qsp.query_id = qsq.query_id + AND qsp.database_id = qsq.database_id + WHERE qsws.plan_id = qsrs.plan_id + AND qsws.database_id = qsrs.database_id + ) AS x ORDER BY - qsr.replica_group_id - OPTION(RECOMPILE); + qsws.plan_id, + qsws.total_query_wait_time_ms DESC + OPTION(RECOMPILE);' + ); + + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql; + + /* + Wait stats in total + */ + SELECT + @current_table = 'selecting wait stats in total'; + + SET @sql = N''; + + SELECT + @sql = + CONVERT + ( + nvarchar(MAX), + N' + SELECT + source = + ''query_store_wait_stats_total'', + database_name = + DB_NAME(qsws.database_id), + qsws.wait_category_desc, + total_query_wait_time_ms = ' + + + CONVERT + ( + nvarchar(max), + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(qsws.total_query_wait_time_ms), ''N0'')' + ELSE N'SUM(qsws.total_query_wait_time_ms)' + END + + N', + total_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(x.total_duration_ms), ''N0'')' + ELSE N'SUM(x.total_duration_ms)' + END + + N', + avg_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(qsws.avg_query_wait_time_ms), ''N0'')' + ELSE N'SUM(qsws.avg_query_wait_time_ms)' + END + + N', + avg_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(x.avg_duration_ms), ''N0'')' + ELSE N'SUM(x.avg_duration_ms)' + END + + N', + last_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(qsws.last_query_wait_time_ms), ''N0'')' + ELSE N'SUM(qsws.last_query_wait_time_ms)' + END + + N', + last_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(x.last_duration_ms), ''N0'')' + ELSE N'SUM(x.last_duration_ms)' + END + + N', + min_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(qsws.min_query_wait_time_ms), ''N0'')' + ELSE N'SUM(qsws.min_query_wait_time_ms)' + END + + N', + min_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(x.min_duration_ms), ''N0'')' + ELSE N'SUM(x.min_duration_ms)' + END + + N', + max_query_wait_time_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(qsws.max_query_wait_time_ms), ''N0'')' + ELSE N'SUM(qsws.max_query_wait_time_ms)' + END + + N', + max_query_duration_ms = ' + + + CASE + WHEN @format_output = 1 + THEN N'FORMAT(SUM(x.max_duration_ms), ''N0'')' + ELSE N'SUM(x.max_duration_ms)' + END + ) + N' + FROM #query_store_wait_stats AS qsws + CROSS APPLY + ( + SELECT + qsrs.avg_duration_ms, + qsrs.last_duration_ms, + qsrs.min_duration_ms, + qsrs.max_duration_ms, + qsrs.total_duration_ms, + qsq.object_name + FROM #query_store_runtime_stats AS qsrs + JOIN #query_store_plan AS qsp + ON qsrs.plan_id = qsp.plan_id + AND qsrs.database_id = qsp.database_id + JOIN #query_store_query AS qsq + ON qsp.query_id = qsq.query_id + AND qsp.database_id = qsq.database_id + WHERE qsws.plan_id = qsrs.plan_id + ) AS x + GROUP BY + qsws.wait_category_desc, + qsws.database_id + ORDER BY + SUM(qsws.total_query_wait_time_ms) DESC + OPTION(RECOMPILE);' + ); + + IF @debug = 1 + BEGIN + PRINT LEN(@sql); + PRINT @sql; + END; + + EXECUTE sys.sp_executesql + @sql; END; ELSE BEGIN SELECT - result = 'Availability Group information is empty'; + result = + '#query_store_wait_stats is empty' + + CASE + WHEN + ( + @product_version = 13 + AND @azure = 0 + ) + THEN ' because it''s not available < 2017' + WHEN EXISTS + ( + SELECT + 1/0 + FROM #database_query_store_options AS dqso + WHERE dqso.wait_stats_capture_mode_desc <> 'ON' + ) + THEN ' because you have it disabled in your Query Store options' + ELSE ' for the queries in the results' + END; END; END; - END; + END; /*End wait stats queries*/ IF @expert_mode = 1 BEGIN @@ -10978,24 +9719,19 @@ BEGIN dqso.desired_state_desc, dqso.actual_state_desc, dqso.readonly_reason, - current_storage_size_mb = - FORMAT(dqso.current_storage_size_mb, ''N0''), - flush_interval_seconds = - FORMAT(dqso.flush_interval_seconds, ''N0''), - interval_length_minutes = - FORMAT(dqso.interval_length_minutes, ''N0''), - max_storage_size_mb = - FORMAT(dqso.max_storage_size_mb, ''N0''), + dqso.current_storage_size_mb, + dqso.flush_interval_seconds, + dqso.interval_length_minutes, + dqso.max_storage_size_mb, dqso.stale_query_threshold_days, - max_plans_per_query = - FORMAT(dqso.max_plans_per_query, ''N0''), + dqso.max_plans_per_query, dqso.query_capture_mode_desc,' + CASE WHEN ( @azure = 1 - OR @product_version > 13 + OR @product_version > 13 ) THEN N' dqso.wait_stats_capture_mode_desc,' @@ -11005,28 +9741,28 @@ BEGIN CASE WHEN ( - @azure = 1 - OR @product_version > 14 + @azure = 1 + OR @product_version > 14 ) THEN N' - capture_policy_execution_count = - FORMAT(dqso.capture_policy_execution_count, ''N0''), - capture_policy_total_compile_cpu_time_ms = - FORMAT(dqso.capture_policy_total_compile_cpu_time_ms, ''N0''), - capture_policy_total_execution_cpu_time_ms = - FORMAT(dqso.capture_policy_total_execution_cpu_time_ms, ''N0''), - capture_policy_stale_threshold_hours = - FORMAT(dqso.capture_policy_stale_threshold_hours, ''N0''),' + dqso.capture_policy_execution_count, + dqso.capture_policy_total_compile_cpu_time_ms, + dqso.capture_policy_total_execution_cpu_time_ms, + dqso.capture_policy_stale_threshold_hours,' ELSE N'' END - ); + ); SELECT - @sql += N' + @sql += + CONVERT + ( + nvarchar(MAX), + N' dqso.size_based_cleanup_mode_desc FROM #database_query_store_options AS dqso - OPTION(RECOMPILE);'; - + OPTION(RECOMPILE);' + ); IF @debug = 1 BEGIN @@ -11037,8 +9773,7 @@ BEGIN EXECUTE sys.sp_executesql @sql; END; - -END; /*End expert mode = 1, format output = 1*/ +END; /*End Expert Mode*/ IF @query_store_trouble = 1 BEGIN @@ -11073,6 +9808,7 @@ BEGIN SELECT x.all_done, x.period, + x.databases, x.support, x.help, x.problems, @@ -11119,6 +9855,35 @@ BEGIN ), all_done = 'brought to you by darling data!', + databases = + N'processed: ' + + CASE + WHEN @get_all_databases = 0 + THEN ISNULL(@database_name, N'None') + ELSE + ISNULL + ( + STUFF + ( + ( + SELECT + N', ' + + d.database_name + FROM #databases AS d + ORDER BY + d.database_name + FOR + XML + PATH(''), + TYPE + ).value('.', 'nvarchar(max)'), + 1, + 2, + N'' + ), + N'None' + ) + END, support = 'for support, head over to github', help = @@ -11172,8 +9937,35 @@ BEGIN ), all_done = 'https://www.erikdarling.com/', + databases = + N'skipped: ' + + ISNULL + ( + STUFF + ( + ( + SELECT + N', ' + + rbs.database_name + + N' (' + + rbs.reason + + N')' + FROM #requested_but_skipped_databases AS rbs + ORDER BY + rbs.database_name + FOR + XML + PATH(''), + TYPE + ).value('.', 'nvarchar(MAX)'), + 1, + 2, + N'' + ), + N'None' + ), support = - 'https://github.com/erikdarlingdata/DarlingData', + 'https://code.erikdarling.com', help = 'EXECUTE sp_QuickieStore @help = 1;', problems = @@ -11198,22 +9990,27 @@ BEGIN CATCH */ IF @current_table IS NOT NULL BEGIN - RAISERROR('error while %s with @expert mode = %i and format_output = %i', 11, 1, @current_table, @em, @fo) WITH NOWAIT; + RAISERROR('current dynamic activity', 10, 1) WITH NOWAIT; + RAISERROR('error while %s with @expert mode = %i and format_output = %i', 10, 1, @current_table, @em, @fo) WITH NOWAIT; END; - /* - Query that caused the error - */ + /* + Query that caused the error + */ IF @sql IS NOT NULL BEGIN - RAISERROR('offending query:', 10, 1) WITH NOWAIT; + RAISERROR('current dynamic sql:', 10, 1) WITH NOWAIT; RAISERROR('%s', 10, 1, @sql) WITH NOWAIT; END; - - /* - This reliably throws the actual error from dynamic SQL - */ - THROW; + + IF @debug = 1 + BEGIN + GOTO DEBUG; + END; + IF @debug = 0 + BEGIN; + THROW; + END; END CATCH; /* @@ -11297,6 +10094,10 @@ BEGIN @format_output, get_all_databases = @get_all_databases, + include_databases = + @include_databases, + exclude_databases = + @exclude_databases, workdays = @workdays, work_start = @@ -11414,7 +10215,40 @@ BEGIN work_start_utc = @work_start_utc, work_end_utc = - @work_end_utc; + @work_end_utc, + column_sql = + @column_sql, + param_name = + @param_name, + param_value = + @param_value, + temp_table = + @temp_table, + column_name = + @column_name, + data_type = + @data_type, + is_include = + @is_include, + requires_secondary_processing = + @requires_secondary_processing, + split_sql = + @split_sql; + + SELECT + table_name = '@ColumnDefinitions', + cd.* + FROM @ColumnDefinitions AS cd + WHERE cd.column_id LIKE '%15' + ORDER BY + cd.column_id; + + SELECT + table_name = '@FilterParameters', + fp.* + FROM @FilterParameters AS fp + ORDER BY + fp.parameter_name; IF EXISTS ( @@ -11438,6 +10272,75 @@ BEGIN result = '#databases is empty'; END; + + IF EXISTS + ( + SELECT + 1/0 + FROM #include_databases AS id + ) + BEGIN + SELECT + table_name = + '#include_databases', + id.* + FROM #include_databases AS id + ORDER BY + id.database_name + OPTION(RECOMPILE); + END; + ELSE + BEGIN + SELECT + result = + '#include_databases is empty'; + END; + + IF EXISTS + ( + SELECT + 1/0 + FROM #exclude_databases AS ed + ) + BEGIN + SELECT + table_name = + '#exclude_databases', + ed.* + FROM #exclude_databases AS ed + ORDER BY + ed.database_name + OPTION(RECOMPILE); + END; + ELSE + BEGIN + SELECT + result = + '#exclude_databases is empty'; + END; + + IF EXISTS + ( + SELECT + 1/0 + FROM #requested_but_skipped_databases AS rsdb + ) + BEGIN + SELECT + table_name = + '#requested_but_skipped_databases', + rsdb.* + FROM #requested_but_skipped_databases AS rsdb + ORDER BY + rsdb.database_name + OPTION(RECOMPILE); + END; + ELSE + BEGIN + SELECT + result = + '#requested_but_skipped_databases is empty'; + END; IF EXISTS ( @@ -11498,7 +10401,7 @@ BEGIN poi.* FROM #procedure_object_ids AS poi ORDER BY - poi.[object_id] + poi.object_id OPTION(RECOMPILE); END; ELSE @@ -12404,6 +11307,28 @@ BEGIN result = '#troubleshoot_performance is empty'; END; + IF EXISTS + ( + SELECT + 1/0 + FROM #query_hash_totals AS qht + ) + BEGIN + SELECT + table_name = + '#query_hash_totals', + qht.* + FROM #query_hash_totals AS qht + ORDER BY + qht.database_id + OPTION(RECOMPILE); + END; + ELSE + BEGIN + SELECT + result = + '#query_hash_totals is empty'; + END; RETURN; /*Stop doing anything, I guess*/ END; /*End debug*/ RETURN; /*Yeah sure why not?*/ diff --git a/sp_WhoIsActive Logging/README.md b/sp_WhoIsActive Logging/README.md new file mode 100644 index 00000000..23d63f65 --- /dev/null +++ b/sp_WhoIsActive Logging/README.md @@ -0,0 +1,83 @@ + + +# sp_WhoIsActive Logging + +This toolkit automates the collection and management of SQL Server activity data using Adam Machanic's popular sp_WhoIsActive stored procedure. It creates a comprehensive logging framework that captures server activity in daily tables and provides useful views for analysis. + +## Overview + +The sp_WhoIsActive Logging toolkit consists of several components: + +1. **Daily Table Creation**: Automatically creates tables named WhoIsActive_YYYYMMDD to store server activity data +2. **Data Collection**: Executes sp_WhoIsActive and logs output to the daily tables +3. **Data Retention**: Automatically manages retention by removing tables older than a specified period +4. **Analysis Views**: Creates views for querying across all tables and analyzing blocking chains +5. **Automated Collection**: Includes an Agent job for scheduling regular collection (default: every minute) + +## Prerequisites + +- Adam Machanic's sp_WhoIsActive stored procedure must be installed + - If you need to get or update: [https://github.com/amachanic/sp_whoisactive](https://github.com/amachanic/sp_whoisactive) + - If you get an error about @get_memory_info parameter, you need to update sp_WhoIsActive + +## Components + +The toolkit includes four scripts that should be executed in order: + +1. **01 sp_WhoIsActive Logging Views.sql**: Creates the stored procedure that manages the views +2. **02 sp_WhoIsActiveLogging Main.sql**: Creates the main logging procedure +3. **03 sp_WhoIsActiveLogging_Retention.sql**: Creates the data retention procedure +4. **04 sp_WhoIsActive Logging Agent Job.sql**: Creates the SQL Agent job for automated collection + +## Stored Procedures + +### sp_WhoIsActiveLogging_Main + +The main procedure that handles data collection and table management. + +| Parameter Name | Data Type | Default Value | Description | +|----------------|-----------|---------------|-------------| +| @RetentionPeriod | integer | 10 | Number of days to keep data | + +### sp_WhoIsActiveLogging_Retention + +Handles the removal of tables older than the specified retention period. + +| Parameter Name | Data Type | Default Value | Description | +|----------------|-----------|---------------|-------------| +| @RetentionPeriod | integer | 10 | Number of days to keep data | + +### sp_WhoIsActiveLogging_CreateViews + +Creates two views for data analysis (no parameters): +- **dbo.WhoIsActive**: UNION ALL of all WhoIsActive_YYYYMMDD tables +- **dbo.WhoIsActive_blocking**: Recursive CTE that traverses blocking chains + +## Usage Examples + +```sql +-- Run the main logging procedure with default retention (10 days) +EXECUTE dbo.sp_WhoIsActiveLogging_Main; + +-- Run the main logging procedure with custom retention (30 days) +EXECUTE dbo.sp_WhoIsActiveLogging_Main + @RetentionPeriod = 30; + +-- Manually run the retention procedure to clean up old tables +EXECUTE dbo.sp_WhoIsActiveLogging_Retention + @RetentionPeriod = 15; + +-- Recreate the views (useful after adding new tables) +EXECUTE dbo.sp_WhoIsActiveLogging_CreateViews; +``` + +## Notes + +- The scripts use the master database by default +- New tables are created daily with the format WhoIsActive_YYYYMMDD +- The Agent job runs on a one-minute schedule by default +- Views are automatically refreshed when new tables are created +- Old tables are automatically dropped based on the retention period + +Copyright 2025 Darling Data, LLC +Released under MIT license \ No newline at end of file