Improve compatibility profile performance and make profiles a part of central configuration

[rjmholt]

Compatibility profiles are currently very slow to start up and use opaque and duplicated configurations across rules.

Instead:

• They should be more modular so they can be more easily read in
• They should ideally have a custom parser for speed
• Rather than being configured by name, they should be specified by PS and OS version and defaults should be implicitly listed in each PSSA version
• Profiles should be lazily taken from NuGet rather than distributed with PSSA

[iRon7]

We are using PSScriptAnalyzer in our GitLab street and currently experiencing the issue described here:

After opening a new PowerShell session, the initial Invoke-ScriptAnalyzer with the following settings takes more than 15 secondes:

Measure-Command {
    Invoke-ScriptAnalyzer -Path .\Test.ps1 -Settings @{
        Rules = @{
            PSUseCompatibleCommands = @{
                Enable = $true
                TargetProfiles = @(
                    'win-8_x64_10.0.17763.0_5.1.17763.316_x64_4.0.30319.42000_framework',
                    'win-8_x64_10.0.17763.0_7.0.0_x64_3.1.2_core'
                )
            }
            PSUseCompatibleSyntax = @{
                Enable = $true
                TargetVersions = @(
                    '5.1',
                    '7.0'
                )
            }
        }
    }
}

Any successive invocation takes less than .5 seconds.
Is there any way to workaround this issue?

PSScriptAnalyzer version: 1.20
PowerShell version table (it also happens on PowerShell 7):

Name                           Value
----                           -----
PSVersion                      5.1.19041.2673
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.2673
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

[rjmholt]

Is there any way to workaround this issue?

The performance issues here occur because the profiles are stored in very large JSON files, which are parsed using reflection with Newtonsoft code. There's probably also some large object heap involvement too. Those files have to be loaded up on cold start, and even parsing them lazily doesn't avoid the huge memory consumption that probably drives much of the performance issues.

A workaround might be to amortise cold starts by turning this into a service shared across CI workers.

An interim solution would be to change out the JSON parsing code to use pre-generated stream parsing, probably using Microsoft's JSON parsing library.

The best solution would be to convert the compatibility profiles into a Sqlite DB and ship PSSA with bindings for that

[iRon7]

@rjmholt,

A workaround might be to amortise cold starts by turning this into a service shared across CI workers.

Thank you for the feedback. Unfortunately, this is quiet hard for us to implement.
Anyways, as we actually only using the script analyzer to automatically generate issues/emails on a commit and do not preventing it from still building a merge request, we now have script analyzer completely asynchrone with the rest.

From the reply from @bergmeister

I don't think so although I cannot speak for PS team. Your referenced issue is isolated to the workings of compatibility rules and not related to PSSA rewrite

Does that mean that I actually should create a separate issue for this?

[rjmholt]

Unfortunately, this is quiet hard for us to implement.

Yeah PSSA as a service is just something I suggest if no changes will be made to the compatibility rules. I suspect it would be easier to implement than rewriting the compatibility rules yourself.

I no longer work at Microsoft so I don't know what the plans or processes are beyond what you know already, but I don't imagine a separate issue would help here

[Glober777]

I'm also observing a performance issue which is related to the cold start. In my case, every time I run the tests, they start in a fresh PS Session, so I'm always hitting the fresh start penalty. Usually, the delay out of initialization is longer than all the PSSA related tests. It would be great, if there was a way to trigger the cold start process in the very beginning (i.e. before all the other tests and prep tasks) so that it would run asynchronously (i.e. in a separate thread or runspace) and by the time my script gets to running PSSA tests, those large JSONs should already be loaded, i.e.:

1. Initialize-PSSA (starts init task in the background and returns back the prompt on the main thread)
2. Prepare the folder for the tests (cleanup artifacts folder, check dependencies and update if needed, etc.)
3. Start all the other Tests (leaving PSSA tests to the end)
4. Wait-PSSAInit (a cmdlet that would hold the main thread until the PSSA is fully initialized)
5. Run PSSA Tests
... other steps ...