Auto generate manifests for newer versions

[felipecrs]

Description of the new feature/enhancement

To support some kind of CI that, based on previously set logic, can auto-generate manifests for newer versions of the packages.

Proposed technical implementation details

A very good example is the Chocolatey Automatic Package Updater Module: https://github.com/majkinetor/au

Based on custom rules, such as scripts for finding newer versions, URLs for downloading them, a CI can run continuously based on a scheduled trigger, and if it finds a newer version, it try to build a new manifest automatically, test it, and submit a PR (or push directly).

So then, we don't need to manually create manifests for every new version of an existing package.

[superusercode]

Scoop also has their own implementation of this: https://github.com/lukesampson/scoop/wiki/App-Manifest-Autoupdate
So does chocolatey: https://github.com/majkinetor/au and https://docs.chocolatey.org/en-us/create/automatic-packages

[felipecrs]

If we settle on an implementation archtecture I think the community would be excited to help. I also think this is something to put on the v1 backlog.

I was thinking of something like:

Every package (which has its own folder inside the Publisher) can have a file called update.yml. This file can be like

template:
  Id: GitHub.cli
  Name: GitHub CLI
  AppMoniker: gh
  InstallerType: MSI
updaters:
  - arch: x64
     uses: updater-github-releases
     with: 
       repository: 'cli/cli'
       regex: '*_windows_amd64.msi'
  - arch: x86
     uses: updater-github-releases
     with:
       repository: 'cli/cli'
       regex: '*_windows_386.msi'

And then, updater-github-releases would be a plugin ready to be used for the packages delivered through GitHub Releases.

This enables people to create every kind of plugin, and this way making them reusable without having to share code. People could create plugins which are applicable for a single distributor, but for that case, we can also have some generic plugins, such as updater-html-webscrap:

updaters:
  - arch: x64
     uses: updater-html-webscrap
     with:
       url: https://contoso.com/download
       versionFilter: '.list-group-item:contains(version)'
       downloadUrl: "https://contoso.com/download/contoso_${{ version }}_amd64.msi"

These plugins must return some values, such as version, x64.url, and x64.hash for instance. They can discover hashes by downloading or by any other mechanism, such as another field in the previous example hashFilter: ".list-group-item:contains(${{ version }}).hash".

These plugins could be npm modules for easier community engagement.

[felipecrs]

And then, a new project called winget-updater (could be written in TypeScript for example).

It needs to be a CLI which can be called passing the update.yml or an option for recursively check a folder for update.yml's, and then:

1. Run all the desired plugins which will return the required fields IF a new version is found.
2. IF a new version is found, generate a new manifest using the template area from the update.yml and the values returned by the plugin(s).

[felipecrs]

And finally, to write a CI pipeline (#1515) which would:

1. Run winget-updater on the manifests folder.
2. Submit a PR for each of the newly created manifests

And then the process goes as usual:

1. The Azure Pipelines start testing the PR
2. If passes, Azure Pipelines can check if this PR was opened by the updater bot and automatically merge
3. If not pass, leave the PR open for manual review.

[WtfJoke]

@felipecassiors sounds awesome if you ask me

[dfreilich]

+1 to the idea of adding in some automatic way of registering for updates of our package from Github. I was thinking of adding in a winget package for the Pack CLI (buildpacks/pack#707), and it would be great to minimize the number of steps necessary to keep it updated (so we don't have to do something like this: nodejs/node#34014)

[mimi89999]

I think that the just-install-updater-go (https://github.com/just-install/just-install-updater-go) is very nice. Rules are very simple to write and understand and they work for the vast majority of apps in the registry (https://github.com/just-install/just-install-updater-go/blob/master/jiup/rules/rules.go).

[denelon]

We have started the work for this feature. The first instance of this automation is related to packages that are distributed through a "vanity URL". If the installer changes, the SHA256 becomes invalid. We run a recurring scan on all manifests. If the package has changed, we will generate a new manifest and run the validation. Assuming all of that passes, the new manifest is merged, and the new version is available.