Git Resource

Tracks commits, tags, or branches in a git repository.

Source Configuration

Field Name	Description
version_type (Optional)	commits (Default): Resource will return commits from the specified branch. Can also be used to find tags on a branch. tags: Resource will find matching tags from the git repo. branches: Resource will return a list of branches from the git repo.

The following fields are used by all version_type's.

Field Name	Description
uri (Required)	The location of the repository.
private_key (Optional)	Private key to use when using an ssh@ format uri. Example: private_key: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY-----
private_key_user (Optional)	Enables setting User in the ssh config.
private_key_passphrase (Optional)	To unlock private_key if it is protected by a passphrase.
forward_agent (Optional)	Enables ForwardAgent SSH option when set to true. Useful when using proxy/jump hosts. Defaults to false.
username (Optional)	Username for HTTP(S) auth when pulling/pushing. This is needed when only HTTP/HTTPS protocol for git is available (which does not support private key auth) and auth is required.
password (Optional)	Password for HTTP(S) auth when pulling/pushing.
skip_ssl_verification (Optional)	Skips git ssl verification by exporting GIT_SSL_NO_VERIFY=true.
submodule_credentials (Optional)	List of credentials for HTTP(s) or SSH auth when pulling git submodules which are not stored in the same git server as the container repository or are protected by a different private key. http(s) credentials: host : The host to connect to. Note that host is specified with no protocol extensions. username : Username for HTTP(S) auth when pulling submodule. password : Password for HTTP(S) auth when pulling submodule. ssh credentials: url : Submodule url, as specified in the .gitmodule file. Support full or relative ssh url. private_key : Private key for SSH auth when pulling submodule. private_key_passphrase : Optional. To unlock private_key if it is protected by a passphrase. example: submodule_credentials: # http(s) credentials - host: github.com username: git-user password: git-password # ssh credentials - url: git@github.com:org-name/repo-name.git private_key: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----- private_key_passphrase: ssh-passphrase # (optional) # ssh credentials with relative url - url: ../org-name/repo-name.git private_key: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----- private_key_passphrase: ssh-passphrase # (optional)
git_config (Optional)	Specified as a list of pairs name and value. It will configure git global options, setting each name with each value. This can be useful to set options like credential.helper or similar. See the git-config(1) manual page for more information and documentation of existing git options. Example: git_config: - name: push.recurseSubmodules value: "no"
disable_ci_skip (Optional)	Allows for commits that have been labeled with [ci skip] or [skip ci] previously to be discovered by the resource.
commit_verification_keys (Optional)	Array of GPG public keys that the resource will check against to verify the commit (details below).
commit_verification_key_ids (Optional)	Array of GPG public key ids that the resource will check against to verify the commit (details below). The corresponding keys will be fetched from the key server specified in gpg_keyserver. The ids can be short id, long id or fingerprint.
gpg_keyserver (Optional)	GPG keyserver to download the public keys from. Defaults to hkp://keyserver.ubuntu.com/.
git_crypt_key (Optional)	Base64 encoded git-crypt key. Setting this will unlock / decrypt the repository with git-crypt. To get the key simply execute git-crypt export-key -- - | base64 in an encrypted repository.
https_tunnel (Optional)	Information about an HTTPS proxy that will be used to tunnel SSH-based git commands over. Has the following sub-properties: proxy_host: Required. The host name or IP of the proxy server proxy_port: Required. The proxy server's listening port proxy_user: Optional. If the proxy requires authentication, use this username proxy_password: Optional. If the proxy requires authenticate, use this password
debug (Optional)	Set to true to enable. Sets the following for check/get/put steps of the resource. Secrets may not be correctly redacted due the JSON encoding of longer secret strings. set -x export GIT_TRACE=1 export GIT_TRACE_PACKFILE=1 export GIT_CURL_VERBOSE=1

The following fields are used exclusively by version_type: commits.

Field Name	Description
branch (Optional)	The branch to track. This is optional if the resource is only used in get steps; however, it is required when used in a put step. If unset, get steps will checkout the repository's default branch; usually master but could be different.
paths (Optional)	If specified (as a list of glob patterns), only changes to the specified files will yield new versions from check. Example: - name: repo type: git source: paths: - some-folder/* - another/folder/path/*
sparse_paths (Optional)	If specified (as a list of glob patterns), only these paths will be checked out. Should be used with paths to only trigger on desired paths. paths and sparse_paths may be the same or you can configure sparse_paths to check out other paths. Example: - name: repo type: git source: paths: - some-folder/* - another/folder/path/* sparse_paths: - some-folder/* - another/folder/path/*
ignore_paths (Optional)	The inverse of paths; changes to the specified files are ignored. Note that if you want to push commits that change these files via a put, the commit will still be "detected", as check and put both introduce versions. To avoid this you should define a second resource that you use for commits that change files that you don't want to feed back into your pipeline - think of one as read-only (with ignore_paths) and one as write-only (which shouldn't need it). Example: - name: repo type: git source: ignore_paths: - some-folder/* - another/folder/path/*
tag_filter (Optional)	If specified, the resource will only detect commits that have a tag matching the expression that have been made against the branch. Patterns are glob(7) compatible (as in, bash compatible).
tag_regex (Optional)	If specified, the resource will only detect commits that have a tag matching the expression that have been made against the branch. Patterns are grep compatible (extended matching enabled, matches entire lines only). Ignored if tag_filter is also specified.
tag_behaviour (Optional)	If match_tagged (the default), then the resource will only detect commits that are tagged with a tag matching tag_regex and tag_filter, and match all other filters. If match_tag_ancestors, then the resource will only detect commits matching all other filters and that are ancestors of a commit that are tagged with a tag matching tag_regex and tag_filter.
fetch_tags (Optional)	If true the flag --tags will be used to fetch all tags in the repository. If false no tags will be fetched.
commit_filter (Optional)	Object containing commit message filters exclude: Optional. Array containing strings that should cause a commit to be skipped exclude_all_match: Optional. Boolean wheater it should match all the exclude filters "AND", default: false include: Optional. Array containing strings that MUST be included in commit messages for the commit to not be skipped include_all_match: Optional. Boolean wheater it should match all the include filters "AND", default: false Note: You must escape any regex sensitive characters, since the string is used as a regex filter. For example, using [skip deploy] or [deploy skip] to skip non-deployment related commits in a deployment pipeline: commit_filter: exclude: ["\\[skip deploy\\]", "\\[deploy skip\\]"]
version_depth (Optional)	The number of versions to return when performing a check
search_remote_refs (Optional)	True to search remote refs for the input version when checking out during the get step. This can be useful during the get step after a put step for unconventional workflows. One example workflow is the refs/for/<branch> workflow used by gerrit which 'magically' creates a refs/changes/nnn reference instead of the straight forward refs/for/<branch> reference that a git remote would usually create. See also out params.refs_prefix.

The following fields are used exclusively by version_type: tags.

Field Name	Description
tag_filters (Optional)	A list of glob patterns used to filter tags. Only matching tags will be returned. Patterns are glob(7) compatible (as in, bash compatible). If you're only specifying one glob pattern you can use tag_filter.
tag_regex (Optional)	Regex pattern used to filter tags. Only matching tags will be returned. Patterns are grep compatible (extended matching enabled). Ignored if tag_filter(s) is also specified.
tag_sort (Optional)	Sorting is applied after filtering. Accepts the following values: creatordate (Default): Uses git's built-in sorting by the creation date of the tag semver: Uses sort -V to sort all matching tags.

The following fields are used exclusively by version_type: branches.

Field Name	Description
branch_filters (Optional)	A list of glob patterns used to filter branches. Only matching branches will be returned. Patterns are glob(7) compatible (as in, bash compatible).
branch_regex (Optional)	Regex pattern used to filter branches. Only matching branches will be returned. Patterns are grep compatible (extended matching enabled). Ignored if branch_filters is also specified.

Example

Resource configuration for a private repo with an HTTPS proxy:

resources:
- name: source-code
  type: git
  source:
    uri: git@github.com:concourse/git-resource.git
    branch: master
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----
    git_config:
    - name: core.bigFileThreshold
      value: 10m
    disable_ci_skip: true
    git_crypt_key: AEdJVEN...snip...AAA==
    https_tunnel:
      proxy_host: proxy-server.mycorp.com
      proxy_port: 3128
      proxy_user: myuser
      proxy_password: myverysecurepassword

Resource configuration for a private repo with a private submodule from different git server:

resources:
- name: source-code
  type: git
  source:
    uri: git@github.com:concourse/git-resource.git
    branch: master
    submodule_credentials:
    - host: some.other.git.server
      username: user
      password: verysecurepassword
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----

Fetching a repo with only 100 commits of history:

- get: source-code
  params: {depth: 100}

Pushing local commits to the repo:

- get: some-other-repo
- put: source-code
  params: {repository: some-other-repo}

Fetching a repo pinned to a specific commit:

resources:
- name: source-code
  type: git
  source:
    uri: git@github.com:concourse/git-resource.git
    branch: master
  version:
    ref: commit-sha

Behavior

The behavior of the resource changes based on the specified version_type. Expand the relevant section to learn more about how the check/get/put steps work for each version_type.

version_type: commits

check: Check for new commits

The repository is cloned (or pulled if already present), and any commits from the given version on are returned. If no version is given, the ref for HEAD is returned.

Any commits that contain the string [ci skip] will be ignored. This allows you to commit to your repository without triggering a new version.

get: Clone the repository, at the given ref

Clones the repository to the destination, and locks it down to a given ref. It will return the same given ref as version.

git-crypt encrypted repositories will automatically be decrypted, when the correct key is provided set in source.git_crypt_key.

get Parameters

Field Name	Description
depth Optional	If a positive integer is given, shallow clone the repository using the --depth option. To prevent newer commits that do not pass a paths filter test from skewing the cloned history away from version.ref, this resource will automatically deepen the clone until version.ref is found again. It will deepen with exponentially increasing steps until a maximum of 127 + depth commits or else resort to unshallow the repository.
fetch Optional	Additional branches to fetch and make available in the cloned repository. This is useful when you need to perform operations like rebasing or merging with branches other than the one being checked out. Specify as a list of branch names. Each branch will be fetched from origin and made available as a local branch. Example: Fetching additional branches for rebasing - get: source-code params: fetch: - develop - main - feature/experimental After the get step completes, you can perform operations like: - task: rebase-on-main config: platform: linux inputs: - name: source-code run: path: sh args: - -exc - | cd source-code git rebase main
fetch_tags Optional	If true the flag --tags will be used to fetch all tags in the repository. If false no tags will be fetched. Will override fetch_tags source configuration if defined.
submodules Optional	If none, submodules will not be fetched. If specified as a list of paths, only the given paths will be fetched. If not specified, or if all is explicitly specified, all submodules are fetched.
submodule_recursive Optional	If false, a flat submodules checkout is performed. If not specified, or if true is explicitly specified, a recursive checkout is performed.
submodule_remote Optional	If true, the submodules are checked out for the specified remote branch specified in the .gitmodules file of the repository. If not specified, or if false is explicitly specified, the tracked sub-module revision of the repository is used to check out the submodules.
disable_git_lfs Optional	If true, will not fetch Git LFS files.
clean_tags Optional	If true all incoming tags will be deleted. This is useful if you want to push tags, but have reasonable doubts that the tags cached with the resource are outdated. The default value is false.
short_ref_format Optional	When populating .git/short_ref use this printf format. Defaults to %s.
timestamp_format Optional	When populating .git/commit_timestamp use this options to pass to git log --date. Defaults to iso8601.
describe_ref_options Optional	When populating .git/describe_ref use this options to call git describe. Defaults to --always --dirty --broken.
all_branches Optional	If true the flag --single-branch will be excluded and all branches will be fetched from the repository. If false or not specified, only a single branch (either source.branch or the default branch) will be fetched.
debug (Optional)	Set to true to enable. Sets the following for check/get/put steps of the resource. Secrets may not be correctly redacted due the JSON encoding of longer secret strings. set -x export GIT_TRACE=1 export GIT_TRACE_PACKFILE=1 export GIT_CURL_VERBOSE=1

GPG signature verification

If commit_verification_keys or commit_verification_key_ids is specified in the source configuration, it will additionally verify that the resulting commit has been GPG signed by one of the specified keys. It will error if this is not the case.

Additional files populated

• .git/ref: Version reference detected and checked out. It will usually contain the commit SHA-1 ref, but also the detected tag name when using tag_filter or tag_regex.

• .git/commit: Full SHA-1 commit hash.

• .git/short_ref: Short (first seven characters) of the .git/ref. Can be templated with short_ref_format parameter.

• .git/branch: Name of the original branch that was cloned.

• .git/tags: Comma-separated list of tags associated with this commit.

• .git/author: Commit author name.

• .git/author_date: Timestamp when the author originally created the commit.

• .git/committer: For committer notification on failed builds. This special file .git/committer which is populated with the email address of the author of the last commit. This can be used together with a resource to send notifications in an on_failure step.

• .git/committer_name: Name of the commit author.

• .git/committer_date: Timestamp when the commit was added to the repository.

• .git/commit_message: For publishing the Git commit message on successful builds.

• .git/commit_timestamp: For tagging builds with a timestamp.

• .git/describe_ref: Version reference detected and checked out. Can be templated with describe_ref_options parameter. By default, it will contain the <latest annoted git tag>-<the number of commit since the tag>-g<short_ref> (eg. v1.6.2-1-g13dfd7b). If the repo was never tagged before, this falls back to a short commit SHA-1 ref.

• .git/url: Web URL to view the commit (if applicable).

• .git/metadata.json: Complete metadata object in JSON format containing all metadata fields.

put: Push to a repository

Push the checked-out reference to the source's URI and branch. All tags are also pushed to the source. If a fast-forward for the branch is not possible and the rebase parameter is not provided, the push will fail.

Parameters

Field Name	Description
repository Required	The path of the repository to push to the source.
rebase Optional	If pushing fails with non-fast-forward, continuously attempt rebasing and pushing.
rebase_strategy Optional	Merge strategy to use during rebase (e.g., recursive, ort, octopus, ours, subtree). Passed via --strategy flag.
rebase_strategy_option Optional	Strategy-specific options passed via -X flag. Can be a string (space-separated) or array of options (e.g., theirs, ours, ignore-space-change).
merge Optional	If pushing fails with non-fast-forward, continuously attempt to merge remote to local before pushing. Only one of merge or rebase can be provided, but not both.
returning Optional	When passing the merge flag, specify whether the merge commit or the original, unmerged commit should be passed as the output ref. Options are merged and unmerged. Defaults to merged.
tag Optional	If this is set then HEAD will be tagged. The value should be a path to a file containing the name of the tag.
only_tag Optional	When set to 'true' push only the tags of a repo.
tag_prefix Optional	If specified, the tag read from the file will be prepended with this string. This is useful for adding v in front of version numbers.
force Optional	When set to 'true' this will force the branch to be pushed regardless of the upstream state.
annotate Optional	If specified the tag will be an annotated tag rather than a lightweight tag. The value should be a path to a file containing the annotation message.
notes Optional	If this is set then notes will be added to HEAD to the refs/notes/commits ref. The value should be a path to a file containing the notes.
branch Optional	The branch to push commits. Note that the version produced by the put step will be picked up by subsequent get steps even if the branch differs from the branch specified in the source. To avoid this, you should use two resources of read-only and write-only.
refs_prefix Optional	Allows pushing to refs other than heads. Defaults to refs/heads. Useful when paired with source.search_remote_refs in cases where the git remote renames the ref you pushed.
push_options (Optional)	An array of strings that will be passed as --push-option flags when pushing to the repository. Example: - push-option-1 - push-option-2
debug (Optional)	Set to true to enable. Sets the following for check/get/put steps of the resource. Secrets may not be correctly redacted due the JSON encoding of longer secret strings. set -x export GIT_TRACE=1 export GIT_TRACE_PACKFILE=1 export GIT_CURL_VERBOSE=1

version_type: tags

check: Check for new tags

Checks for new tags, filtering by tag_filters or tag_regex if specified. Each version emitted will be the tag and the ref it points to.

get: Clone the repository at the given tag

Shallow clones the repository to the destination, checking out the given tag in a detached HEAD state.

git-crypt encrypted repositories will automatically be decrypted, when the correct key is provided set in source.git_crypt_key.

get Parameters

Field Name	Description
submodules Optional	If none, submodules will not be fetched. If specified as a list of paths, only the given paths will be fetched. If not specified, or if all is explicitly specified, all submodules are fetched.
submodule_recursive Optional	If false, a flat submodules checkout is performed. If not specified, or if true is explicitly specified, a recursive checkout is performed.
submodule_remote Optional	If true, the submodules are checked out for the specified remote branch specified in the .gitmodules file of the repository. If not specified, or if false is explicitly specified, the tracked sub-module revision of the repository is used to check out the submodules.
disable_git_lfs Optional	If true, will not fetch Git LFS files.
short_ref_format Optional	When populating .git/short_ref use this printf format. Defaults to %s.
timestamp_format Optional	When populating .git/commit_timestamp use this options to pass to git log --date. Defaults to iso8601.
debug (Optional)	Set to true to enable. Sets the following for check/get/put steps of the resource. Secrets may not be correctly redacted due the JSON encoding of longer secret strings. set -x export GIT_TRACE=1 export GIT_TRACE_PACKFILE=1 export GIT_CURL_VERBOSE=1

GPG signature verification

If commit_verification_keys or commit_verification_key_ids is specified in the source configuration, it will additionally verify that the resulting commit has been GPG signed by one of the specified keys. It will error if this is not the case.

Additional files populated

• .git/tag: Tag detected and checked out.

• .git/ref: Full SHA-1 commit hash.

• .git/short_ref: Short (first seven characters) of the .git/ref. Can be templated with short_ref_format parameter.

• .git/author: Commit author name.

• .git/author_date: Timestamp when the author originally created the commit.

• .git/committer: For committer notification on failed builds. This special file .git/committer which is populated with the email address of the author of the last commit. This can be used together with a resource to send notifications in an on_failure step.

• .git/committer_name: Name of the commit author.

• .git/committer_date: Timestamp when the commit was added to the repository.

• .git/commit_message: For publishing the Git commit message on successful builds.

• .git/commit_timestamp: For tagging builds with a timestamp.

• .git/url: Web URL to view the commit (if applicable).

• .git/metadata.json: Complete metadata object in JSON format containing all metadata fields.

put: No-op

There is no implementation for this step. It will error if you try to use it. If you want to push new tags, use version_type: commits.

version_type: branches

check: Check for new branches

The list of remote branches are enumerated, filtered by branch_filters or branch_regex if specified, and compared to the existing set of branches. If any branches are new or removed, a new version is emitted. Branches are sorted lexicographically using sort before comparing.

If no branches are found a special NONE version is emitted.

get: List the given branches

Produces a branches.json file containing a JSON array of the branches from the given version. Example of the contents of the file:

[
  "feature/add-button",
  "feature/refactor-model",
  "fix/ui-bug"
]

put: No-op

There is no implementation for this step. It will error if you try to use it.

Development

Prerequisites

• golang is required - version 1.9.x is tested; earlier versions may also work.
• docker is required - version 17.06.x is tested; earlier versions may also work.

Running the tests

The tests have been embedded with the Dockerfile; ensuring that the testing environment is consistent across any docker enabled platform. When the docker image builds, the test are run inside the docker container, on failure they will stop the build.

Run the tests with the following command:

docker build -t git-resource --target tests .

Note about the integration tests

If you want to run the integration tests, a bit more work is required. You will require an actual git repo to which you can push and pull, configured for SSH access. To do this, add two files to integration-tests/ssh (note that names are important):

• test_key: This is the private key used to authenticate against your repo.
• test_repo: This file contains one line of the form test_repo_url[#test_branch]. If the branch is not specified, it defaults to master. For example, git@github.com:concourse-git-tester/git-resource-integration-tests.git or git@github.com:concourse-git-tester/git-resource-integration-tests.git#testing

Contributing

Please make all pull requests to the master branch and ensure tests pass locally.