Skip to content

perf: improve sync command performance - #3766

Merged
olaszakos merged 10 commits into
masterfrom
akos/asset-upload-strategies
May 27, 2024
Merged

perf: improve sync command performance#3766
olaszakos merged 10 commits into
masterfrom
akos/asset-upload-strategies

Conversation

@olaszakos

@olaszakos olaszakos commented May 22, 2024

Copy link
Copy Markdown
Contributor

Description

This PR has the following changes:

  1. Make chunk creation faster, by increasing parallelization 4=>25, improves performance significantly for lots of small assets
  2. Make asset properties query faster by parallelization (major time save for canisters that have a ton of assets, the internetcomputer.org canister has 11k+ assets)
  3. icx-asset: Support log levels, and default to info

How Has This Been Tested?

I have not added any new tests, we are using this already with icx-asset sync for dfinity/portal PR preview deployments.

Deployment time has gone down from ~40m to <10m.

Checklist:

  • The title of this PR complies with Conventional Commits.
  • I have edited the CHANGELOG accordingly.
  • I have made corresponding changes to the documentation.

@olaszakos olaszakos changed the title feat(icx-asset)!: improve upload/sync commands feat(icx-asset)!: improve sync command performance May 23, 2024
@olaszakos olaszakos changed the title feat(icx-asset)!: improve sync command performance feat(icx-asset): improve sync command performance May 23, 2024
@olaszakos
olaszakos marked this pull request as ready for review May 23, 2024 10:04
@olaszakos
olaszakos requested review from a team and chenyan-dfinity as code owners May 23, 2024 10:04
@olaszakos olaszakos changed the title feat(icx-asset): improve sync command performance feat: improve sync command performance May 24, 2024

@ghost ghost left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a note to the changelog. This kind of performance improvement is exciting!

@olaszakos olaszakos changed the title feat: improve sync command performance perf: improve sync command performance May 27, 2024
@olaszakos
olaszakos merged commit f3c5b68 into master May 27, 2024
@olaszakos
olaszakos deleted the akos/asset-upload-strategies branch May 27, 2024 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant