docs: clearer datastore layouts and init errors - #11449
Draft
lidel wants to merge 2 commits into
Draft
Conversation
Profile names and docs now say which store holds what: 'flatfs-levelds' (blocks in flatfs, everything else in leveldb) is the canonical name of the default layout, with 'flatfs' and 'flatfs-measure' kept as aliases. 'flatfs-pebbleds' and its '-measure' twin name the flatfs plus pebble layout that otherwise needs a hand-written Datastore.Spec. - docs: why flatfs holds only blocks, measure overhead note, Datastore.Spec example, profile sections - cmd/ipfs: datastore paragraph in 'ipfs init --help' - config: spec functions named after the profiles they back - test/cli: layout on disk, restart, aliases, default, measure metrics - test/sharness: profiles added to t0025 Closes #11423
The flatfs shard depth can only be set when a repo is created, by
passing a config file to 'ipfs init', and that route had traps.
- init: refuse a config without Identity.PrivKey ('ipfs config show'
output) on both 'ipfs init <file>' and 'ipfs daemon --init', and
validate Datastore.Spec before writing anything
- config profile apply: refuse profiles that change the on-disk layout
- fsrepo: mismatch error labels config and datastore_spec correctly
and says the layout is fixed at init
- docs: why and when to use next-to-last/3, the config-file procedure,
migration by moving data to a new repo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Just docs, better error handling and tests, nothing interesting.
Problem
Keeping flatfs for blocks with pebble instead of leveldb meant hand-writing
Datastore.Spec, and nothing said the defaultflatfsprofile stores pins, MFS, and provider records in leveldb. Setting the flatfs shard depth had traps of its own:ipfs initacceptedipfs config showoutput and built a repo that panics on start, a typo inshardFuncleft a half-made repo, and a hand-edited spec failed with its two values mislabeled.Fix
flatfs-pebbledsprofile (plus-measure); the default layout is nowflatfs-levelds, withflatfsandflatfs-measureas aliases; pebble profiles are marked experimental,-measureones as overheadipfs initrefuses a config without a private key and validatesDatastore.Specbefore writing;ipfs config profile applyrefuses layout changes; the mismatch error is labeled correctly and says the layout is fixed at initnext-to-last/3is worth it, and how to set it or move a repoDefaults are unchanged. Out of scope: re-sharding an existing repo in place.
Closes #11423
Related: #11186, the badgerds migration text now points at
flatfs-pebbleds; #11435, one reason the pebble profiles stay labeled experimental.Footnotes
go-ds-flatfs restrictions: content-addressed keys only, first-successful-writer-wins on concurrent writes, file-name-restricted keys. ↩