State sync fails on featureless chain

[dasanchez]

Hi!

We noticed that state sync fails with this error when a chain is run off a plain spawn scaffold:

panic: invalid denom: 

goroutine 281 [running]:
github.com/cosmos/cosmos-sdk/types.NewCoin({0x0, 0x0}, {0x4013558?})
        github.com/cosmos/cosmos-sdk@v0.50.13/types/coin.go:26 +0x48
github.com/cosmos/cosmos-sdk/x/bank/keeper.BaseKeeper.GetSupply({{{{0x40276b0, 0xc001ab7f00}, {0x3fd02e0, 0xc000e502b0}, {0x4030950, 0xc001690640}, {0x4013480, 0xc0003a2d90}, {0xc001862798, {...}, ...}, ...}, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.50.13/x/bank/keeper/keeper.go:204 +0xa5
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.StakingTokenSupply({{0x3fd02e0, 0xc000e502d0}, {0x40276b0, 0xc001ab7f00}, {0x4012c58, 0xc001690780}, {0x4023560, 0xc0012b2508}, {0x4027720, 0xc001862c78}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.50.13/x/staking/keeper/pool.go:94 +0x162
github.com/cosmos/cosmos-sdk/x/mint/keeper.Keeper.StakingTokenSupply(...)
        github.com/cosmos/cosmos-sdk@v0.50.13/x/mint/keeper/keeper.go:83
github.com/cosmos/cosmos-sdk/x/mint.BeginBlocker({_, _}, {{0x40276b0, 0xc001ab7f00}, {0x3fd02e0, 0xc000e502e0}, {0x3fe1d48, 0xc000852780}, {0x3fe9c50, 0xc0012b2f08}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.50.13/x/mint/abci.go:28 +0x299
github.com/cosmos/cosmos-sdk/x/mint.AppModule.BeginBlock(...)
        github.com/cosmos/cosmos-sdk@v0.50.13/x/mint/module.go:162
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x400a140, 0x5e37200}, {0x4028050, 0xc0059cb940}, {{0x0, 0x0}, {0xc0013e0200, 0xc}, 0x15f2d, ...}, ...})
        github.com/cosmos/cosmos-sdk@v0.50.13/types/module/module.go:778 +0x17c
github.com/rollchains/cosmos/app.(*ChainApp).BeginBlocker(...)
        github.com/rollchains/cosmos/app/app.go:974
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).beginBlock(0xc000dd7b08, 0xc0059c4d80?)
        github.com/cosmos/cosmos-sdk@v0.50.13/baseapp/baseapp.go:736 +0xb5
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc000dd7b08, {0x400a140, 0x5e37200}, 0xc0059c4d80)
        github.com/cosmos/cosmos-sdk@v0.50.13/baseapp/abci.go:763 +0xe74
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc000dd7b08, 0xc0059c4d80)
        github.com/cosmos/cosmos-sdk@v0.50.13/baseapp/abci.go:887 +0x16c
github.com/rollchains/cosmos/app.(*ChainApp).FinalizeBlock(0xc001289508?, 0xc0059c4d80?)
        github.com/rollchains/cosmos/app/app.go:942 +0x57
github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)
        github.com/cosmos/cosmos-sdk@v0.50.13/server/cmt_abci.go:44
github.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0xe0000000402b9c0?, {0x400a6c0?, 0x5e37200?}, 0xc0?)
        github.com/cometbft/cometbft@v0.38.17/abci/client/local_client.go:185 +0xc8
github.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc00150bec0, {0x400a6c0, 0x5e37200}, 0xc0059c4d80)
        github.com/cometbft/cometbft@v0.38.17/proxy/app_conn.go:104 +0x157
github.com/cometbft/cometbft/state.(*BlockExecutor).applyBlock(_, {{{0xb, 0x0}, {0x3364961, 0x7}}, {0xc002089120, 0xc}, 0x1, 0x15f2c, {{0xc001c14820, ...}, ...}, ...}, ...)
        github.com/cometbft/cometbft@v0.38.17/state/execution.go:224 +0x519
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyVerifiedBlock(...)
        github.com/cometbft/cometbft@v0.38.17/state/execution.go:202
github.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc0020ea000, 0x1)
        github.com/cometbft/cometbft@v0.38.17/blocksync/reactor.go:549 +0x14aa
github.com/cometbft/cometbft/blocksync.(*Reactor).SwitchToBlockSync.func1()
        github.com/cometbft/cometbft@v0.38.17/blocksync/reactor.go:161 +0x51
created by github.com/cometbft/cometbft/blocksync.(*Reactor).SwitchToBlockSync in goroutine 35
        github.com/cometbft/cometbft@v0.38.17/blocksync/reactor.go:159 +0xf3

The full sync log is here: https://paste.c-net.org/PretzelRimbaud

We did the following:

1. Run spawn new-chain cosmos --binary cosmosd --consensus proof-of-stake
2. Select no features
3. Run make sh-testnet and restart the node after setting snapshot-interval to 100 and snapshot-keep-recent to 2 in app.toml
4. Attempt to state sync from a new node locally

[reecepbcups]

This invalid denom: issue is EVM logic specific, so I would make sure the other node uses the same binary as well with EVM off, and that the binary was properly put in the system PATH. I do not see this in the IAVL storage logs though, so very odd.

I tested state sync in the past but not any time recently, especially since adding EVM support

note: spawn is no longer maintained due to dropped support from the ICL, so it would rely on your debugging here and a PR to resolve