chore(ci): run cargo machete to catch unused dependencies

[7487]

Which issue does this PR close?

Closes #2410.

Rationale for this change

We do not currently check for unused dependencies in CI, so dead manifest entries accumulate — they still get resolved, downloaded, and often compiled. This adds cargo machete to the lint surface, following what DataFusion and other Arrow subprojects already do.

What changes are included in this PR?

Removed dependencies that are genuinely unused (no references anywhere in the source tree):

• ballista-core: aws-config, aws-credential-types (also dropped from the build-binary feature; S3 support goes through object_store's own builders), md-5, prost-types
• ballista-scheduler: prost-types
• ballista-executor: tracing (also dropped from the build-binary feature; tracing-subscriber/tracing-appender pull it in transitively)
• ballista-cli: tracing-appender (also dropped from the tui feature)
• pyballista: async-trait
• root workspace: prost-types (no longer referenced by any member)

Moved: scheduler's insta from [dependencies] to [dev-dependencies] — it is only used through ballista_core::assert_plan! in cfg(test) modules, so it no longer compiles into release scheduler builds.

Kept, with [package.metadata.cargo-machete] ignored entries and comments:

• ballista-scheduler: insta (macro expansion, see above) and tonic-prost (used by the keda-scaler gRPC code generated into OUT_DIR, which machete cannot see)
• ballista-executor: arrow (not imported directly, but pulls arrow's ipc_compression feature into the build so shuffle IPC compression — lz4 by default — works through datafusion's arrow re-export)
• ballista-cli: critical-section (provides the critical-section implementation ratatui-core needs at link time in the wasm32 web build)
• msrvcheck: time (never imported; pins the transitive time version so cargo keeps compiling)

CI wiring: new ci/scripts/rust_machete.sh, added to dev/rust_lint.sh, and a cargo-machete job in the Ballista Rust workflow (installs cargo-machete@0.9.2 via the already-pinned taiki-e/install-action).

Verified locally: cargo machete is clean; taplo format --check (0.10.0) passes; the CI clippy commands for the four touched crates pass with -D warnings (including --all-features, which exercises keda-scaler); cargo check -p ballista-scheduler -p ballista-executor -p ballista-core -p ballista --no-default-features --locked passes; scheduler tests using assert_plan! still pass; python/Cargo.lock stays in sync (cargo metadata --locked).

Are there any user-facing changes?

The implicit aws-config/aws-credential-types cargo features of ballista-core no longer exist, and ballista-core/ballista-executor lose a few transitive dependencies. No API or behavior changes.

🤖 Generated with Claude Code

[andygrove]

Thanks @7487

[NoahKusaba]

@7487 can you resolve conflicts and update branch, if you are still managing this change?
I think it's really important we get this merged in before the pending ballista version update.

[7487]

@NoahKusaba Done — merged upstream main and resolved the conflicts. Both were against the workspace-dependency refactor: ballista-core now takes chrono = { workspace = true } (aws-config/aws-credential-types stay removed — still no references in the source tree on current main), and ballista-cli keeps tracing-appender removed (still only used by the scheduler/executor binaries). Re-verified locally: cargo machete clean, taplo format --check passes, and both lockfiles are in sync.