Add direct HuggingFace safetensors loader for Gemma 4 (E2B/E4B)

[ssfdre38]

Summary

Adds a new code path to load Gemma 4 weights directly from HuggingFace *.safetensors files, bypassing the BlobStore conversion step. This avoids any potential weight precision loss from format conversion and lets you use freshly downloaded HF checkpoints without a separate conversion tool.

Changes

New files

• io/safetensors.h / io/safetensors.cc — SafetensorsIndex class: scans a directory for *.safetensors shards, parses the 8-byte LE header + JSON, and provides ReadTensor() via seek-based I/O. Handles both single-file and sharded (model.safetensors.index.json) checkpoints.
• gemma/load_safetensors.cc — WeightsPtrs::LoadFromSafetensors(): maps HF tensor names to gemma.cpp MatPtr fields. Handles Q/K/V concat, gate/up-proj concat, o_proj direct load, and per-layer token embedding transpose ([V, L*D] → [L*D, V]). Calls Fixup() at the end.

Modified files

• gemma/weights.h — adds LoadFromSafetensors() public declaration
• gemma/model_store.h / model_store.cc — adds ModelStore(const ModelConfig&, const Path& tokenizer_path) constructor for the safetensors path (reads tokenizer directly from file, leaves scales_ empty)
• gemma/gemma.h / gemma.cc — adds Gemma(ModelConfig, tokenizer, safetensors_dir, InferenceArgs, ThreadingContext) constructor; changes BlobReader reader_ to unique_ptr<BlobReader> to allow null when not using BlobStore
• gemma/gemma_args.h — adds --safetensors (directory path) and --model_spec (e.g. gemma4-e4b-bf16-it) flags to LoaderArgs
• gemma/run.cc — wires new flags into Run() with a conditional branch (uses unique_ptr<Gemma> to avoid copy/move)
• CMakeLists.txt — adds new source files; links nlohmann_json to libgemma

Usage

./gemma --safetensors /path/to/gemma-4-e4b-it \
        --model_spec   gemma4-e4b-bf16-it \
        --tokenizer    /path/to/tokenizer.model \
        --prompt       "Hello!"

The --model_spec specifier uses the existing ModelConfig(std::string) format: {model-prefix}-{type}-{wrapping} e.g. gemma4-e2b-bf16-it or gemma4-e4b-bf16-it.

Tested

• Builds cleanly (MSVC + ninja) with the existing CMake setup
• --help shows both new flags with descriptions
• E4B multimodal HF checkpoint (model.language_model.* prefix, 2130 tensors, 42 layers): loads fully, prompt processing begins
• Both single-shard and multi-shard (*.index.json) layouts supported by SafetensorsIndex

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[ssfdre38]

im part of cla already