🌟 Nova: [JSONL Exporter]#1083
Conversation
Implemented a new JSONL export format for trajectories. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 91f8b724ce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let mut redacted_msg = serde_json::to_value(msg).unwrap_or(serde_json::Value::Null); | ||
| if let Some(obj) = redacted_msg.as_object_mut() { | ||
| if let Some(content) = obj.get("content").and_then(|c| c.as_str()) { | ||
| let redacted_content = redactor.redact_text(content, surface::EXPORT).text; |
There was a problem hiding this comment.
When a trajectory contains secrets in MessageRecord.extra—for example extra.actions, provider response, sampling.extra, or other.run_result—this exports those fields unchanged because only content is replaced after serde_json::to_value(msg). Legacy/raw artifacts are view-redacted by exporters rather than trusted to be pre-redacted, and redact_trajectory_for_inspect treats these same extra fields as redaction surfaces, so bench inspect --format jsonl can leak tokens even though the conformance fixture only covers message content. Redact the whole JSON value or each extra field before to_string.
Useful? React with 👍 / 👎.
| formats.push(ExportFormat { | ||
| name: "jsonl", | ||
| tier: StabilityTier::Stable, | ||
| consumer: "log aggregation, streaming pipelines", | ||
| render: JsonlExporter::export, |
There was a problem hiding this comment.
In builds without jsonl-export, this registry entry is compiled out, and because FEATURE_GATED_FORMATS still lacks jsonl, is_export_format("jsonl") returns false. That sends bench inspect --format jsonl down the generic unknown-format path instead of the documented format_unavailable error with rebuild guidance, which makes the new feature undiscoverable in default builds. Add ("jsonl", "jsonl-export") to FEATURE_GATED_FORMATS.
Useful? React with 👍 / 👎.
1 participant
💡 The Spark:
Trajectories are great for programmatic analysis, but the raw
traj.jsonformat can be difficult to consume easily in some streaming logs or pipeline contexts without fully loading the large file structure. A one-line-per-message JSONL export provides a clean, easily parsable stream of events.🚀 The Feature:
Implemented a new
JsonlExporterbehind thejsonl-exportfeature flag. The exporter properly handles redactions by applyingRedactor::default_enabled().redact_textto the content, and leveragesserde_json::to_stringto create a lightweight, line-delimited export. Included a suite test and updateddocs/spec-export.mdto reflect the new feature in the Supported Formats table and the Discoverability outputs.🔮 The Potential:
The JSONL format makes it trivial for operators to integrate Maxwell's Daemon trajectory outputs with external log aggregation tools (like Elasticsearch, fluentd, datadog) and streaming bash tools like
jqthat naturally parse line-delimited JSON objects.Low. The feature is completely isolated under a new Cargo feature (
jsonl-export) and does not modify any core parsing or execution logic. It strictly implements the existingTrajectoryExportertrait boundary.PR created automatically by Jules for task 17649028088491621705 started by @madmax983