Add support for basic camera streams [SYNTH-10]

[PepperLola]

Task

SYNTH-10

Added the ability to add cameras to the robot and both preview their streams from within Fission and stream them to the robot code via WebSocket. Since HALSim doesn't support cameras, this involves spinning up another WebSocket server on the robot side and streaming the video feed to it from Fission.

Verification

Spawn a robot in Fission and set the brain to the WPILib one. Configure cameras on the robot (within robot configuration - place camera, edit settings). Set up code sim by following the document added in #1360 and keep track of the mean brightness (calculated on the robot code side) in the simulator.

---

Before merging, ensure the following criteria are met:

• All acceptance criteria outlined in the ticket are met.
• Necessary test cases have been added and updated.
• A feature toggle or safe disable path has been added (if applicable).
• User-facing polish:
  • Ask: "Is this ready-looking?"
• Cross-linking between Jira and GitHub:
  • PR links to the relevant Jira issue.
  • Jira ticket has a comment referencing this PR.

[Copilot]

Pull request overview

Adds end-to-end support for simulated “USB cameras” in Synthesis: camera configuration is published via HALSim (SimDevice), while rendered frames are streamed over a side-channel WebSocket to the robot process so existing WPILib/OpenCV vision code can consume frames.

Changes:

• Introduces a simulated UsbCamera/CvSink implementation in SyntheSimJava backed by a local WebSocket frame receiver and HALSim-published camera config.
• Adds Fission UI + scene rendering support to configure cameras on a robot, preview them in-app, and stream JPEG frames to the robot-side WebSocket server.
• Extends preference types/tests and WPILib sim plumbing with a new SimType.CAMERA and camera config field constants.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/UsbCamera.java	Adds a drop-in UsbCamera wrapper that sources frames from Synthesis instead of hardware.
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/CvSink.java	Adds a CvSink wrapper that returns decoded frames from the simulated camera.
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/CameraFrameServer.java	Implements localhost WebSocket server to receive <device>\n<base64-jpeg> frame messages.
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/Camera.java	Publishes camera config via SimDevice and decodes latest JPEG bytes into an OpenCV Mat.
simulation/SyntheSimJava/gradlew.bat	Adds SPDX license tag comment.
simulation/SyntheSimJava/gradlew	Adds SPDX license tag comment + minor wrapper script adjustments.
simulation/SyntheSimJava/gradle/wrapper/gradle-wrapper.properties	Updates Gradle wrapper distribution URL.
simulation/SyntheSimJava/build.gradle	Adds cscore/OpenCV + Java-WebSocket dependencies for camera streaming.
simulation/samples/JavaAutoSample/src/main/java/frc/robot/Robot.java	Demonstrates grabbing frames, computing mean brightness, and republishing via CameraServer.
fission/src/ui/panels/simulation/CameraPreviewPanel.tsx	Adds a panel to preview configured robot cameras by copying their frame canvases.
fission/src/ui/panels/configuring/assembly-config/interfaces/ConfigureCameraInterface.tsx	Adds configuration UI for camera name, parent node transform, FOV, resolution, and FPS.
fission/src/ui/panels/configuring/assembly-config/ConfigurePanel.tsx	Wires the new “USB Cameras” config mode into the Configure panel.
fission/src/ui/panels/configuring/assembly-config/ConfigTypes.ts	Adds ConfigMode.CAMERA.
fission/src/test/PreferencesSystem.test.ts	Updates expected robot preference defaults to include cameras: [].
fission/src/systems/simulation/wpilib_brain/WPILibTypes.ts	Adds SimType.CAMERA + camera config field keys.
fission/src/systems/simulation/wpilib_brain/WPILibState.ts	Extends supplier/receiver type maps for the new camera sim type.
fission/src/systems/simulation/wpilib_brain/sim/SimCamera.ts	Adds helper getters for camera width/height/fps and presence via sim map.
fission/src/systems/simulation/wpilib_brain/sim/SimAccel.ts	Fixes Jolt memory ownership issues when reading rotation/velocity (avoid destroying borrowed refs).
fission/src/systems/simulation/wpilib_brain/CameraFrameSocket.ts	Implements outgoing WS client to stream frames to the robot-side server with backpressure dropping.
fission/src/systems/preferences/PreferenceTypes.ts	Adds CameraPreferences and integrates it into RobotPreferences + defaults.
fission/src/mirabuf/RobotCameraSceneObject.ts	Renders camera views to a render target, previews to a canvas, and streams JPEG frames to robot.
fission/src/mirabuf/MirabufSceneObject.ts	Creates/destroys camera scene objects and exposes camera preview panel entry when cameras exist.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 6 comments.

[AlexD717]

Overall functionality seems pretty good. I am able to add a camera and view its stream.

However, when I try and edit the camera FPS, width, and height, it doesn't seem to update. The field of view does update.

In addition, the frame received value alternates every frame from false to true (this is very minor so don't bother fixing if its hard).

[PepperLola]

However, when I try and edit the camera FPS, width, and height, it doesn't seem to update. The field of view does update.

Do you mean update them in Fission? or through Shuffleboard? I'll make sure that works when updating in Fission, but I don't plan to support that in Shuffleboard (i.e., the FPS and Resolution inputs in the camera widget).

In addition, the frame received value alternates every frame from false to true (this is very minor so don't bother fixing if its hard).

This is all on the side of the robot code and is intended behavior, but I could still change it if it's misleading from a code sample point of view. The robot won't receive frames at the same rate as its periodic methods are called, and the frame received value is true only for a tick where there is a new frame. If the robot is updating at 50tps but the camera is streaming at 30fps, even under nominal network conditions they wouldn't line up and you'd end up with some ticks where there is "no frame received" even though everything's working properly. If it makes more sense to remove this from the sample or replace it with a different metric then I can do that.

[AlexD717]

However, when I try and edit the camera FPS, width, and height, it doesn't seem to update. The field of view does update.

Do you mean update them in Fission? or through Shuffleboard? I'll make sure that works when updating in Fission, but I don't plan to support that in Shuffleboard (i.e., the FPS and Resolution inputs in the camera widget).

I meant in fission.

In addition, the frame received value alternates every frame from false to true (this is very minor so don't bother fixing if its hard).

This is all on the side of the robot code and is intended behavior, but I could still change it if it's misleading from a code sample point of view. The robot won't receive frames at the same rate as its periodic methods are called, and the frame received value is true only for a tick where there is a new frame. If the robot is updating at 50tps but the camera is streaming at 30fps, even under nominal network conditions they wouldn't line up and you'd end up with some ticks where there is "no frame received" even though everything's working properly. If it makes more sense to remove this from the sample or replace it with a different metric then I can do that.

I wouldn't bother, this was just a minor thing I noticed.

[azaleacolburn]

Really incredible work here! I just had a few nitpicks.

[AlexD717]

Could we change the camera UI to look more like the field scoring/protected zones? That way its standardized across our UI instead of having the top buttons for switching cameras.

[AlexD717]

test slack notification

[PepperLola]

Could we change the camera UI to look more like the field scoring/protected zones? That way its standardized across our UI instead of having the top buttons for switching cameras.

I believe I've refactored it appropriately, though I still plan to improve the styling and test further. If you want you can take a look and see if it works well. Note that when adding or deleting cameras with the camera preview panel open, the canvases will turn black until you save and the physics system is unheld again (I think this is the cause but I will investigate more).

[AlexD717]

Some minor UI stuff that I think could be further improved. Up to you if you want to fix them here, or just create another ticket for that (since this ticket was primarily for codesim support).

In our other systems, pressing add new opens that configuration option for the newly added item directly, while for the USB camera panel it just adds a new camera (slight inconsistency, up to you if you want to fix it).

In addition, ideally right clicking on a robot and selecting "preview camera" should open the corresponding robots preview camera (doesn't do that with multiple robots).

stale, multiple cameras works in shuffleboard now

[PepperLola]

@AlexD717 creating a new camera now automatically edits it, and the preview panel now shows only the cameras on the robot that you right clicked on.

[AlexD717]

If you have multiple robots spawned with the same camera name (ex USB Camera 0), Elastic gets confused and sends one frame from one camera and the next from the other resulting in a very bug experience. Just adding a toast warning saying "Having multiple cameras with the same name can lead to unexpected behavior" should be enough. Though since fission handles it fine, you might just trust the people using codesim to figure it out themselves.

[AlexD717]

Are we going to go through and add top bar icons for all missing configuration options in a later ticket (ex this introduces USB Cameras, one of my PRs introduces robot spawn positions)?