Image Transfer Remote Operation via webrtc

[horizoneee]

Thank you very much for your work, I ran through the graphic transfer of webrtc, my idea is like this: pass the image captured by ros to the server side of webrtc through shared memory and publish it, but during the running process, every time I run the program, the video stream will lag and become fuzzy in the first 10s after I enter the web interface, due to the fact that visionpro doesn't provide chrome's developer mode, I can't locate what causes this lagging, is it something wrong with my operation? Or is it because the server-side setup of webrtc and the image keyframe transfer would be the main cause of the lagging, have you encountered this situation?

class ZedVideoTrack(MediaStreamTrack):
kind = "video"
def init(self, array, toggle_streaming,client):
super().init() # Initialize base class
self.img_array = array
self.toggle_streaming = toggle_streaming
self.streaming_started = False
self.timescale = 1000 # Use a timescale of 1000 for milliseconds
# self.frame_interval = 1 / fps
self._last_frame_time = time.time()
self.start_time = time.time()
self.client = client
# self.target_resolution = (360, 1280) # 降低分辨率

async def recv(self):
    """
    This method is called when a new frame is needed.
    """
    
    self.client.do_detect() # 从共享内存中读取图像数据
    now = time.time()
    dt_object = datetime.fromtimestamp(now)
    formatted_date = dt_object.strftime("%H:%M:%S:%f")[:-3]
    print(f"recv start time: {formatted_date}")

    compressed_data = np.frombuffer(self.img_array.tobytes(), np.uint8)  # 确保 compressed_data 是一个 numpy 数组
    compressed_data = cv2.imdecode(compressed_data, cv2.IMREAD_COLOR)
    display_image = cv2.cvtColor(compressed_data, cv2.COLOR_BGR2RGB)
    
    print("display_image shape: ", display_image.shape)
   
    av_frame = VideoFrame.from_ndarray(display_image, format='rgb24')  # Convert numpy array to AVFrame
    timestamp = int((time.time() - self.start_time) * self.timescale)
    av_frame.pts = timestamp
    av_frame.time_base = self.timescale
    # print("Time to process frame: ", time.time() - start)
    upsert_time2 = time.time()
    dt_object = datetime.fromtimestamp(upsert_time2)
    formatted_date1 = dt_object.strftime("%H:%M:%S:%f")[:-3]
    print(f"before upsert time: {formatted_date1}")
    return av_frame