✨ Migrate to new-arch (react-native 0.74, Fabric/TurboModules/CodeGen/bridgeless)

[mrousavy]

What feature or enhancement are you suggesting?

VisionCamera needs to migrate to the new architecture, and use Fabric for the View, and TurboModules + CodeGen for the native methods. Also, this implies compatibility with react-native 0.74, as that currently fails to build. (related to New Arch)

Implementation plan

Since VisionCamera is a bit more complicated, there are a few blockers I have encountered:

1. ❌ I have both a View (CameraView) and two Modules (CameraModule + DevicesModule) in the codebase. Currently, the create-react-native-library template does not have a template for multiple CodeGen specs (views + modules).
   • CameraViewManager.swift
   • CameraDevicesManager.swift
   • CameraView.swift
2. ❌ For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the <Camera> view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet. This is how this works currently:
   • JS: Camera.tsx (uses findNodeHandle(..) + a global setFrameProcessor(..) func)
   • iOS: VisionCameraProxy.mm (uses UIManager::viewForReactTag to find the view)
   • Android: VisionCameraProxy.kt (uses UIManagerHelper.getUIManager to find the View)
3. ❌ For the Frame Processor runtime I need access to the jsi::Runtime as early as possible (on demand). Currently I get the jsi::Runtime from the RCTCxxBridge/CatalystInstance, which is a kinda private and unsafe API.
   • iOS: VisionCameraInstaller::install
   • Android: VisionCameraProxy.kt

And then a few things I wanted to wait for (which aren't blockers) before switching to the new arch are:

1. I want to pass multiple callbacks to a native function. This was a limitation with the Bridge, but should now work with TurboModules I think? In my case startRecording() takes both onRecordingFinished and onRecordingError callbacks, and also returns a Promise which resolves once the recording has actually been started.
2. I couldn't find a clear script to trigger CodeGen (something like npx react-native codegen) to generate both iOS and Android specs.
3. New arch doesn't have first class Swift support as far as I know?
4. New arch doesn't support custom "hybrid" objects (so e.g. I could return an in-memory UIImage/Image instance in takePhoto(), instead of writing it to a file and returning a string as that's the only supported type)

What Platforms whould this feature/enhancement affect?

iOS, Android

Alternatives/Workarounds

Currently only the renderer interop layer can be used, but there are some issues:

• 🐛 Error while trying to take a photo or use frame processors and using React Native New Renderer Interop Layer #2613

Additional information

• I agree to follow this project's Code of Conduct
• I searched for similar feature requests in this repository and found none.

[cipolleschi]

Hi @mrousavy, thanks for this issue, it is super helpful!

1. ❌ I have both a View (CameraView) and two Modules (CameraModule + DevicesModule) in the codebase. Currently, the create-react-native-library template does not have a template for multiple CodeGen specs (views + modules).

This should not be a problem. In your codegenConfig field of the vision-camera package, you can specify all as type, and CodeGen shuld generate the native code for both modules and classes.
In this case, the jsSrcs field should be a parent folder that contains all the specs. For example:

react-native-camera-module
'-> js
    '-> Module
          '-> NativeCameraModule.js
     '-> NativeComponent
         '-> CameraViewComponent.js 

and you pass just js as folder in the jsSrcs.

2. ❌ For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet. This is how this works currently:
   JS: Camera.tsx (uses findNodeHandle(..) + a global setFrameProcessor(..) func)
   iOS: VisionCameraProxy.mm (uses UIManager::viewForReactTag to find the view)
   Android: VisionCameraProxy.kt (uses UIManagerHelper.getUIManager to find the View)
3. ❌ For the Frame Processor runtime I need access to the jsi::Runtime as early as possible (on demand). Currently I get the jsi::Runtime from the RCTCxxBridge/CatalystInstance, which is a kinda private and unsafe API.
   iOS: VisionCameraInstaller::install
   Android: VisionCameraProxy.kt.

I don't have a strong guidance here, but a coleague is working on 3. and perhaps with is work you'll be able to make 2. works as well? Not sure about this.

1. I want to pass multiple callbacks to a native function. This was a limitation with the Bridge, but should now work with TurboModules I think? In my case startRecording() takes both onRecordingFinished and onRecordingError callbacks, and also returns a Promise which resolves once the recording has actually been started.

I read internally a proposal to make this happen, but I don't have an update on the work here.

2. I couldn't find a clear script to trigger CodeGen (something like npx react-native codegen) to generate both iOS and Android specs.

In 0.74, we have npx react-native codegen, but you need to call it twice, one for iOS and one for Android.
It should be easy to add a

"script" : {
  "run-codegen": "scripts/execute-codegen.sh",
  }

to react-native-vision-camera where you run:

npx react-native codegen <ios params>
npx react-native codegen <android params>

and then you can call

yarn run-codegen

3. New arch doesn't have first class Swift support as far as I know?

No, and it will not have in the short term. To make it work properly, we need to radically change the ReactCommon file structure and it is a long work which we can't prioritize at the moment.

Anyway, it is possible to work around the issue: you should be able to create a thin Objective-C++ layer that:

1. implements the module/components specs.
2. holds a Swift object.
3. forward all the invocation to the held Swift object.

It would be actually very interesting to try this out in a library that is as complex as yours: we might be able to generalize the wrapper somehow... 🤔

4. New arch doesn't support custom "hybrid" objects (so e.g. I could return an in-memory UIImage/Image instance in takePhoto(), instead of writing it to a file and returning a string as that's the only supported type)

I think expo manage to achieve something like this for some of their packages. There are ways to do it probably, but I feel that they might depends heavily on your use case.

Currently only the renderer interop layer can be used, but there are some issues:
#2613

We did a lot of work on interop layers for 0.74. The issue author is not mentioning which version of React Native they are using. It is likely that we fix them already in 0.74 (or they will be fixed soon).

[mrousavy]

Thanks so much for your detailed answer @cipolleschi! This is really helpful :)

[rayronvictor]

We did a lot of work on interop layers for 0.74. The issue author is not mentioning which version of React Native they are using. It is likely that we fix them already in 0.74 (or they will be fixed soon).

I'm using RN 0.73.2

[fabOnReact]

2. For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet. This is how this works currently:
   JS: Camera.tsx (uses findNodeHandle(..) + a global setFrameProcessor(..) func)
   iOS: VisionCameraProxy.mm (uses UIManager::viewForReactTag to find the view)
   Android: VisionCameraProxy.kt (uses UIManagerHelper.getUIManager to find the View)

I believe there is an example of implementation in TextInput.js which uses the codegen command setTextAndSelection instead of using UIManager + findNodeHandle.

Codegen Commands replaces setNativeProp, which was used with functionalities like react-native-camera zoom prop. Re-rendering the entire component every time there is a change in zoom, would cause the react-native app to slow down, for this reason we would use setNativeProp({zoom: newZoomValue}).

It is sometimes necessary to make changes directly to a component without using state/props to trigger a re-render of the entire subtree. When using React in the browser for example, you sometimes need to directly modify a DOM node, and the same is true for views in mobile apps. setNativeProps is the React Native equivalent to setting properties directly on a DOM node.

More info here react/react-native#42701 (comment)

The codegen command calls FabricRenderer dispatchCommand which uses createFromHostFunction to register a function on the native Android/iOS API. JavaScript can call the function in the native iOS/Android API.

1. Function::createFromHostFunction() registers a new function with methodNam is dispatchCommand

/// A function which has this type can be registered as a function
/// callable from JavaScript using Function::createFromHostFunction().
/// When the function is called, args will point to the arguments, and
/// count will indicate how many arguments are passed.  The function
/// can return a Value to the caller, or throw an exception.  If a C++
/// exception is thrown, a JS Error will be created and thrown into
/// JS; if the C++ exception extends std::exception, the Error's
/// message will be whatever what() returns. Note that it is undefined whether
/// HostFunctions may or may not be called in strict mode; that is `thisVal`
/// can be any value - it will not necessarily be coerced to an object or
/// or set to the global object.

2. The HostFunction for the methodName dispatchCommand is passed as the last argument of createFromHostFunction

https://github.com/facebook/react-native/blob/8317325fb2bf6563a9314431c42c90ff68fb35fb/packages/react-native/ReactCommon/jsi/jsi/jsi.h#L100-L111

  /// Create a function which, when invoked, calls C++ code. If the
  /// function throws an exception, a JS Error will be created and
  /// thrown.
  /// \param name the name property for the function.
  /// \param paramCount the length property for the function, which
  /// may not be the number of arguments the function is passed.

https://github.com/facebook/react-native/blob/8ff05b5a18db85ab699323d1745a5f17cdc8bf6c/packages/react-native/ReactCommon/react/renderer/uimanager/UIManagerBinding.cpp#L603-L618

[uiManager, methodName, paramCount](
    jsi::Runtime& runtime,
    const jsi::Value& /*thisValue*/,
    const jsi::Value* arguments,
    size_t count) -> jsi::Value {
  validateArgumentCount(runtime, methodName, paramCount, count);


  auto shadowNode = shadowNodeFromValue(runtime, arguments[0]);
  if (shadowNode) {
    uiManager->dispatchCommand(
        shadowNode,
        stringFromValue(runtime, arguments[1]),
        commandArgsFromValue(runtime, arguments[2]));
  }
  return jsi::Value::undefined();
});

3. The dispatchCommand HostFunction calls FabricMountingManager::dispatchCommand:

[fabOnReact]

For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet.

https://react-native-vision-camera.com/docs/guides/frame-processors

If I try to solve the issue without using worklet (we can try to add it later).

function App() {
  const userDefinedFunction = (frame) => {
    const objects = detectObjects(frame)
    console.log(`Detected ${objects.length} objects.`)
  }
  const frameProcessor = useFrameProcessor(userDefinedFunction)

  return (
    <Camera
      {...cameraProps}
      frameProcessor={frameProcessor}
    />
  )
}

1. Register userDefinedFunction using createFromHostFunction (codegen command already does this for you)
2. Java/iOS invoke the callback with the frame as parameter https://github.com/facebook/react-native/blob/57ed0fb30931979742634a1faa9a4d3b5261e50d/packages/react-native/ReactAndroid/src/main/java/com/facebook/react/fabric/FabricUIManager.java#L1048-L1064
3. To implement it as worklet, I would look into other libs like reanimated

https://docs.swmansion.com/react-native-reanimated/docs/2.x/fundamentals/worklets/

https://github.com/software-mansion/react-native-reanimated/blob/6806d3d89983ffacd3b1bebbf4fecf86ba56addb/Common/cpp/ReanimatedRuntime/WorkletRuntimeCollector.h#L12-L15

  // When worklet runtime is created, we inject an instance of this class as a
  // `jsi::HostObject` into the global object. When worklet runtime is
  // terminated, the object is garbage-collected, which runs the C++ destructor.
  // In the destructor, we unregister the worklet runtime from the registry.

Sorry. I'm trying to learn more about this so maybe my comment not clear.

[philIip]

❌ For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet. This is how this works currently:

hiiiii thanks so much for this post; reading this and the code i'm not totally understanding what the core issue is - is it that the UIManager APIs themselves are not supported / behaving as expected in the new architecture?

another question, i'm assuming that the problem you're implying with the worklet is that it needs to be run on a specific thread, but the callback conversion forces it to always be run on JS thread? the piece that i'm missing here is where the native module block conversion is happening? it seems like everything in your code pointers is pure JSI at the moment and doesn't involve the native module infra.

❌ For the Frame Processor runtime I need access to the jsi::Runtime as early as possible (on demand). Currently I get the jsi::Runtime from the RCTCxxBridge/CatalystInstance, which is a kinda private and unsafe API.

right now we made this backward compatible, but i will be writing some docs on what the future-looking APIs are for this!

[mrousavy]

hiiiii thanks so much for this post; reading this and the code i'm not totally understanding what the core issue is - is it that the UIManager APIs themselves are not supported / behaving as expected in the new architecture?

Hey @philIip thanks for tuning in! 👋 so the idea is that VisionCamera's View uses the RN architecture for almost everything:

• device
• format
• isActive
• resizeMode
• ...etc

But there is one prop which the user passes to the Camera that I cannot use the RN architecture for, and that is the frameProcessor. The reason is that I do some special transformations to that function, and need to get it as the pure jsi::Function on the native side to convert it to a callable Worklet on my native side.
If I don't manually get it as a jsi::Function and instead use the bridge for this, it will convert it to a RCTCallback or something, which is not what I want.

This then runs on the Camera Thread and is synchronously invoked with a jsi::HostObject as a parameter.

But this is only one specific prop for the Camera (frameProcessor) the rest should use the RN architecture as normal.

The RN architecture simply doesn't have a mechanism for Worklets, or custom jsi::HostObjects currently, so I need to write that manually in JSI/C++.

another question, i'm assuming that the problem you're implying with the worklet is that it needs to be run on a specific thread, but the callback conversion forces it to always be run on JS thread? the piece that i'm missing here is where the native module block conversion is happening? it seems like everything in your code pointers is pure JSI at the moment and doesn't involve the native module infra.

Exactly, that and that I cannot pass a custom jsi::HostObject ("Frame") to the callback.

This is where I eventually convert the jsi::Function to a normal Objective-C/Swift method:

• https://github.com/mrousavy/react-native-vision-camera/blob/bca9472ab8d973d8079da89144f7b58a0321f5ec/package/ios/Frame%20Processor/VisionCameraProxy.mm#L77
• https://github.com/mrousavy/react-native-vision-camera/blob/bca9472ab8d973d8079da89144f7b58a0321f5ec/package/ios/Frame%20Processor/FrameProcessor.mm#L33-L43

[philIip]

ok i see, so bear with me just want to make sure i'm getting it right! this worklet scenario is not properly supported in the old or new architecture, but you've figured out a way to make it work doing all this custom JSI stuff.

is it a blocker because this is something you feel strongly should be baked into the framework, or is there actually a gap in the old / new architecture here? i think that's the last part i'm missing

[mrousavy]

No worries! Yes this is something that does not work in old or new arch, and I don't think it even should be part of core.
It's very custom stuff.

The reason I thought it was a blocker for me is because the workaround in old arch was using the RCTCxxBridge/CatalystInstance to get the Runtime, and this now no longer works in new arch - so I want to use a Cxx TurboModule to get access to the runtime directly, but I have my doubts that codegen will work in this "hybrid" mode (generate everything for C++, but also bridge 90% of the props over to Java/Swift, while the others remain in C++ JSI)

[joacub]

hi @mrousavy thanks for this great project, im looking for this as we are using the scanner in a ver small application for cuba and the phones in there are very old ones, and i think this will be a really good for the performance of the overall scanner, do you ahve any time on when this will be migrated to the new arch ?, the react-native-workles is a blocker for us to use the new arch