Access to camera feed for custom computer vision

Feature Requests
1

Hello!

First of all thank you so much for making this incredibly awesome sdk. It’s capabilities so far are amazing and the documentation and examples really useful.

I wanted to know if you could provide an example of accessing the camera feed so that we could do our own custom computer vision stuff on it - for example hand tracking. In the past I managed to integrate mediapipe’s handtracking with arcore native android sdk and allowed for some nice natural interaction.

I see in the documentation that there is ARCameraFeed (Niantic AR Development Kit (ARDK): ARCameraFeed) but it is not clear how to go about accessing this.

Thanks again!
Jennifer

1 Like
2

Hi Jennifer!

I’ve never done it personally myself but one way could be to create a UI RawImage in Unity and put the camera feed texture into that. Sorta like they do in this article (this is just talking about doing it within Unity and AR Foundations, not ARDK, but it may still give you an idea of how to link the two?)

The relevant section is " Loading your model, creating model inputs, and reading your model outputs in Unity"

While not directly a “here is how to do it in the ARDK”, I’m hoping it helps a little in the meantime before someone from Niantic can respond :slight_smile:

1 Like
3

Thanks so much for the quick response and this useful info. Will check it out!

4

Hi Jennifer, we looked into this with one of our engineers, and something like this should be possible if you grab the current frame (IARSession.CurrentFrame, or IARSession.FrameUpdated) and access pointers to the frame’s image texture(s).

On Android, this is a single ARGB32 texture. On iOS there are two Y + CbCr textures.

For example code, you can take a look at the _ARKitFrameRenderer and _ARCoreFrameRenderer scripts that are included in the ARDK Lightship Unity package. To access these, you can import the ardk-(version).unitypackage file into your project, and then search for them in the project window.

Please note that there is already some internal overhead here (queuing onto the Unity main thread = ~1/2 a frame latency on average), so feeding this pointer into a further ML pipeline may add more latency/visual lag before rendering. Also, as it is a pointer to a native texture, ARDK may write over the previous data if it is held for more than a frame. Do a copy if this is the case.

1 Like
5

Thank you for the response. It is not clear to me how the _ARCoreFrameRenderer code illustrates the use of the frame’s image texture but nevertheless I tried the following code:

var frame = _session.CurrentFrame;
if (frame != null)
{
          tex = new Texture2D(resolution.width, resolution.height, TextureFormat.ARGB32, false);
          tex.LoadRawTextureData(frame.CapturedImageTextures[0], width *height * 4);
          tex.Apply();
}

Where tex is the texture I would like to pass on to do the further computer vision processing.
Is this correct?

For some reason the current frame is always null so I never get to execute the loading of the image texture. Do you perhaps know why the current frame would stay null? I also tried to listen to the FrameUpdated event but this doesnt seem to fire either.

Thanks!

6

Sorry just found the section of code in _ARCoreFrameRender that does indeed illustrate the use of the frame’s image texture so will try implement it like that now and keep you posted.

I think I also realised why I was getting a null frame.

Thanks for the help!

1 Like
7

Hmmm… I am still not having any luck. I see though that the code in _ARCoreFrameRenderer calls this function Texture2D.CreateExternalTexture and when I read the Unity documentation on it, it says " This function is mostly useful for native code plugins that create platform specific texture objects outside of Unity" which makes me wonder if my first post here led you to believe that I was trying to integrate with a native android app.

I am in fact just trying to get the camera feed from ardk to a Unity plugin (this one to be exact GitHub - homuler/MediaPipeUnityPlugin: Unity plugin to run MediaPipe graphs) which can accept a texture2d as input. Is there a simple way to do this?

I have tried multiple things today including trying to access the ARVideoFeed.GPUTexture and reading it to Texture2d texture as well as the trying to access the ARCameraFeed’s VideoTextureBGRA but none of these seem to work.

Thanks again for you earlier response. Hope I am making myself clearer here :slight_smile:

8

Sorry for re-opening this thread after such a delay but i’m trying to do the same thing. Did anyone make some more progress on this in the meantime? Thanks in advance