ARDK 2.x -> 3.4 Upgrade: Lightship seems to break Image Tracking

[Jess_Woodward]

• Issue category: ARDK
• Device type & OS version: Huawei P30 Pro (Android 12), Samsung Galaxy S20 FE (Android 12), Samsung Galaxy S20 5G (Android 13)
• Host machine & OS version: Windows 11
• Issue Environment: On Device
• Xcode version: n/a
• ARDK version: freshly upgraded from 2.3.0 to 3.4.0-2404081151
• Unity version: 2021.3.3f1, have also tried 2022.3.10f1, 2022.3.16f1

Bug reproduction steps:
Upgrade an existing Unity project in 2021.3.3f1 with ARDK 2.3.0 to ARDK 3.4.0, following the steps outlined in the documentation here: Migration Guide | Niantic Lightship

Description of the issue
I’ve been trying to upgrade an existing project to ARDK3, but Image Tracking isn’t working whenever I have Lightship enabled.

It’s not an issue with my code or image markers; I’ve been testing with a fresh scene with just Unity base components, the Unity sample image library, and a simple debug log script.

No prefab spawns when the camera is pointed at the image marker, and ARTrackedImageManager.trackedImagesChanged can be subscribed to but does not seem to trigger, regardless of how I position the image marker (distance, angle, printed, on screen, etc).

Screen recordings, LogCat messages, and test APK here: ARDK3 Image Tracking issues - Google Drive (APK is just a blank image tracking scene with 1 script for debug logs, using Unity’s test image library. Debug log script is also in that folder under Unity_SetupScreenshots)

Worth noting:

• I have two other projects using the exact same dev environments, also both upgraded from 2.x, in which image tracking is working perfectly. Even when I match the settings to those projects, the issue persists.
• If I uninstall Lightship and use just AR Core, Image Tracking works perfectly in the problem-project.
• I don’t want to uninstall Lightship as we’re using Semantic Segmentation in other scenes (which are working perfectly).
• AR initialises & the AR Foundation debug menu says that Image Tracking is available in this configuration, but trackedImagesChanged never fires from AR Foundation.
• Creating a fresh project to get around the issue is not an option; this is a well established project.

Possibly not related but mentioning just in case it’s relevant:

• We don’t have the SharedAR package installed.
• When I make a change to the XR config (eg disable Semantic Segmentation, which isn’t used in this scene anyway), the apk often crashes on load if I don’t completely delete the previous version of it off my phone first.
• In order to delete this project’s Library I have to use force-delete due to Windows permissions (despite being an Admin account); in other projects I can just move to trash.

Things I’ve tried:
Is one of the other features part of the problem? Maybe! Object detection being enabled seems to prevent AR from initialising at all, but it’s enabled in another project that works just fine
Check Lightship version is the same as our working projects. They’re the same version, as is AR Foundation/ARKit/ARCore
Check Android LogCat for error messages that aren’t showing up in ConsoleLogs. Nothing that looks related to image tracking to me - screenshots in the link above.
Disable custom manifest no difference
Rebuild the library no difference
Use the Android Dependency Resolver no difference
Uninstall and Reinstall Lightship I noticed AR Foundation/ARCore/ARKit were visible in the package manager. I uninstalled them and Lightship, then reinstalled Lightship only. This time around AR Foundation didn’t show up in the Package Manager in 2021.3.3f1 (it does in 2022.3.10/16), though the packages are visible in the Project>Packages files. First test build after the uninstall/reinstall crashed straight after I enabled camera permissions. Rebuilt the library again and the next build didn’t crash but image tracking still didn’t work.
Upgrade to Unity 2022.3.10f1 & 2022.3.16f1 no difference

I’m deeply at a loss on what else to try to resolve this. I understand from the other threads that this is near impossible to repro, so happy trial and error on request (my timezone is Wellington (GMT+12) so I may be async-timing).

Other people having the same/similar issue:

ARDK 3.4 Image Tracking

Got it. Let me take a look at these logs and I’ll get back to you. I did also want to confirm with you if you’re switching between both AR packages in the same project or if you’re using different projects for each AR package. The reason I ask is that if you have Lightship installed, you can’t pick the standalone ARCore or ARKit even if you uncheck Lightship. If you do, it’ll just automatically default back to Lightship SDK + ARCore (or ARKit) If you’re able to switch between Lightship SDK + AR…

AR Tracked Image Manager doesn't work

Hi Extesa, I’m sorry to hear you’re having trouble using image tracking with Lightship. When you say it wasn’t working, were you receiving an error message? If not, could you please test again with development mode enabled and your device connected to your development computer to read output? Kind Regards, Maverick L.

Discord - A New Way to Chat with Friends & Communities

Discord - A New Way to Chat with Friends & Communities

[Jesus_Hernandez]

Hello Jess,

Thanks for your patience. I was looking at the information provided along with the other users’ support requests. I’m still looking into this, it looks like you and another user are currently experiencing this but I’m yet to be able to reproduce this issue, image detection is working for me.

Can you send a screenshot of your Tracked Image Manager and if you’re instantiating objects through code once the image is detected can you show me how you’re writing said code?

[Jess_Woodward]

Yeah, as mentioned it’s fine in my other projects - only this project that seems to be having the issue.

I stripped everything back to a test scene when I first ran into the problem, so I’m not instantiating anything through code at present. Just using the Tracked Image Prefab in the AR Tracked Image Manager + a simple debug log script (below).

You can see the debug logs at the end of this video: https://drive.google.com/file/d/1U9dyqurSfWizUdcQT2r1xrayERBMAC2c/view?usp=sharing

trackedImagesChanged gets subscribed to successfully, but doesn’t seem to be called when I point the camera at an image marker - and given that the Tracked Image Manager doesn’t instantiate the prefab, I infer that the marker is not being detected rather than it being a code issue. (The same code also works fine in my other projects)

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using static UnityEngine.CullingGroup;

public class ImageTrackDebug : MonoBehaviour
{
    [SerializeField] private ARTrackedImageManager _arTrackedImageManager;
    private bool _sessionHasInitialized = false;

    void Start()
    {
        ARSession.stateChanged += OnStateChanged;
        Debug.Log("Subscribed to ARSession.stateChanged");
    }

    private void OnDestroy()
    {
        ARSession.stateChanged -= OnStateChanged;
        _arTrackedImageManager.trackedImagesChanged -= OnTrackedImagesChanged;
    }

    private void OnStateChanged(ARSessionStateChangedEventArgs args)
    {
        if ((args.state == ARSessionState.SessionTracking || args.state == ARSessionState.Ready || args.state == ARSessionState.SessionInitializing) && !_sessionHasInitialized)
        {
            _arTrackedImageManager.trackedImagesChanged += OnTrackedImagesChanged;
            Debug.Log("Subscribed to arTrackedImageManager.trackedImagesChanged");
            _sessionHasInitialized = true;
        }
        if (args.state != ARSessionState.SessionTracking && _sessionHasInitialized) Debug.Log($"Session is not tracking. Reason: {ARSession.notTrackingReason}");
    }

    private void OnTrackedImagesChanged(ARTrackedImagesChangedEventArgs eventArgs)
    {
        Debug.Log("Starting OnTrackedImagesChanged");

        foreach (var newImage in eventArgs.added)
        {
            Debug.Log("Image added: " + newImage.name);
        }

        foreach (var updatedImage in eventArgs.updated)
        {
            Debug.Log("Image updated: " + updatedImage.name);
        }
    }
}

[Jesus_Hernandez]

Can you try subscribing to the events in OnEnable? Similar to how it’s done in the Unity documentation: https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@5.1/manual/features/image-tracking.html

By the looks of your output log everything looks good I mean you’re getting confirmation that you’re subscribing to these events but just to rule out the slight possibility that maybe something down the chain isn’t initializing correctly, try using OnEnable instead.

I see some errors related to depth in the LogCat but I don’t think that should interfere with an image being tracked.

[Jess_Woodward]

At this stage, every possible quirk is worth trying! Unfortunately subscribing in OnEnable makes no difference.

I tried disabling & reenabling the TrackedImageManager at runtime on the off-chance that would kick it into gear, but that didn’t make a difference.

And I had the thought “what if the reference library is somehow empty”, but that’s coming back as having the expected Count (2).

[Jesus_Hernandez]

Can you go into the Play Store and search for Google Play Services for AR?
Check to see if it’s up to date, if it’s not, update it.
If it is up to date, see if you can uninstall it and reinstall it.

We had a user this week that had issues getting some AR features to run and this resolved it for them

[Jess_Woodward]

It was up to date, yeah. Just tried uninstalling and reinstalling, and no change to Image Tracking, unfortunately.

Given that image tracking was working in this project with Lightship uninstalled/just AR Foundation installed, I’m not surprised this didn’t do the trick for me though.

[Jesus_Hernandez]

Ok, let’s try this. Can you send in logcats for the vanilla ARCore version where image tracking does work and can you also send a new set for Lighship + ARCore? I’m not seeing anything in the logcat that could imply any sort of error so hopefully comparing the logs of the version that works to the version that doesn’t give some clues.

I appreciate your patience with this issue, I’m hoping we can get it resolved as soon as possible

[Jess_Woodward]

No worries, I really appreciate the help!

I’ve done two sets of logs for each, all linked in the Drive folder below. The folders contain Warn level screenshots, and the videos are screen records from Debug level. (Too many logs to screenshot at that level - you might need to download the videos rather than preview in browser to get a legible video resolution.)

https://drive.google.com/drive/folders/1iCNREPtV9L9J-JzV2s_qnZxqDtHJ04zJ?usp=sharing

I’ve got both APKs installed under different names now, so if you want me to do new LogCats with any specific filters, let me know.

[Jess_Woodward]

Just following up to see how you got on with the LogCats, and if there’s anything more I can do?

This morning I’ve tried upgrading the project to ARDK 3.5 (including uninstalling Lightship and reinstalling it), but unfortunately that hasn’t fixed it.

[Jesus_Hernandez]

Hi Jess,

Sorry for the delay, I was going to suggest 3.5 hoping it would fix things since it comes with various bug fixes but it looks like that didn’t resolve the issue either. The logs on both sessions look nearly identical.

I’m currently in the process of reproducing the issue again. I have two projects, one with standalone ARCore and another with Lightship + ARCore. I’m going to compare all the screenshots you’ve sent me with my setups and take note of as many differences as possible to see if maybe there’s a slight difference in project configuration that can be causing this

[Jesus_Hernandez]

Hi Jess,

So I have two separate projects at the moment. One of them just with ARCore and the other with Lightship + ARCore. The first thing I noticed was that my standalone ARCore project uses a slightly newer version of ARFouncation, version 5.1.4, while Lightship + ARCore uses 5.1.1.

Can you start by backing up your current project in case anything goes wrong and then try updating ARFoundation? You can do so by going to the Package Manager, locating ARFoundation, then clicking on Version History, and clicking the Update button to the left of 5.1.4.

5.1.4 also uses a slightly newer version of the ARCore plugin which has a fix for image detection where images that would score low on their trackability would crash the app. Your app isn’t crashing but it may be worth a shot in case this is related. In the end, your ARFoundation version should be 5.1.4 and Google ARCore XR Plugin should be 5.1.4 as well. If the ARCore Plugin wasn’t updated, update it manually.

Let me know if that works.

[Jess_Woodward]

Good thought! I’ve just given it a try, and unfortunately it didn’t resolve the issue. I noticed my other two projects (that are working) use AR Foundation 5.1.0 so I tried rolling back to that version and that didn’t do the trick either. I’ve attached a screenshot of my package managers in case anything there sparks ideas though (all 3 projects are in Unity 2021.3.3f1).

The faulty project:

Non-faulty project A (This was migrated from ARDK 2 → 3, and has been built fault-free in the last month):

Non-faulty project B (This was migrated from ARDK 2 → 3, and was fault-free when last built a year ago):

[Jesus_Hernandez]

Hi Jess,

I presented the information in this ticket to our development team to take a deeper look. One of the things I was asked to have you double check was if the option for Read/Write is checked for the images

[Jess_Woodward]

Thanks for getting the dev team to take a closer look! The image did not have the Read/Write checkbox ticked, so I’ve just tried a test build with it ticked, but there has been no change.

[Jesus_Hernandez]

Hey Jess,

So it looks like I’ll be officially filing a bug report for this issue given all the material you’ve provided it should be enough for our team to look into it. I’ll keep you posted on how things go

[Jesus_Hernandez]

Hi Jess,

I reported this as a bug but I did want to do one more sanity check. You added the Tracked Image Manager in the XROrigin correct? It requires an XROrigin in the same GameObject that it’s added into so if you added the component to something else, it will automatically add another XROrigin essentially giving you 2 XROrigins in the same project.

Doesn’t sound like that’s the case since ARCore works but thought I’d ask.

[Jess_Woodward]

Every sanity check is worth it at this stage! There is only one XR Origin in the scene, and the Tracked Image Manager component is on the XR Origin game object.

[Jesus_Hernandez]

Thanks for checking. Ok so as mentioned I’ve noted this as a bug, I’ll keep you updated with any news I get and/or if we need additional information from you

[Jesus_Hernandez]

Hi Jess,

Earlier in our conversation, I remember you saying that you stripped the project of everything but the basics and were having the same issue, correct? If so, I’m wondering if you can share that project with me so that I can pass it on to some of our developers who are looking into this. We haven’t been able to replicate the behavior so we want to take a look at the project itself to see if we find anything