How to Detect Touch GameObject while using Semantic Masking?

I'm Stuck
1

Include the following details (edit as applicable):

  • Issue category: Semantic Segmentation
  • Device type & OS version: Android / iOS
  • Host machine & OS version: Mac / Windows / Mac on Big Sur x.x
  • Issue Environment : Unity Remote / Unity Mock / On Device / Dev Portal
  • Xcode version: Latest
  • ARDK version: Latest
  • Unity version: 2020.3.LTS

Description of the issue:

I am playing around with the Semantic Segmentation and Masking Templates. If I place an object in the sky segmentation, how can I detect a touch of that object. The Object is being rendered using masking onto a flat 2DTexture RawImage to allow occlusion of real world objects using the RawImage mask. So is there a way to detect the texture of the 3DObject being rendered onto the RawImage? Using Semantic Masking I know if it touches the sky, but if I want to know if it touches the sky object that I put in that segmentation. HitTest are to detect real world objects, but what if I want to detect when a user touches an AR GameObject I placed on the screen.

For example , in the Semantic Masking Template, there are 2 Hot Air Balloons masked to the Sky Segmentation. How could I detect if a user touches one of those Hot Air Ballons??

Also, when using Semantic Segmentation, can you include UI Buttons? When I include touch.IsTouchOverUIObject() I get an error

NullReferenceException: Object reference not set to an instance of an object
Niantic.ARDK.Utilities.Input.Legacy.PlatformAgnosticInput.IsTouchOverUIObject (UnityEngine.Touch touch) (at Assets/ARDK/Utilities/Input/Legacy/PlatformAgnosticInput.cs:110)
Niantic.ARDK.Templates.ObjectMaskingController.Update () (at Assets/LightshipHUB/Runtime/Scripts/ObjectMaskingController.cs:141)

because the Semantic Masking is updating the buffer and hasn’t rendered complete. This is my first experimentation with this so maybe I am missing how to I include UI buttons while having Semantic Masking? The buttons I have on the screen are not being touched or onClicked methods called.

2

Hello Jamie,

For anything involving a hit-test I would recommend taking a look at the hit test code provided in our documentation. It should be easy to integrate that into a scene that involves Semantic Segmentation. My suggestion is to take a look at the sample code and compare it to what you have. I have included a link to the documentation on the subject. Please let us know what you come up with.

https://lightship.dev/docs/ardk/ar_world_tracking/hit_tests.html#

3

If I simply create a new scene and use the Semantic Masking Template. Then I add a UI Button and OnClick method. No UI Button clicks are being recognized using this template. Can you tell me how to fix that? I can record a video if you need me to show you this on the computer.

4

Hello Jamie,
Here is a link to the ARDK user manual. This describes how to go about using the different managers.

https://lightship.dev/docs/ardk/ardk_fundamentals/using_managers.html#running-the-session-manually

Try using different managers to potentially address your issue. I believe the problem you are having can be resolved by properly linking things in your code or adding different managing components to your scene. Sometimes even changing the execution order can fix problems that occur. Please let us know if this helps you.