How to read the semantic buffer?

Include the following details (edit as applicable):

  • Issue category: Semantic Segmentation
  • Device type & OS version: iOS15.1 on iPhone 12 Pro
  • Host machine & OS version: Mac
  • Issue Environment : Native XCode build (=on device?)
  • Xcode version: 13.0
  • ARDK version: 0.9
  • Unity version: 2019.4

Description of the issue:
I’m unsure how to read the semantic buffer correctly.
I have the RAW image in place to visualize the buffer and it looks correct.
To debug the readout, I’m feeding a touch.position and checking the boolean. However it gives results that don’t correspond to what the RAW Image is showing on screen.
What am I doing wrong?

using UnityEngine;
using UnityEngine.UI;
using Niantic.ARDK.Extensions;

public class SemanticDebug : MonoBehaviour
{
    [SerializeField]
    Text debugText;

    [SerializeField]
    ARSemanticSegmentationManager arSemanticSegmentationManager;

    bool result;

    void Update()
    {
        if (Input.touchCount > 0)
        {
            Touch touch = Input.GetTouch(0);

            if (touch.phase == TouchPhase.Moved)
            {
                result = arSemanticSegmentationManager.LatestSemanticBuffer.DoesChannelExistAt(touch.position, Screen.width, Screen.height, 1);
                debugText.text = $"result is: {result} for {touch.position}";
            }
        }
    }
}

Hi!
I’m a normal user, just like you, but i hope i could help.

I suggest you to look at the ‘Querying Semantic Tutorial’ in the official documentation.
Basically i suggest you to use the ‘PlatformAgnosticInput’ (instead of the Unity Input) and use the ‘arSemanticSegmentationManager.SemanticBufferProcessor.GetChannelNamesAt(x, y);’ function to get the channel’s name. I’ll leave below the official documentation’s code.

void Update()
    {
        if (PlatformAgnosticInput.touchCount <= 0) { return; }
 
        var touch = PlatformAgnosticInput.GetTouch(0);
        if (touch.phase == TouchPhase.Began)
        {
            //list the channels that are available
            Debug.Log("Number of Channels available " + _semanticManager.SemanticBufferProcessor.ChannelCount);
            foreach (var c in _semanticManager.SemanticBufferProcessor.Channels)
                Debug.Log(c);
 
 
            int x = (int)touch.position.x;
            int y = (int)touch.position.y;
 
            //return the indices
            int[] channelsInPixel = _semanticManager.SemanticBufferProcessor.GetChannelIndicesAt(x, y);
            
            //print them to console
            foreach (var i in channelsInPixel)
                Debug.Log(i);
 
            //return the names
            string[] channelsNamesInPixel = _semanticManager.SemanticBufferProcessor.GetChannelNamesAt(x, y);
 
            //print them to console
            foreach (var i in channelsNamesInPixel)
                Debug.Log(i);
 
        }
 
    }

Hope I was helpful,
Cheers.

3 Likes

Thanks, that was an excellent tip!
That SemanticBufferProcessor did the trick.

Would you happen to know how to pause/temporarily switch off the semantic system? I’m not sure if disabling the ARSemanticSegmentationmanager stops the underlying processes…?

This topic was automatically closed 2 hours after the last reply. New replies are no longer allowed.