How to read the semantic buffer?

Include the following details (edit as applicable):

  • Issue category: Semantic Segmentation
  • Device type & OS version: iOS15.1 on iPhone 12 Pro
  • Host machine & OS version: Mac
  • Issue Environment : Native XCode build (=on device?)
  • Xcode version: 13.0
  • ARDK version: 0.9
  • Unity version: 2019.4

Description of the issue:
I’m unsure how to read the semantic buffer correctly.
I have the RAW image in place to visualize the buffer and it looks correct.
To debug the readout, I’m feeding a touch.position and checking the boolean. However it gives results that don’t correspond to what the RAW Image is showing on screen.
What am I doing wrong?

using UnityEngine;
using UnityEngine.UI;
using Niantic.ARDK.Extensions;

public class SemanticDebug : MonoBehaviour
    Text debugText;

    ARSemanticSegmentationManager arSemanticSegmentationManager;

    bool result;

    void Update()
        if (Input.touchCount > 0)
            Touch touch = Input.GetTouch(0);

            if (touch.phase == TouchPhase.Moved)
                result = arSemanticSegmentationManager.LatestSemanticBuffer.DoesChannelExistAt(touch.position, Screen.width, Screen.height, 1);
                debugText.text = $"result is: {result} for {touch.position}";

I’m a normal user, just like you, but i hope i could help.

I suggest you to look at the ‘Querying Semantic Tutorial’ in the official documentation.
Basically i suggest you to use the ‘PlatformAgnosticInput’ (instead of the Unity Input) and use the ‘arSemanticSegmentationManager.SemanticBufferProcessor.GetChannelNamesAt(x, y);’ function to get the channel’s name. I’ll leave below the official documentation’s code.

void Update()
        if (PlatformAgnosticInput.touchCount <= 0) { return; }
        var touch = PlatformAgnosticInput.GetTouch(0);
        if (touch.phase == TouchPhase.Began)
            //list the channels that are available
            Debug.Log("Number of Channels available " + _semanticManager.SemanticBufferProcessor.ChannelCount);
            foreach (var c in _semanticManager.SemanticBufferProcessor.Channels)
            int x = (int)touch.position.x;
            int y = (int)touch.position.y;
            //return the indices
            int[] channelsInPixel = _semanticManager.SemanticBufferProcessor.GetChannelIndicesAt(x, y);
            //print them to console
            foreach (var i in channelsInPixel)
            //return the names
            string[] channelsNamesInPixel = _semanticManager.SemanticBufferProcessor.GetChannelNamesAt(x, y);
            //print them to console
            foreach (var i in channelsNamesInPixel)

Hope I was helpful,


Thanks, that was an excellent tip!
That SemanticBufferProcessor did the trick.

Would you happen to know how to pause/temporarily switch off the semantic system? I’m not sure if disabling the ARSemanticSegmentationmanager stops the underlying processes…?

This topic was automatically closed 2 hours after the last reply. New replies are no longer allowed.