ARMeshManager remove small isolated meshes and clustering

I'm Stuck
1

Hi everyone, there must be an easier way than what I’ve got working until now :slight_smile:
In one scene I’m using the MeshManager together with the Lightship Meshing Extension to only create meshes that are also identified by the SegmentationManager as “tree_trunk_experimental”.

Now i’m looking for a way to:
1 - remove small meshes that are just outliers and not part of any tree trunk
2 - remove earlier detected meshes that have incorrect depth to them when a better candidate has been found.

For 1 I have experimented with the following:

  • Subscribe to the meshManager.meshesChanged event and each time check trackable meshes not earlier identified and:
    a. just remove meshes under a certain volume:
    for anyone interested:
float CalculateMeshVolume(Mesh mesh)
{
        Vector3[] vertices = mesh.vertices;
        int[] triangles = mesh.triangles;

        float volume = 0f;
        Vector3 meshCenter = mesh.bounds.center;

        for (int i = 0; i < triangles.Length; i += 3)
        {
            Vector3 v1 = vertices[triangles[i]];
            Vector3 v2 = vertices[triangles[i + 1]];
            Vector3 v3 = vertices[triangles[i + 2]];

            // Calculate the signed volume of the tetrahedron formed by the triangle and the center
            float tetrahedronVolume = Vector3.Dot(Vector3.Cross(v1 - meshCenter, v2 - meshCenter), v3 - meshCenter) / 6.0f;

            volume += tetrahedronVolume;
        }

        return Mathf.Abs(volume);
}

and for meshes that are within the size many times they are actually just many small outliers as part of one mesh:
a. weld vertices if they’re close together
b. separate submeshes by performing a depth first search to find all connected vertices
c. only keeping vertices larger than a certain threshold

This is computationally intensive so if theres a way to filter out small meshes in another way I’d love to hear it.

Another thing which helps is just deleting vertices after x seconds since they’re found.

As for 2 removing earlier meshes with incorrect depth thats even more of a headache and currently I’m just indexing and clustering meshes through a DBSCAN and for new meshes checking if theyre close to an existing cluster or forming a new one. (results not there yet, see the image)
image
and

image
image783×585 175 KB

Red Spheres: represent individual mesh center points
Yellow Spheres: represent cluster centroids
Lines: is clustering algo in process

Here I’m really just looking for the best candidate mesh of a single tree trunk in view, so the biggest somewhat continuous mesh OR just a way to get the inner center point of a tree trunk in view. Anyone any suggestions or pointers where to look further in removing all meshes not of importance (as they’re in the way of raycasts to proper meshes behind them:
image

I can get there in a more hacky way (but again not very performant or consistent) by doing a scan left and right as a line, Sampling the SegmentationManager’s GetSemanticChannelTexture at its frameReceived event getting the left and right boundary and the middle point on the x distance. Similarly however as the mesh the depth buffer for the z axis also isnt very correct unless from close up (getting it through the occlusionManager TryAcquireEnvironmentDepthCpuImage and sampling the right screen coordinate from its texture.

Tnx!

2

Hi Roelof,

Instead of creating your own algorithm for this, I would highly recommend researching the following algorithms: Ball Pivoting, Poisson Surface Reconstruction, Alpha Shape, and Marching Cubes. In addition, I would consider moving your computationally intensive logic to a compute shader. Next, for the outliers, you could try sampling several frames over a short timespan before generating the meshes, or you could occasionally run some “cleanup logic” that checks for outliers by comparing to the new, known to be accurate data.

I hope that helps you get on the right track.

Kind regards,
Maverick L.

3

Thanks Maverick, will have a look!