ARKit PlaneDetectionProvider: adding an entity to an anchor

Joseph Simpson Updated on

Placing virtual content on a plane anchor.

Overview

Let’s build on the last example and place an entity when we tap on a plane. We’ll add InputTargetComponent to the plane entities so we can tap them.

func createSimplePlaneEntity(for anchor: PlaneAnchor) -> Entity {
    let extent = anchor.geometry.extent
    let mesh = MeshResource.generatePlane(width: extent.width, height: extent.height)
    let material = OcclusionMaterial()

    let entity = ModelEntity(mesh: mesh, materials: [material])
    entity.transform = Transform(matrix: matrix_multiply(anchor.originFromAnchorTransform, extent.anchorFromExtentTransform))

    entity.generateCollisionShapes(recursive: true, static: true)
    entity.components.set(InputTargetComponent())

    return entity
}

In the example creating simple planes from anchors, we talked about how to combine the anchor.originFromAnchorTransform and the extent.anchorFromExtentTransform to place each entity on the anchors. That applies to placing other entities too. For example, if we wanted to automatically place the frame on the first detected anchor we could do something like this.

let anchorMatrix = anchor.originFromAnchorTransform
let extentMatrix = anchor.geometry.extent.anchorFromExtentTransform
frameEntity.transform = Transform(matrix: matrix_multiply(anchorMatrix, extentMatrix))

Special thanks to Arthur Schiller and John Haney for explaining this when I asked a question.

When we tap one of the walls, we’ll get the transform for the wall anchor and use it to position the frame. Reminder, that transform was calculated when we created the mesh for the plane anchor.

.gesture(TapGesture()
    .targetedToAnyEntity()
    .onEnded { value in
        // Place the frame on on the tapped plane
        frameEntity.transform = value.entity.transform
        frameEntity.isEnabled = true
    })

This approach works, but it has some issues.

  • The frame is placed in the center of the tapped wall, which may not be useful. You can see in the video demo that on one of my walls it appears behind a lamp.
  • Because we hid the wall meshes with occlusion material, we don’t have a way to show visual feedback for where to tap.

We could improve this a bit by using Spatial Tap Gesture to place the frame on a given point on the wall. Apple also has a an example project showing how determine placement on a plane using raycasts.

Video Demo

Full Example Code

struct Example074: View {

    @State var session = ARKitSession()
    @State var planeAnchorsSimple: [UUID: Entity] = [:]
    @State var frameEntity = Entity()

    var body: some View {
        RealityView { content in

            // Load the asset that we'll use to render the portal
            guard let scene = try? await Entity(named: "PortalFrame", in: realityKitContentBundle) else { return }
            content.add(scene)

            // Load the scene that will serve as the content for the portal
            guard let portalContent = try? await Entity(named: "PortalFrameContent", in: realityKitContentBundle) else { return }
            portalContent.components.set(WorldComponent())
            scene.addChild(portalContent)

            // Save the frame so we can access it from the tap gesture
            guard let frame = scene.findEntity(named: "picture_frame_02") else { return }
            frameEntity = frame
            frameEntity.isEnabled = false

            // Set up the portal
            guard let portalEntity = scene.findEntity(named: "portal_entity") else { return }
            portalEntity.components[ModelComponent.self]?.materials[0] = PortalMaterial()
            portalEntity.components.set(PortalComponent(target: portalContent))

        } update: { content in

            for (_, entity) in planeAnchorsSimple {
                if !content.entities.contains(entity) {
                    content.add(entity)
                }
            }

        }
        .gesture(TapGesture()
            .targetedToAnyEntity()
            .onEnded { value in
                // Place the frame on on the tapped plane
                frameEntity.transform = value.entity.transform
                frameEntity.isEnabled = true
            })
        .task {
            try! await setupAndRunPlaneDetection()
        }
    }

    func setupAndRunPlaneDetection() async throws {
        let planeData = PlaneDetectionProvider(alignments: [.vertical])
        if PlaneDetectionProvider.isSupported {
            do {
                try await session.run([planeData])
                for await update in planeData.anchorUpdates {
                    switch update.event {
                    case .added, .updated:
                        let anchor = update.anchor
                        // Only add walls for this demo
                        if(anchor.surfaceClassification == .wall) {
                            let planeEntitySimple = createSimplePlaneEntity(for: anchor)
                            planeAnchorsSimple[anchor.id] = planeEntitySimple
                        }
                    case .removed:
                        let anchor = update.anchor
                        planeAnchorsSimple.removeValue(forKey: anchor.id)
                    }
                }
            } catch {
                print("ARKit session error \(error)")
            }
        }
    }

    func createSimplePlaneEntity(for anchor: PlaneAnchor) -> Entity {
        let extent = anchor.geometry.extent
        let mesh = MeshResource.generatePlane(width: extent.width, height: extent.height)
        let material = OcclusionMaterial()

        let entity = ModelEntity(mesh: mesh, materials: [material])
        entity.transform = Transform(matrix: matrix_multiply(anchor.originFromAnchorTransform, extent.anchorFromExtentTransform))

        entity.generateCollisionShapes(recursive: true, static: true)
        entity.components.set(InputTargetComponent())

        return entity
    }
}

Updated for visionOS 26 to reflect a change in how we access surface classification

// prior to visionOS 26 we use anchor.classification
anchor.classification

// Now we use anchor.surfaceClassification
anchor.surfaceClassification

Support our work so we can continue to bring you new examples and articles.

Support our Work

Download the Xcode project with this and many more examples from Step Into Vision.
Some examples are provided as standalone Xcode projects. You can find those here.

Follow Step Into Vision

Questions or feedback?