ARKit: Setting up and running a session

Covering the basics to creating and running an ARKitSession with a DataProvider.

The first thing we need to do is update the project plist with the permissions we need.

If we need hand tracking, then we’ll need to provide NSHandsTrackingUsageDescription. We’ll need NSWorldSensingUsageDescription for features like Plane and World detection.

Let’s create a session:

@State var session = ARKitSession()

Then create a function to set up a data provide and run the session. This will use the PlaneDetectionProvider. We provide the alignments we need, then check of plane detection is supported. We can run the session, passing in an array of data providers.

func setupAndRunPlaneDetection() async throws {
    let planeData = PlaneDetectionProvider(alignments: [.horizontal, .vertical, .slanted])
    if PlaneDetectionProvider.isSupported {
        do {
            try await session.run([planeData])
            for await update in planeData.anchorUpdates {
                // Do someing with the data
            }
        } catch {
            print("ARKit session error \(error)")
        }
    }
}

We’ll run this function in a task in the view for our immersive space.

RealityView { content in
    // scene set up 
}
.task {
    try! await setupAndRunPlaneDetection()
}

To recap, the basic steps are

Update the plist with the permissions we need
Create an ARKitSession
Create a DataProvider
Run the session, passing in the DataProvider
Process updates

Let’s take a closer look at the update process

for await update in planeData.anchorUpdates {
    // ARKIt will have updates for us it detects new or changed anchors
}

How we process these updates will depend on the DataProvider, as well as the requirements for our app/scene. Let’s take a look at an example that draws pastel plane entity for each detected plane. We can watch the event types for the anchor updates. For example, when an anchor is added or updated, we generate a new plane. When an anchor is removed, we remove the related plane entity.

for await update in planeData.anchorUpdates {
    switch update.event {
    case .added, .updated:
        let anchor = update.anchor
        if planeColors[anchor.id] == nil {
            planeColors[anchor.id] = generatePastelColor()
        }
        let planeEntity = createPlaneEntity(for: anchor, color: planeColors[anchor.id]!)
        planeAnchors[anchor.id] = planeEntity
    case .removed:
        let anchor = update.anchor
        planeAnchors.removeValue(forKey: anchor.id)
        planeColors.removeValue(forKey: anchor.id)
    }
}

We’ll dive further into plane detection–and the other data providers–in some future examples.

This creates planes based on the extend of the bounds for the anchor. This isn’t ideal, as that may not line up well with what ARKit actually detects. See the posts on PlaneDetectionProvider for some better examples of rendering and working with planes.

struct Example068: View {
    @State var session = ARKitSession()
    @State private var planeAnchors: [UUID: Entity] = [:]
    @State private var planeColors: [UUID: Color] = [:]

    var body: some View {
        RealityView { content in

        } update: { content in
            for (_, entity) in planeAnchors {
                if !content.entities.contains(entity) {
                    content.add(entity)
                }
            }
        }
        .task {
            try! await setupAndRunPlaneDetection()
        }
    }

    func setupAndRunPlaneDetection() async throws {
        let planeData = PlaneDetectionProvider(alignments: [.horizontal, .vertical, .slanted])
        if PlaneDetectionProvider.isSupported {
            do {
                try await session.run([planeData])
                for await update in planeData.anchorUpdates {
                    switch update.event {
                    case .added, .updated:
                        let anchor = update.anchor
                        if planeColors[anchor.id] == nil {
                            planeColors[anchor.id] = generatePastelColor()
                        }
                        let planeEntity = createPlaneEntity(for: anchor, color: planeColors[anchor.id]!)
                        planeAnchors[anchor.id] = planeEntity
                    case .removed:
                        let anchor = update.anchor
                        planeAnchors.removeValue(forKey: anchor.id)
                        planeColors.removeValue(forKey: anchor.id)
                    }
                }
            } catch {
                print("ARKit session error \(error)")
            }
        }
    }
    
    private func generatePastelColor() -> Color {
        let hue = Double.random(in: 0...1)
        let saturation = Double.random(in: 0.2...0.4)
        let brightness = Double.random(in: 0.8...1.0)
        return Color(hue: hue, saturation: saturation, brightness: brightness)
    }
    
    private func createPlaneEntity(for anchor: PlaneAnchor, color: Color) -> Entity {
        let mesh = MeshResource.generatePlane(
            width: anchor.geometry.extent.width,
            depth: anchor.geometry.extent.height
        )
        
        var material = PhysicallyBasedMaterial()
        material.baseColor.tint = UIColor(color)
        
        let entity = ModelEntity(mesh: mesh, materials: [material])
        entity.transform = Transform(matrix: anchor.originFromAnchorTransform)
        
        return entity
    }
}

Download the Xcode project with this and many more examples from Step Into Vision.

Some examples are provided as standalone Xcode projects. You can find those here.

Questions or feedback?Cancel reply

Discover more from Step Into Vision