Lab 079 – Using Physics with Manipulation Component

This labs explores how the new Manipulation Component works with physics bodies.

I was wondering how well the new component worked with physics bodies so I set up a basic example. This lab has a table with a static physics body and three subjects.

• Red box: static body
• Green box: dynamic body
• Blue box: kinematic body

Gravity has been deactivated for all three bodies. Each entity has a basic box collision shape and an input target component.

I loaded this scene in a RealityView and added the default version of manipulation to each entity.

struct Lab079: View {

    var body: some View {
        RealityView { content in

            guard let scene = try? await Entity(named: "ManipulatePhysics", in: realityKitContentBundle) else { return }
            content.add(scene)

            scene.position.y = -0.4

            if let subject1 = scene.findEntity(named: "Subject_1") {
                let mc = ManipulationComponent()
                subject1.components.set(mc)
            }

            if let subject2 = scene.findEntity(named: "Subject_2") {
                let mc = ManipulationComponent()
                subject2.components.set(mc)
            }

            if let subject3 = scene.findEntity(named: "Subject_3") {
                let mc = ManipulationComponent()
                subject3.components.set(mc)
            }

        }
    }
}

struct Lab079: View {

    var body: some View {
        RealityView { content in

            guard let scene = try? await Entity(named: "ManipulatePhysics", in: realityKitContentBundle) else { return }
            content.add(scene)

            scene.position.y = -0.4

            if let subject1 = scene.findEntity(named: "Subject_1") {
                let mc = ManipulationComponent()
                subject1.components.set(mc)
            }

            if let subject2 = scene.findEntity(named: "Subject_2") {
                let mc = ManipulationComponent()
                subject2.components.set(mc)
            }

            if let subject3 = scene.findEntity(named: "Subject_3") {
                let mc = ManipulationComponent()
                subject3.components.set(mc)
            }

        }
    }
}

One thing I was hoping to accomplish was to be able to manipulate an entity as normal, but disallow it from passing through the static table. I could benefit from that in Project Graveyard–which exists in a Volume where I can’t use ARKit Hand Tracking.

Here are the results I’m seeing as of visionOS 26 Beta 8 when using manipulation to move each entity.

• Red box: Most of the time this static body behaves just like an entity without physics. There were a few times when I was able to move the dynamic object with the static one, but I could not reproduce it reliably.
• Green box: The dynamic body behaves normally until I get close to a static body. I can get it to bump into or rest of the table, but with a lot of stuttering. It also still passes through the table and sometimes flashes out of view.
• Blue box: The kinematic body behaves as expected. I can manipulate it as normal and I can use it to move the dynamic body.

The green dynamic box got me close to what I want. If I can find a way to reduce the stutter when it is near a static body I may be able to use this. I might be able to smooth out the translation by intercepting the values using the DidUpdateTransform event. Do you have any ideas? How can we get Manipulation and Physics working well together?

Download the Xcode project with this and many more labs from Step Into Vision.