I’m trying to implement a 3D avatar in Three.js by extracting 3D pose landmarks of a person (frame-by-frame) from a video (using MediaPipe / PoseFormat). I currently have a .pose file that provides per-frame landmarks for body, left hand, and right hand extracted from a video. Each landmark has (x,y,z) plus a confidence score c. Because they originally come in pixel units, I’ve normalized them so I can render them in 3D space.
What I’ve Done So Far:
Stick Figure Animation: I created small spheres for each joint (e.g. shoulder, elbow, wrist) and connected them with cylinders to form a “stick figure.” By positioning each sphere at a landmark’s (x,y,z) and drawing a cylinder between spheres, I see a rough character that does move correctly, mimicking the person in the video correctly.
Loaded a Rigged 3D Model: I’ve imported a .gltf humanoid model that has a skeleton (bones for each body part).
like:
model.traverse((child) => {
if (child.isBone) {
bonesMap[child.name] = child;
}
});
Attempt to Retarget: Per frame, I compute direction vectors for limbs (e.g. upperArm → foreArm = (elbowPos – shoulderPos)) in world space. Then I try to set the bone’s rotation via a function like:
function setBoneDirection(bone, parentPos, childPos) {
const direction = new THREE.Vector3()
.subVectors(childPos, parentPos)
.normalize();
// Then set the bone’s quaternion somehow...
}
The bones do move, but the directions are off and the arm/forearm flips awkwardly. The elbow bends somewhat correctly but twists in the wrong orientation.
My Main Questions:
Local vs. World Space: My landmarks are in world space, but each bone’s transform in a skeleton is defined in local space (relative to its parent). So if I move the upper arm, the lower arm’s rotation is affected. How do I properly convert my pose data to each bone’s local space so that the chain moves correctly?
Bind Pose / Rest Pose Directions: The rig’s bones likely aren’t aligned along
(0,1,0) (0,1,0) or (0,−1,0) (0,−1,0) in their neutral (bind) pose. How do I figure out each bone’s default axis so I can compute the correct rotation offset?
Robust Approach / Library: Is there a library, algorithm, or code pattern that:
- Transforms the landmarks from world space → parent bone’s local space
- Determines a bone’s “rest pose direction” by measuring it in the rig’s bind pose
- Computes the final rotation for all bones so that the model matches my stick-figure animation?
Here is my current implementation in StackBlitz containing the stickfigure animation and 3d model retargeting attempt: https://stackblitz.com/~/github.com/AzizStark/3DAvatar
Code example: https://github.com/AzizStark/3DAvatar
Any helpful suggestions or code samples on accurately applying frame-by-frame landmark data to a rigged model in Three.js would be super helpful. Thank you!