How to retarget 3D pose landmarks (points in 3d space) onto a rigged humanoid model in Three.js?

I’m trying to implement a 3D avatar in Three.js by extracting 3D pose landmarks of a person (frame-by-frame) from a video (using MediaPipe / PoseFormat). I currently have a .pose file that provides per-frame landmarks for body, left hand, and right hand extracted from a video. Each landmark has (x,y,z) plus a confidence score c. Because they originally come in pixel units, I’ve normalized them so I can render them in 3D space.

What I’ve Done So Far:

Stick Figure Animation: I created small spheres for each joint (e.g. shoulder, elbow, wrist) and connected them with cylinders to form a “stick figure.” By positioning each sphere at a landmark’s (x,y,z) and drawing a cylinder between spheres, I see a rough character that does move correctly, mimicking the person in the video correctly.

Loaded a Rigged 3D Model: I’ve imported a .gltf humanoid model that has a skeleton (bones for each body part).

like:

model.traverse((child) => {
  if (child.isBone) {
    bonesMap[child.name] = child;
  }
});

Attempt to Retarget: Per frame, I compute direction vectors for limbs (e.g. upperArm → foreArm = (elbowPos – shoulderPos)) in world space. Then I try to set the bone’s rotation via a function like:

function setBoneDirection(bone, parentPos, childPos) {
  const direction = new THREE.Vector3()
    .subVectors(childPos, parentPos)
    .normalize();
  // Then set the bone’s quaternion somehow...
}

The bones do move, but the directions are off and the arm/forearm flips awkwardly. The elbow bends somewhat correctly but twists in the wrong orientation.

My Main Questions:

Local vs. World Space: My landmarks are in world space, but each bone’s transform in a skeleton is defined in local space (relative to its parent). So if I move the upper arm, the lower arm’s rotation is affected. How do I properly convert my pose data to each bone’s local space so that the chain moves correctly?

Bind Pose / Rest Pose Directions: The rig’s bones likely aren’t aligned along
(0,1,0) (0,1,0) or (0,−1,0) (0,−1,0) in their neutral (bind) pose. How do I figure out each bone’s default axis so I can compute the correct rotation offset?

Robust Approach / Library: Is there a library, algorithm, or code pattern that:

  1. Transforms the landmarks from world space → parent bone’s local space
  2. Determines a bone’s “rest pose direction” by measuring it in the rig’s bind pose
  3. Computes the final rotation for all bones so that the model matches my stick-figure animation?

Here is my current implementation in StackBlitz containing the stickfigure animation and 3d model retargeting attempt: https://stackblitz.com/~/github.com/AzizStark/3DAvatar

Code example: https://github.com/AzizStark/3DAvatar

Any helpful suggestions or code samples on accurately applying frame-by-frame landmark data to a rigged model in Three.js would be super helpful. Thank you!