Hey Lens Studio Community!
You might have seen the Lens Challenge we released a couple weeks ago where cute skeletons would dance and repeat your arm and head movements. This was made possible by the new Skeletal Tracking feature available in Lens Studio 2.1, which gives us screen positions of the upper body joints of the human body. We decided to use this information to make a 3D character follow Snapchatter movements.
In this post we wanted to share how this works!
Preparing our character
To make mapping human joints easier, we used a humanoid like 3D model which has similar joints to our skeletal tracking data (this could work with 2D characters as well). Then, we set our character in a T-Pose so that we could have a good starting reference for rotating the joints.
Extracting information from Skeletal Tracking
Next we need to get information about our user to drive our character. To do this, we’ll get the position data from the skeletal tracking and use it to calculate the rotation between each joints (this is done in the Character Controller script). In other words, we take the positions of three tracking points and find the angle between them (bone start position, bone end position and parent position). Then. we use this information to apply that rotation to our model.
Applying the rotation
Now that we have the rotation data, we can use it on the limbs of our model. To do this, we select the model’s bone that we want to rotate, specify the bone forward direction (which axis to rotate around), and a smoothing coefficient. We smooth out the rotation since we care more that the movement looks fluid than precise. We set this up in the Limbs section of the Character Controller script.
Rotating the head and spine is a bit more complicated. Let’s walk through it:
Driving the Head Rotation
Unlike the limbs rotation, we want to make sure the character’s head rotates appropriately to our character’s head. Like in the Limbs, in the controller script, we first specify the character’s head bone. But, instead of driving the head bone directly, we set an “Influence” coefficient to modify how much human head rotation would affect the character head bone’s.
One interesting thing to note is that skeletal tracking provides 2D rotation of the head. We can use Head Binding to get the 3D rotation of the head. Of course, we have to be careful since there's a tradeoff here--the Lens will be more resource hungry since it's adding another tracker.
Making your character more flexible
You can notice that just having arm and head rotations look pretty rigid. Just as the joints of our body influence one another, we can make our model feel more natural by using the tracking data we have to influence other joints. What we can do is use our head rotation to influence our spine, with the bone farthest from the head having the least amount of influence. To do this, in our controller script, we set the spine (or neck as well) bones to the Bones array and set corresponding values in the Influence array, Our character looks much more flexible now!
That is mostly enough, but our character’s lower body is not moving and that looks boring. To make sure our character continues to feel alive we added animation to the bottom. Additionally, we added a second full animation in case the human body is not tracked; we’ll play whole body animation to keep character busy. See examples of animations below:
Animation settings can be found at the and of the CharacterController script UI.
- We have hands and arms that are driven by the skeletal tracking joint positions
- We have a head which rotates relative to the user’s head (using head binding for increased detail)
- We have spine bones which follow the head rotation based on how close it is to the head
- We have legs driven by 3D animation when skeletal tracking is active, and a fallback animation that drives the entire character when it is not active.
Next week we'll take a look at how we can use Inverse Kinematics controllers to add natural movements to the pelvis and legs.
Let us know if there's something you're interested in creating!