The forum on this site is closed for new comments and posts. Continue the conversation in the new forum, and learn more here.
  • Somewhat possible, but not entirely. 

    You can use the skeletal tracking: https://lensstudio.snapchat.com/templates/object/skeletal/

    which will track the arms and head but it doesn't track legs.

    Personally would probably detect the position by checking the position of the tracking points on the user against targets displayed on screen to the user.

    I believe object tracking is already giving screen position and not world position so you can simply just compare the distance between the vectors.

    Comment actions Permalink
  • Hi, Laurent

    Definitely this is possible, I've been playing with this concept some time ago. I can share some stuff with you 

    So first I started with adding some helpers

    I defined a Bone as a combination of three Skeletal Tracking Points - parent, start  and end point. Very similar to this forum post.

    var Bone = function (parent, start, end) {
    this.parent = { transform: parent.getSceneObject().getFirstComponent("ScreenTransform"), tracking: parent };
    this.start = { transform: start.getSceneObject().getFirstComponent("ScreenTransform"), tracking: start };
    this.end = { transform: end.getSceneObject().getFirstComponent("ScreenTransform"), tracking: end };
    }

    Bone.prototype.rotation = function () {
    var parentDir = (new vec2(this.start.transform.position.x, this.start.transform.position.y).sub(new vec2(this.parent.transform.position.x, this.parent.transform.position.y))).normalize();
    var dir = (new vec2(this.end.transform.position.x, this.end.transform.position.y).sub(new vec2(this.start.transform.position.x, this.start.transform.position.y))).normalize();
    return new vec2(parentDir.dot(dir), parentDir.x * dir.y - parentDir.y * dir.x);
    }

    Bone.prototype.isTracking = function () {
    return (this.parent.tracking.isTracking() && this.start.tracking.isTracking() && this.end.tracking.isTracking())
    }

    Then I represented a Skeleton class which is basically a list of bones

    var Skeleton = function () {
    this.bones = {};
    }

    it has a couple of methods for adding bones and debug

    Skeleton.prototype.add = function (name, parent, start, end) {
    if (start != null && end != null) {
    this.bones[name] = new Bone(parent, start, end);
    }
    else {
    throw ("Need 3 points");
    }
    }

    Also it has a getPose method - that gives you all bone data in specific format

    keleton.prototype.getPose = function () {
    var pose = {};
    for (var boneName in this.bones) {
    var rotation = this.bones[boneName].rotation();
    pose[boneName] = { x: rotation.x.toFixed(5), y: rotation.y.toFixed(5) };
    }
    return pose;
    }

    And also a method that checks if pose is matching with given threshold. 

    Skeleton.prototype.isMatching = function (pose) {
    for (var boneName in pose) {
    if (this.bones[boneName] != undefined) {
    if (!this.bones[boneName].isTracking())
    return false;
    if (pose[boneName].distance(this.bones[boneName].rotation()) >= threshold) {
    return false;
    }
    }
    }
    return true;
    }

    Great! Next we have to create some poses.

    In order to do this I created a bunch of photos of myself and loaded them to the preview panel of the Lens Studio

    Also I created a helper, that outputs current pose in required format to the logger. 

    script.createEvent("TapEvent").bind(function (eventData) {
    print(JSON.stringify(skeleton.getPose()));
    });

    It outputs something like this. The pose is output as a string version of a JSOn format. 

    Then I collected like 10 of different poses, added them as a list of strings to the PoseManager script 

     

    var poseJSONs = []
    //1
    poseJSONs.push('{"Upperarm.L":{"x":"0.71652","y":"-0.69756"},"Forearm.L":{"x":"0.08380","y":"-0.99648"},"Upperarm.R":{"x":"0.69033","y":"0.72349"},"Forearm.R":{"x":"0.28301","y":"0.95912"}}');
    //2
    poseJSONs.push('{"Upperarm.L":{"x":"0.99877","y":"0.04968"},"Forearm.L":{"x":"0.96434","y":"0.26467"},"Upperarm.R":{"x":"0.32683","y":"-0.94508"},"Forearm.R":{"x":"0.97679","y":"-0.21419"}}');
    //3
    poseJSONs.push('{"Upperarm.L":{"x":"0.99877","y":"0.04968"},"Forearm.L":{"x":"0.96434","y":"0.26467"},"Upperarm.R":{"x":"0.32683","y":"-0.94508"},"Forearm.R":{"x":"0.97679","y":"-0.21419"}}');
    //4
    poseJSONs.push('{"Upperarm.L":{"x":"0.99784","y":"0.06575"},"Forearm.L":{"x":"0.99707","y":"0.07650"},"Upperarm.R":{"x":"0.96594","y":"0.25878"},"Forearm.R":{"x":"0.97441","y":"-0.22479"}}');
    //5

    and created a textures for them to display (sorry for my drawing skills ;) )

    I will not get deeper into details of my gameplay but basically if gives you a couple seconds to fit the given pose. It compares the directions of upper-body tracking points with the current pose and reacts on the result. 

    I added behavior script that sends custom trigger to start the game when recording started. For debugging purposes in studio you can change event type to touch

    You can find my project here, it is not optimal, just a draft. Please extend it to fulfill your needs

    Let me know if there are any questions !

    Best

    Olha

    Comment actions Permalink
  • Apoc and Olha thanks you very much for your help ! 

    I will try to understand everything ... once again thanks for your support

    Comment actions Permalink

We're here to help! We just need a little bit of information...

What system are you using?

Have you downloaded the latest version of Lens Studio?

Please download the latest version of Lens Studio. If you still run into this issue, please come back and report it!

Is this issue causing Lens Studio to crash?

What system do you run Lens Studio on?

Version

Graphics

Follow the next steps to help us solve your issue:

  • Copy and paste this text into your TerminalCommand Window
    open ~/Library/Preferences/Snap/Lens\ Studio/ %LOCALAPPDATA%\Snap\Lens Studio Copy Text
  • Press ReturnEnter to run the command. The Lens Studio folder will automatically open
  • Prepare to upload your files: zip the "Log" Folder by right-clicking and choosing "compress."
    Locate the Log.txt file right above it.

    Attach a screenshot of the issue:

Name:

Email:

What is this most relevant to?

Please enter a brief description of your issue:

Thanks for submitting this issue.

Unfortunately, it's likely due to the operating system or hardware you're using – since they don't meet the system requirements for Lens Studio.

Still, we hear you loud and clear, and are logging the issue in case there's ever a workaround we can provide!

Minimum Requirements

Operating System: Windows 10 (64 bit); MacOS 10.11+

Hardware: Minimum of Intel Core i3 2.5Ghz or AMD Phenom II 2.6Ghz with 4 GB RAM; Intel HD Graphics 4000 / Nvidia GeForce 710 / AMD Radeon HD 6450 or better; screen resolution of 1280x768 or higher

We'll try to resolve this issue as soon as possible. Thanks for letting us know about it!

Keep an eye out for a followup email from us. We may have a couple more questions for you, or we might already have a solution to offer.

Happy creating!