Real Time World Query

This sample project provides an example Lens that teaches you how you might use the Real Time World Query API. The Real Time World Query enables virtual objects to interact with the real world, by understanding the real world geometry as seen from the Spectacles.

NOTE: Real Time World Query runs on custom API for Spectacles. It will not run on your phone or in Lens Studio. Due to the experimental nature of the API, the Lens cannot be published, but can be pushed directly to your device.

Project Download

Download Here

Setting up

Compatibility

Make sure you have updated to the latest version of the Snapchat App on your phone, the Spectacles firmware, and Lens Studio.

• Snapchat v11.63.x and above
• Spectacles v4.073.0038 and above
• Lens Studio v4.13.0 and above

Enable Experimental API in Lens Studio

The experimental features in this API require that you update to Lens Studio 4.13 or later.  In your Lens Studio project, you must enable experimental features:

1. Open Lens Studio -> Preferences and check “Allow Experimental API.”  Now the “Allow Experimental API” option will show in Project Info.  
2. Open File -> Project Info and check “Allow Experimental API.”

Real Time World Query API 

Spectacles understanding the geometry of the world allows the interaction between virtual and real objects. In order to access the Real Time World Query in your script, you can do the following:

var worldQuery = RealTimeWorldQuery.createContext();

Methods

Line of Sight Intersection API

The Line of Sight Intersection API gives the user information about the real world at the center of the FOV. That information includes a 3D world position and a normal at that point. For example, one could place a photo on the wall, or a cup on a desk just by looking at the target position and triggering the API call.

Declaration

WorldQueryPoint getLineOfSightIntersection()

Returns

WorldQueryPoint - Reference to the WorldQueryPoint which contains vec3 position and vec3 normal of the point of collision. If no collision is detected it will return null.

Examples

//@input Asset.ObjectPrefab objPrefab
//@input Component.Text debugText

if(typeof(RealTimeWorldQuery) === 'undefined' || !global.deviceInfoSystem.isSpectacles()){
    return;
}

var worldQuery = RealTimeWorldQuery.createContext();

var objSceneObject = script.objPrefab.instantiate(null);
var objTransform = objSceneObject.getTransform();

function onUpdate(eventData){    
    if(objTransform == null){
        return;
    }
    
    var results = worldQuery.getLineOfSightIntersection();
    if(results == null){
        script.debugText.text = "No Point of Collision";        
        return;
    } else {
        script.debugText.text = "Collision Detected!!";
    }
    
    var hitPosition = results.position;
    var hitNormal = results.normal;
    
    var lookDirection;
    if(1 - Math.abs(hitNormal.normalize().dot(vec3.up())) < 0.01){
        lookDirection = vec3.forward();
    } else {
        lookDirection = hitNormal.cross(vec3.up()); 
    }
    
    var toRotation = quat.lookAt(lookDirection, hitNormal);
    
    objTransform.setWorldPosition(hitPosition);
    objTransform.setWorldRotation(toRotation);
}

script.createEvent("UpdateEvent").bind(function(){
    onUpdate();
})

Empty Space API

The Empty Space API finds a position within the user’s view that won’t collide with real world objects. This API provides a sensation of collision avoidance using sphere approximation that is not mathematically guaranteed to be empty in all cases. Moreover, the returned 3D point is smoothed over time and therefore during fast movement it may collide with the scene.  For example, one could place a space station with orbiting spaceships that are free to move around it without colliding with real world objects.

Declaration

WorldQueryPoint trackEmptySpace(float freeSpaceRadius, float distanceFromCamera)

Parameters

If the parameters are 0, the default value for freeSpaceRadius is 100cm and distanceFromCamera is 20cm.

Returns

WorldQueryPoint - Reference to the WorldQueryPoint which contains the vec3 position that has the required radius of free space and is within the distance of the camera in world coordinate space. If no free space is detected, it will return null.

Examples

//@input Asset.ObjectPrefab objPrefab
//@input Component.Camera camera
//@input float moveToSpeedAmount = 0.05;
//@input Component.Text debugText

if(typeof(RealTimeWorldQuery) === 'undefined' || !global.deviceInfoSystem.isSpectacles()){
    return;
}
var worldQuery = RealTimeWorldQuery.createContext();

var objSceneObject = script.objPrefab.instantiate(null);
var objTransform = objSceneObject.getTransform();
var cameraTransform = script.camera.getTransform();

var lastSafePosition = vec3.zero();

function onUpdate(eventData){    
    if(objTransform == null){
        return;
    }
    
    var results = worldQuery.trackEmptySpace(0, 0);
    if(results == null){
        script.debugText.text = "No Free Safe Space";  
        moveToDestination(objTransform, lastSafePosition);
        return;
    } else {
        script.debugText.text = "Free Safe Space Detected!";
    }
    
    var safePosition = results.position;
    moveToDestination(objTransform, safePosition);
    lastSafePosition = safePosition;
}
function moveToDestination(object, toPosition){
    var toPosition = vec3.lerp(object.getWorldPosition(), toPosition, script.moveToSpeedAmount);    
    var toRotation = quat.lookAt(cameraTransform.forward, vec3.up());
    object.setWorldPosition(toPosition);
    object.setWorldRotation(toRotation);
}
script.createEvent("UpdateEvent").bind(function(){
    onUpdate();
})

Real World Collision API Overview

The Real World Collision API provides a simple collision check of your virtual content to the real world environment using a Box or Sphere collision detection. The API requires a collider description - a box (for which a center point, dimensions, and rotation must be provided) or a sphere (requires center point and radius).

Declaration

float detectBoxCollision(vec3 position, vec3 rotation, vec3 dimensions)

float detectSphereCollision(vec3 position, float radius)

Parameters

detectBoxCollision

detectSphereCollision

Returns

float - The percentage value [0-1] of the intersection of the collider to the real world. A higher the value means more of the surface of the collider is colliding with the real world.

Examples

// -----JS CODE-----
//@input Asset.ObjectPrefab objPrefab
//@input Component.Text debugText
//@input Component.Camera camera
//@input float maximumDistanceTravel = 1000 
//@input float moveToSpeedAmount = 0.1
//@input float hitThreshold = 0.2

if(typeof(RealTimeWorldQuery) === 'undefined' || !global.deviceInfoSystem.isSpectacles()){
    return;
}
var worldQuery = RealTimeWorldQuery.createContext();

var toPos = null;
var objSceneObject = null;
var objTransform = null;
var colliderTransform = null;
var cameraTransform = script.camera.getTransform();

script.debugText.text = "Tap on Touchpad";

function onUpdate(eventData){  
    if(toPos == null){
        return;
    } 
    
    var smoothPosition = vec3.lerp(objTransform.getWorldPosition(), toPos, 0.1);
    objTransform.setWorldPosition(smoothPosition);
    
    var colliderPosition = objTransform.getWorldPosition();
    var colliderRotation = objTransform.getWorldRotation();
    var colliderDimension = colliderTransform.getWorldScale();
    result = worldQuery.detectSphereCollision(colliderPosition, colliderDimension.x/2);
    
    if(script.hitThreshold < result){
        toPos = null;
        script.debugText.text = "Object Hit";
    } else if(smoothPosition.distance(toPos) <= 0.5){
        script.debugText.text = "No Hit";
        toPos = null;
    }
}
function fireObject(){
    var startPosition = cameraTransform.getWorldPosition();
    objTransform.setWorldPosition(startPosition);
    
    var travelDirection = cameraTransform.back;
    toPos = startPosition.add(travelDirection.uniformScale(script.maximumDistanceTravel));
}

script.createEvent("TapEvent").bind(function(){
    if(objSceneObject == null){
        objSceneObject = script.objPrefab.instantiate(null);
        objTransform = objSceneObject.getTransform();
        colliderTransform = objSceneObject.getChild(0).getTransform();
    }
    fireObject();
    script.debugText.text = "Firing Object";
})

script.createEvent("UpdateEvent").bind(function(){
    onUpdate();
})

Previewing Your Lens

You’re now ready to preview your lens on Spectacles 2021.  To pair your lens with your Spectacles 2021, follow the Setting up and Pairing your Next Generation Spectacles article.  To enable Real Time World Query on your Spectacles 2021:  

• Update Snapchat on your phone
• Go to Spectacles Home in Snapchat
• Tap “Experimental Features”  
• Enable “Capture Depth Lenses”