Real Time World Query
This sample project provides an example Lens that teaches you how you might use the Real Time World Query API. The Real Time World Query enables virtual objects to interact with the real world, by understanding the real world geometry as seen from the Spectacles.
NOTE: Real Time World Query runs on custom API for Spectacles. It will not run on your phone or in Lens Studio. Due to the experimental nature of the API, the Lens cannot be published, but can be pushed directly to your device.
Project Download
Setting up
Compatibility
Make sure you have updated to the latest version of the Snapchat App on your phone, the Spectacles firmware, and Lens Studio.
- Snapchat v11.63.x and above
- Spectacles v4.073.0038 and above
- Lens Studio v4.13.0 and above
Enable Experimental API in Lens Studio
The experimental features in this API require that you update to Lens Studio 4.13 or later. In your Lens Studio project, you must enable experimental features:
- Open Lens Studio -> Preferences and check “Allow Experimental API.” Now the “Allow Experimental API” option will show in Project Info.
- Open File -> Project Info and check “Allow Experimental API.”
Real Time World Query API
Spectacles understanding the geometry of the world allows the interaction between virtual and real objects. In order to access the Real Time World Query in your script, you can do the following:
var worldQuery = RealTimeWorldQuery.createContext();
Methods
getLineOfSightIntersection |
Gets the position and normal in world space coordinates of the point of collision that is being focused by the user from the center of the display. |
trackEmptySpace |
Gets the position in world space coordinates where there are no collisions with the given area. |
detectBoxCollision |
Gets the overlap percentage [0 - 1] of the volume’s projection on the camera that is in contact with the real world environment. |
detectSphereCollision |
Gets the overlap percentage [0 - 1] of the volume’s projection on the camera that is in contact with the real world environment. |
Line of Sight Intersection API
The Line of Sight Intersection API gives the user information about the real world at the center of the FOV. That information includes a 3D world position and a normal at that point. For example, one could place a photo on the wall, or a cup on a desk just by looking at the target position and triggering the API call.
Declaration
WorldQueryPoint getLineOfSightIntersection()
Returns
WorldQueryPoint - Reference to the WorldQueryPoint which contains vec3 position and vec3 normal of the point of collision. If no collision is detected it will return null.
Examples
//@input Asset.ObjectPrefab objPrefab //@input Component.Text debugText if(typeof(RealTimeWorldQuery) === 'undefined' || !global.deviceInfoSystem.isSpectacles()){ return; } var worldQuery = RealTimeWorldQuery.createContext(); var objSceneObject = script.objPrefab.instantiate(null); var objTransform = objSceneObject.getTransform(); function onUpdate(eventData){ if(objTransform == null){ return; } var results = worldQuery.getLineOfSightIntersection(); if(results == null){ script.debugText.text = "No Point of Collision"; return; } else { script.debugText.text = "Collision Detected!!"; } var hitPosition = results.position; var hitNormal = results.normal; var lookDirection; if(1 - Math.abs(hitNormal.normalize().dot(vec3.up())) < 0.01){ lookDirection = vec3.forward(); } else { lookDirection = hitNormal.cross(vec3.up()); } var toRotation = quat.lookAt(lookDirection, hitNormal); objTransform.setWorldPosition(hitPosition); objTransform.setWorldRotation(toRotation); } script.createEvent("UpdateEvent").bind(function(){ onUpdate(); })
Empty Space API
The Empty Space API finds a position within the user’s view that won’t collide with real world objects. This API provides a sensation of collision avoidance using sphere approximation that is not mathematically guaranteed to be empty in all cases. Moreover, the returned 3D point is smoothed over time and therefore during fast movement it may collide with the scene. For example, one could place a space station with orbiting spaceships that are free to move around it without colliding with real world objects.
Declaration
WorldQueryPoint trackEmptySpace(float freeSpaceRadius, float distanceFromCamera)
Parameters
freeSpaceRadius |
The float value, in centimeters, of the radius of collision to the world. |
distanceFromCamera |
The float value, in centimeters, of the maximum distance the SceneObject should be from the camera. |
If the parameters are 0, the default value for freeSpaceRadius is 100cm and distanceFromCamera is 20cm.
Returns
WorldQueryPoint - Reference to the WorldQueryPoint which contains the vec3 position that has the required radius of free space and is within the distance of the camera in world coordinate space. If no free space is detected, it will return null.
Examples
//@input Asset.ObjectPrefab objPrefab
//@input Component.Camera camera
//@input float moveToSpeedAmount = 0.05;
//@input Component.Text debugText
if(typeof(RealTimeWorldQuery) === 'undefined' || !global.deviceInfoSystem.isSpectacles()){
return;
}
var worldQuery = RealTimeWorldQuery.createContext();
var objSceneObject = script.objPrefab.instantiate(null);
var objTransform = objSceneObject.getTransform();
var cameraTransform = script.camera.getTransform();
var lastSafePosition = vec3.zero();
function onUpdate(eventData){
if(objTransform == null){
return;
}
var results = worldQuery.trackEmptySpace(0, 0);
if(results == null){
script.debugText.text = "No Free Safe Space";
moveToDestination(objTransform, lastSafePosition);
return;
} else {
script.debugText.text = "Free Safe Space Detected!";
}
var safePosition = results.position;
moveToDestination(objTransform, safePosition);
lastSafePosition = safePosition;
}
function moveToDestination(object, toPosition){
var toPosition = vec3.lerp(object.getWorldPosition(), toPosition, script.moveToSpeedAmount);
var toRotation = quat.lookAt(cameraTransform.forward, vec3.up());
object.setWorldPosition(toPosition);
object.setWorldRotation(toRotation);
}
script.createEvent("UpdateEvent").bind(function(){
onUpdate();
})
Real World Collision API Overview
The Real World Collision API provides a simple collision check of your virtual content to the real world environment using a Box or Sphere collision detection. The API requires a collider description - a box (for which a center point, dimensions, and rotation must be provided) or a sphere (requires center point and radius).
Declaration
float detectBoxCollision(vec3 position, vec3 rotation, vec3 dimensions)
float detectSphereCollision(vec3 position, float radius)
Parameters
detectBoxCollision
position |
The vec3 value of the world position of the collider. |
rotation |
The quaternion value of the rotation of the collider. |
dimensions |
The vec3 value of the width, height, and depth of the box collider. 1 unit equals to 1 cm. |
detectSphereCollision
position |
The vec3 value of the world position of the collider. |
radius |
The float value of the radius of the sphere collider. 1 unit equals to 1 cm. |
Returns
float - The percentage value [0-1] of the intersection of the collider to the real world. A higher the value means more of the surface of the collider is colliding with the real world.
Examples
// -----JS CODE----- //@input Asset.ObjectPrefab objPrefab //@input Component.Text debugText //@input Component.Camera camera //@input float maximumDistanceTravel = 1000 //@input float moveToSpeedAmount = 0.1 //@input float hitThreshold = 0.2 if(typeof(RealTimeWorldQuery) === 'undefined' || !global.deviceInfoSystem.isSpectacles()){ return; } var worldQuery = RealTimeWorldQuery.createContext(); var toPos = null; var objSceneObject = null; var objTransform = null; var colliderTransform = null; var cameraTransform = script.camera.getTransform(); script.debugText.text = "Tap on Touchpad"; function onUpdate(eventData){ if(toPos == null){ return; } var smoothPosition = vec3.lerp(objTransform.getWorldPosition(), toPos, 0.1); objTransform.setWorldPosition(smoothPosition); var colliderPosition = objTransform.getWorldPosition(); var colliderRotation = objTransform.getWorldRotation(); var colliderDimension = colliderTransform.getWorldScale(); result = worldQuery.detectSphereCollision(colliderPosition, colliderDimension.x/2); if(script.hitThreshold < result){ toPos = null; script.debugText.text = "Object Hit"; } else if(smoothPosition.distance(toPos) <= 0.5){ script.debugText.text = "No Hit"; toPos = null; } } function fireObject(){ var startPosition = cameraTransform.getWorldPosition(); objTransform.setWorldPosition(startPosition); var travelDirection = cameraTransform.back; toPos = startPosition.add(travelDirection.uniformScale(script.maximumDistanceTravel)); } script.createEvent("TapEvent").bind(function(){ if(objSceneObject == null){ objSceneObject = script.objPrefab.instantiate(null); objTransform = objSceneObject.getTransform(); colliderTransform = objSceneObject.getChild(0).getTransform(); } fireObject(); script.debugText.text = "Firing Object"; }) script.createEvent("UpdateEvent").bind(function(){ onUpdate(); })
Previewing Your Lens
You’re now ready to preview your lens on Spectacles 2021. To pair your lens with your Spectacles 2021, follow the Setting up and Pairing your Next Generation Spectacles article. To enable Real Time World Query on your Spectacles 2021:
- Update Snapchat on your phone
- Go to Spectacles Home in Snapchat
- Tap “Experimental Features”
- Enable “Capture Depth Lenses”