Custom Crop Texture

Hi there!

You may already be familiar with texture type, that represents some area cropped from another texture based on some object. For example Face Inset, FaceTexture or Face From Image Picker texture. 

With the release of SnapML two new texture asset types have been added - a ScreenCropTexture and a FaceCropTexture. Read more here.

What is new about these? Besides the regular properties, this type of texture also has information about the area of input (original) texture where it was cut from. 

For example Screen Crop Texture that allows you to crop an area of the texture relatively to the Input Texture. The space cropRectangle property is using the same local space ( see picture below)  just like ScreenTransform Anchors. This makes it really convenient to place Screen Images that use crop textures on the screen at the place where they were cropped from. And from the other side - it allows us to create Crop based on some Screen Transform.

 Why do we need this? 

Often a Machine Learning task consists of two steps - for example - detecting the object on the screen (aka bounding box) and then analyzing only that part of a Camera Texture  that contains that object. As we mentioned before ML Models perform the best on the input data, that is similar to the data it was trained on.

The best example -  is a model trained on human faces (Classification Template). Obviously we should use FaceCropTexture for this model input.

Another case - a classification model that is trained on a square image. And its output will be much more accurate if you take a center crop of a device camera texture. See how classifications is more accurate if input image is not stretched:

Or let’s imagine that you want to run a finger joint detection model. But a Finger Tracking Model most probably expects an area of the Screen that has a Hand on it as an input rather than the whole device camera texture.  

Let’s take a look at this last case and try to get a better understanding of a crop texture.

Task 

Create a Custom Crop Texture based on some object. 

First logical step here would be to detect where the object of interest is on the input image. This sounds exactly like a task that Object Detection does. Conveniently we have a couple Object Tracking presets built into Lens Studio, let’s utilize them to find the area of interest. Here we used Cat Tracking, but you can use any of our object tracking types too. 

Tip: The object tracker also works with Screen Transforms

Let’s start with couple steps :

1. First let’s add a built in Object Tracking to find an area of interest. 

• To do this In the new in the  Resources panel click “+” and select ObjectTracking -> Cat Tracking 

1. Secondly let’s create new Screen Crop Texture to get the texture from the area of interest and add it to the scene :

• In the Resources panel click “+” and select  “ScreenCropTexture” . I’ll rename that to Custom Crop Texture
• Then let’s create a new screen image  “+” and select Screen Image 

At this point we should have a hierarchy like this. Notice how both these objects exist in a Screen Space.

1. Lastly  - create a Script and add it to the Scene as well:

• In the Resources panel click “+” and select  “Script” . I’ll rename that to CustomCropController
• Double click on the script to open the Script editor and copy-paste the code provided below.
• Add this script to the scene by dragging and dropping it from the Resources onto the Inspector panel with some SceneObject selected.

Let’s take a look into the code: 

What does this script do? It takes an area of the screen based on the Object Tracking and configures the crop texture to match these parameters. 

//@input Component.ScreenTransform sourceST {"hint" : "Anchors of this screen transform are used for the crop rectangle of a crop texture"}
//@input Asset.Texture cropTexture {"hint" : "Screen Crop Texture Asset"}
//@input vec2 size = {1.0, 1.0} {"hint" : "Size multiplier"}
//@input bool forceSquare
//@input bool rotate
//@ui {"widget" : "separator"}
//@input bool setRectangle {"hint" : "Set this to place screen transform to the place where it was cropped from, works like RectangleSetter"}
//@input Component.ScreenTransform targetST {"showIf" : "setRectangle", "hint" : "Screen transform Rectangle has to ne applied to"}

var cropProvider = script.cropTexture.control;
var aspect; //aspect of device camera texture 

script.createEvent("TurnOnEvent").bind(onTurnOn);

function onTurnOn() {
aspect = cropProvider.inputTexture.control.getAspect()// aspect of device camera texture - available on turn on
script.createEvent("UpdateEvent").bind(updateCrop);
}

function updateCrop() { 
//update crop region
var size = script.sourceST.anchors.getSize();
var center = script.sourceST.anchors.getCenter();

// if we want a square image - take aspect into account
if (script.forceSquare) {
size.x = size.y / aspect;
}
size.x *= script.size.x;
size.y *= script.size.y;

//set crop rect size and center
cropProvider.cropRect.setSize(size);
cropProvider.cropRect.setCenter(center);

//set rotation if needed
if (script.rotate) {
cropProvider.rotation = -script.sourceST.rotation.toEulerAngles().z;
}
//apply same rectangle position and rotation to a screen transform 
if(script.setRectangle) {
script.targetST.anchors.setSize(size);
script.targetST.anchors.setCenter(center);
if (script.rotate) {
script.targetST.rotation = script.sourceST.rotation;
}
}
}

Let’s select the Scene object with a script in the Objects panel, take a look at the script inputs and set them up. 

Source ST - Object Tracking screen transform 

Crop Texture - Screen Crop texture asset 

Size - size multiplier applied to the crop rect

Force square - if enabled - instead of the input texture aspect it will be forced to 1:1 

Rotate - rotation will be applied

Set Rectangle - Enable If you want to use crop information to place another screen transform to the place if was cut from. Works similar to Rectangle Setter (https://lensstudio.snapchat.com/guides/machine-learning/ml-component/crop-textures/), but can be better fit for this case because screen transforms will be updated in the current frame. 

Target St - screen transform to apply to.

Result

As an example here I have a Style Transfer applied only to the cat’s face :)

You might say - what do I do with the cat’s faces now? Not very useful :) But this information tends to reveal some details about crop texture and cover functionality, needed for certain ML projects.

There is also a couple thighs to keep in mind : 

• We are updating the crop in the script Update Event. Make sure that your model starts running after the crop is updated. See more about run modes of ML Component.
• Also don’t forget to add some logic based on whether the object tracking is tracking or not. If the object is not tracking texture probably is not valid.
• If you want to go more advanced you can even create a custom face crop based on the face landmarks. To do this you’ll need to transform world positions of landmarks to the Local Space [-1, 1], then calculate size and rotation of a crop rectangle.

Let’s summarize 

• Learned what the crop texture is and how it can be used

• Created a Screen Crop Texture based on Object Tracking 

• Covered some details that may be useful in creating complex ml projects.

Project File can be found here

Best

Olha