Segmentation using TensorFlow
Hello guys,
I am super new on Lense studio. I wanted to use the library js of tensor flow BodyPix https://github.com/tensorflow/tfjs-models/tree/master/body-pix . I dont know how to include models that I compile using TypeScript.
Another option is to use the output of the segmentation of the example code, but I do not have idea how can I get the image from that type of segmentation.
I also wanted to include a canvas into the filter. I mean, draw my own image and manipulate the pixels of the canvas.
Finally, I wanted to know if it is possible to have a transparent background with on the camera (0,0,0,0) (RGBA)
if someone can share to me any tutorial about this I would be grateful.
Best
Good questions, iam also curious how to do this. Would be nice if someone has an answer!
Cheers,
SirQu3ntin
Hey Meneses,
There is unfortunately no pixel-by-pixel manipulation inside Lens Studio. You can import images/videos and create materials, but no changing of individual pixels.
With regards to ML models, I don't believe it is possible to use TensorflowJS. Instead, you'll have to convert your model to onnx or pb format and use the ML Component. Not all layer types are supported by Lens Studio, so if your model doesn't import you'll need to find one with a different architecture or train your own. Also, make sure the model version you chose isn't too big to import into Lens Studio.
https://lensstudio.snapchat.com/guides/machine-learning/ml-component/ml-component-overview/
If your model outputs a mask, it is possible to access that mask as a texture and then use that as a custom segmentation texture. I'm not sure the alpha channel works on cameras, but the outputted mask from your model can be used to show/hide other elements/effects/layers.
https://lensstudio.snapchat.com/guides/general/segmentation/
https://lensstudio.snapchat.com/guides/general/camera/