Segmentation using TensorFlow

  • Good questions, iam also curious how to do this. Would be nice if someone has an answer!

     

    Cheers,
    SirQu3ntin

    Comment actions Permalink
  • Hey Meneses,

    There is unfortunately no pixel-by-pixel manipulation inside Lens Studio. You can import images/videos and create materials, but no changing of individual pixels.

    With regards to ML models, I don't believe it is possible to use TensorflowJS. Instead, you'll have to convert your model to onnx or pb format and use the ML Component. Not all layer types are supported by Lens Studio, so if your model doesn't import you'll need to find one with a different architecture or train your own. Also, make sure the model version you chose isn't too big to import into Lens Studio.

    https://lensstudio.snapchat.com/guides/machine-learning/ml-component/ml-component-overview/

    If your model outputs a mask, it is possible to access that mask as a texture and then use that as a custom segmentation texture. I'm not sure the alpha channel works on cameras, but the outputted mask from your model can be used to show/hide other elements/effects/layers.

    https://lensstudio.snapchat.com/guides/general/segmentation/

    https://lensstudio.snapchat.com/guides/general/camera/

     

    Comment actions Permalink
Have a comment?

Please sign in to leave a comment.

We're here to help! We just need a little bit of information...

What system are you using?

Have you downloaded the latest version of Lens Studio?

Please download the latest version of Lens Studio. If you still run into this issue, please come back and report it!

Is this issue causing Lens Studio to crash?

What system do you run Lens Studio on?

Version

Graphics

Follow the next steps to help us solve your issue:

  • Copy and paste this text into your TerminalCommand Window
    open ~/Library/Preferences/Snap/Lens\ Studio/ %LOCALAPPDATA%\Snap\Lens Studio Copy Text
  • Press ReturnEnter to run the command. The Lens Studio folder will automatically open
  • Prepare to upload your files: zip the "Log" Folder by right-clicking and choosing "compress."
    Locate the Log.txt file right above it.

    Attach a screenshot of the issue:

Name:

Email:

What is this most relevant to?

Please enter a brief description of your issue:

Thanks for submitting this issue.

Unfortunately, it's likely due to the operating system or hardware you're using – since they don't meet the system requirements for Lens Studio.

Still, we hear you loud and clear, and are logging the issue in case there's ever a workaround we can provide!

Minimum Requirements

Operating System: Windows 10 (64 bit); MacOS 10.11+

Hardware: Minimum of Intel Core i3 2.5Ghz or AMD Phenom II 2.6Ghz with 4 GB RAM; Intel HD Graphics 4000 / Nvidia GeForce 710 / AMD Radeon HD 6450 or better; screen resolution of 1280x768 or higher

We'll try to resolve this issue as soon as possible. Thanks for letting us know about it!

Keep an eye out for a followup email from us. We may have a couple more questions for you, or we might already have a solution to offer.

Happy creating!