Using the template that Snapchat provides, there are a few variables you can change when using the notebook. Thinking of the style transfer, you have the notebook, the style image, and the test image. As far as I know, changing the test image doesn't do anything, but the style image plays a huge factor in the outcome of your lens. Mainly Style Transfer is a long trial and error process because it takes hours to see the result. Here is what I know so far about Style Transfer ML:
Style Image: Getting an image from the internet or even taking one, holds information. When you bring that into a photo editing application, you can play around with the colors, brightness, shadows, and contrast. It's best to use these tools to your advantage before you plug your style image into the notebook. Usually, I brighten the shadows and lower the highlights a tad to make the image flat, but not too flat that the computer can't differentiate the objects in the image. After I'm done editing the photo, I export it as a PNG rather than saving, even if it says that it is already one online, just to double-check. You don't want to go through hours of waiting on your ONNX file, and it has an error right before export.
Training Steps: Training Steps can be found in the "Global Variables For Training" section- a quick way to find that section, is to go to the left panel right under the CO to find the Table of Contents- in light green numbers you'll see "10000" which is the default amount of steps. Using this, it typically takes 2.5hours more or less. Changing this number will make the duration of the training loop extend past 2.5 hours, but it makes the model look way better and more exact. It looks better because the longer it trains, the more time the computer has to recognize the details in the image. I'm not too sure, but I think the model caps at a certain point. In my 100k training steps one, I noticed that the loss kept on going back and forth and stopped reducing after ~70k. I typically do 40k-60k steps for most of my models. For reference, my 100k steps took 19hours to complete and failed once because my computer turned off ~17hours the first time.
- Training Steps Underlined In Red:
Weight Decay: I heard of this factor, but I haven't tried it out yet. So far, I know changing the weight decay marginally changes the output.
- Weight Decay Underlined In Red:
Knowing How Long It Takes Till It's Done: Underlined in red in the image below, is where it shows the estimated time left till your ONNX file should be completed and ready to download. Where it says " [03:36<07:10] ", usually the time leans more toward the second group of numbers, so "7:10" would mean 7 minutes and 10 seconds. Keep in mind that this is a Style Transfer run at 1k steps for example purposes, not 10k so it's going to finish sooner.
More Steps = More Exact Image = More Time Spend Training.
How To Download: When your Style Transfer Training is done, an ONNX file will pop up in your folder. Hover over that, click the three dots, hit download, and it'll save to your computer and be ready to import into your Lens Studio.
Adding your ONNX file in Lens Studio: When you're done downloading your ONNX file from the notebook, you can drag and drop it into the resources panel. From there, you click the "ML Component" in the Objects panel then drag your ONNX file into the Inspector panel in the section labeled "Model".
Blend Modes: With the Style Transfer, you aren't limited to how it looks when putting in the model straight away, there are different blend modes you can use to your advantage to get the most out of your ML and the many hours put into it! Going into the Objects Panel, under "Style Transfer Output" there's a tab underneath it called "Output Image". Clicking the Output Image and going into the Inspector panel, under images is the Blend Modes where you can choose from various blend modes to see what looks good to you. You may need to reset the lens in the Preview to see how it looks after changing it.
Things to keep in mind: You can't change the training steps when it's already started, you'll have to factory reset the runtime. Error messages tend to pop up a lot, especially when using the regular version of Google Collab. It's easier to make a new google account as a fresh slate for the ML, I ran into this problem in the beginning. I made a few accounts, one of which I got Collab Pro. On the Collab Pro account, things do run smoother and faster, and I've noticed fewer error messages; it is $10, but it's only a smart investment if you don't want to make sure the runtime doesn't go idle and delete your work. I keep my style images in separate folders in different drives so I can run multiple images at once and see which one I like most to maximize my time.
Some of my Style Transfer ML Lenses:
I hope this helped and will save you time later on! :)
Please sign in to leave a comment.