Original video in Chinese.
Key Takeaway
- Fluxgym is an easy-to-use local Flux LoRA fine-tuning tool that supports 12G-24G VRAM GPUs and can generate virtual models so convincingly they can pass for real.
- LoRA (Low-Rank Adaptation) is a fine-tuning technique that adds a “sticky note”-style skill pack to help a large language model adapt to specific tasks and styles.
- Fluxgym combines the front end of AI-toolkit with the back end of Kohya Sripts, providing an intuitive user interface and plenty of advanced adjustment options.
- This article details Fluxgym’s manual installation steps and emphasizes how training data quality and model selection affect LoRA results.
- Although cloud fine-tuning is fast, local fine-tuning has the advantage when you have idle compute or need to train a lot, and it’s very cheap.
Full Content
With just 12G of VRAM, you can fine-tune Flux LoRA on your own computer and create virtual models that are completely realistic enough to pass for the real thing.
The Fluxgym project supports GPUs with anywhere from 12G to 24G of VRAM, so its hardware requirements are pretty forgiving. It’s extremely simple to use, basically just three steps:
Fill in the parameters according to your needs, set the trigger keywords, upload the source images for training, and then click Start and wait.
After the LoRA is trained, put it into the corresponding folder in ComfyUI, then add a LoRA node in the workflow and you can use it. Super simple, right?
Fluxgym’s front end is forked from the AI-toolkit project, which is why it’s so intuitive and easy to use. Its back end uses the Kohya Sripts project, which allows for a huge number of advanced adjustments. If you understand it, you’ll see a ton of options when you open the Advanced Tab.
That’s why I say Fluxgym is the easiest and best way to fine-tune LoRA.
Hello everyone, welcome to my channel. Humbly speaking, I’m one of the few bloggers in China who can explain both the why and the how of AI clearly. Remember to follow me; if just one of my videos clicks with you, you’ll have gotten your money’s worth. If you want to connect with me, come to the newtype community. More than 500 friends have already joined and paid!
Back to today’s topic: Fluxgym.
In the video before last, I introduced the basic concepts and methods for fine-tuning Flux LoRA. In that video, I used a project deployed in the cloud. With the help of H100s, it only took 20 minutes to train the LoRA. The cost was around two or three dollars.
Honestly, I think that price is actually pretty OK. But if you have idle local compute, might as well use it, and if you need to train a large number of models, you can use the method I’m introducing today.
There are three ways to install Fluxgym. You can install it with one click, install it through Docker, or install it completely manually. I’m going to demonstrate the manual installation here. It’s actually super simple, just follow the official process mechanically.
First, download the project locally. Remember to use the cd command to enter the Fluxgym folder, and also download sd-scripts.
Second, in the root directory, create the environment and start it.
Third, download and install the required dependencies. Note that there are actually two parts here: one is to enter the sd-scripts folder and use pip install to install according to the requirements txt; the other is to go back up one directory and also install via pip install.
The fourth and final step is to install Pytorch.
I installed Fluxgym on both my home PC and my company PC without running into any problems. If you encounter any error during installation, remember: don’t ask me, ask ChatGPT — that is definitely the most effective method. If you ask me, I’ll either ask GPT or Google it and see how other people solved it. By the time you’ve finished asking me, you could have just done it yourself, right?
To start Fluxgym, remember to run this command in the root directory while the environment is activated. Then open this local link and you’ll see the UI.
For the demo, I won’t adjust anything in the parameter configuration section and will just use the default settings.
For model selection, Fluxgym provides three models: dev, schnell, and a dev model they fine-tuned. I’ll just use the official dev model here.
For the training data, I’ll still use the same images as last time, so later we can compare the LoRA results trained by different projects.
After everything is configured, click Start.
The first time you use it, Fluxgym will download the model files, including unet, clip, and vae — four files in total, about 30G altogether. If you’ve already downloaded them before, just copy the files over and place them in the corresponding folders.
For the whole training process, we just wait. I’m doing the demo on the company PC: a 4090 GPU, 64G of RAM, and an i9 CPU. Even with this top-end consumer setup, I can clearly feel the case on the floor heating up much more than before.
When training is finished, you can see that this time it took a total of 70 minutes. How long it takes depends not only on the hardware, but also on your prior settings, such as your step count requirements.
Copy this LoRA file into the LoRA folder in ComfyUI, and you can use it in your workflow. Let’s run a test; the prompt is very simple. From the generation result, this fine-tuning was quite successful and is basically consistent with the source images.
For comparison, let’s look at the LoRA I previously trained in the cloud with AI-Toolkit. Using the same prompt, the generated result this time is basically the same as the one just now.
Earlier I mentioned that Fluxgym supports three models by default, one of which is the fine-tuned one. They recommend using this fine-tuned model for training. The reason is:
the dev model is distilled from the pro model. During the distillation process, certain capabilities or input conditions are weakened or removed. For example, some aesthetic embedding conditions may be constrained during simplification, or the model may have been trained on a smaller dataset. This means that training LoRA with the dev model may not work that well.
To restore the missing capabilities, the developer fine-tuned dev with large-scale, high-quality data. In this way, if you fine-tune again using this customized dev model, you can avoid model collapse or a drop in output quality. When you use it, I still recommend prioritizing this one.
With LoRA, we can achieve the goal of directly generating people or objects with a specified appearance. I know someone will definitely say you can do it without LoRA too. That’s true — for example, with PuLID, you can also generate a person with a specified face based on a reference image.
But there are limitations. For example, the orientation of the generated person will be restricted by the reference image. Also, you’ll often find that one reference image isn’t enough, and you need to add a few more. Then you also need to crop the image and isolate the face area separately.
So, in my view, image-to-image or face swapping are really just stopgap measures. If you want to generate virtual models or objects long-term and stably, you should still train a LoRA. The training cost and usage cost are both very low, so it’s pretty cost-effective.
OK, that’s it for this episode. If you want to discuss AI further, come to the newtype community. See you next time!