Original video in Chinese.
Key Takeaway
- Traditional RAG has limitations when handling complex queries and cannot effectively capture entity relationships and hierarchical structures.
- The Second Me project uses a hybrid memory architecture, combining RAG, GraphRAG, and local model fine-tuning, to address the limitations of traditional knowledge bases and aims to create a “second me” that can mimic a user’s style, tone, and areas of concern.
- The hybrid memory architecture has three layers: L0 (fast retrieval layer, RAG), L1 (memory deep processing and structuring layer, GraphRAG), and L2 (model personalization fine-tuning layer).
- The deployment and training process for Second Me includes self-introduction, uploading materials, entering an API key, and selecting a Base Model, with the core training consisting of five steps.
- This project can combine a personal knowledge base with the capabilities of a large language model, offering a more advanced AI interaction experience, and it also has the imaginative potential to publish a personal digital twin online for others to use.
Everyone building an AI knowledge base has one ultimate dream:
To teach AI all the materials at hand, even everything learned over a lifetime. That way, whatever we know, it knows too. And on top of that, it has the supercomputing ability we don’t have.
In simple terms: a second me.
RAG can’t do this. Over the past year, I’ve made many videos about RAG. But the more deeply I understood it and the more I used it, the more I realized its technical limitations.
Think about it: no matter what strategy you use, what gets retrieved in the end is still those chopped-up, independent text chunks. That won’t change.
To achieve the ultimate goal I just mentioned — a “second me” — today’s RAG is still too basic. But I know people must be researching and developing it.
Second Me is, so far, the project that has made me happiest and given me the most hope.
It uses a hybrid memory architecture, somewhat like the short-term memory and long-term memory of our human brains.
To implement this architecture, Second Me uses both RAG and GraphRAG to deeply process the documents users provide. In the end, based on this processed data, it also fine-tunes a model locally.
In other words, this project has actually gone beyond the scope of a knowledge base. Because what we ultimately interact with is a model customized entirely around your style, tone, way of expressing yourself, and the issues you care about most, paired with a structurally processed knowledge base.
This kind of solution is clearly far more advanced than the AI knowledge bases we’re using now.
Hello everyone, welcome to my channel. Humbly speaking, I’m one of the few bloggers in China who can explain both the Why and the How of AI clearly. What I provide is more valuable than tutorials. Remember to follow. If you want to connect with me, come to the newtype community. More than 1,000 friends have already paid to join!
Back to today’s topic: Second Me.
This project was actually discovered by a friend in the community. He asked me in the Planet how to deploy it locally. I tried it, and there were no problems at all — no errors along the way.
Just like GitHub says, first install Python and the Xcode command line tools. That’s required.
Then download the entire repository. I didn’t use that command at the time; instead, I directly downloaded the zip file.
After that, use the cd command to enter the project folder and run make setup. It will automatically install everything needed. You don’t need to worry about anything — just wait.
Once everything is installed, start the project with the make start command, then open the local link and you can begin training. This part is also very simple — it only requires three things:
First, write a self-introduction. Because the materials you provide ultimately have to be combined with your identity attributes in order to portray a vivid you. Otherwise, it’s just a pile of “dead” data.
Second, upload materials. You can paste text directly into it, or upload documents. I directly dragged in the folder containing my video scripts.
Third, enter the OpenAI API key, and choose a Base Model based on your machine’s performance.
I need to mention something here. My machine is an M4 Pro chip with 48GB of unified memory. I usually have no trouble running a 7B model, so I chose 7B. But in the final fine-tuning stage, the memory pressure was still a bit high. I had to close all the software except the project before I could complete it smoothly. As for the OpenAI API, it doesn’t cost much. Look, for nearly 100 documents on my side, it cost $3. Because it uses GPT-4o mini, which is dirt cheap.
Once you’ve set up these three steps, you can officially begin training. The core of Second Me lies in the five-step training process.
In the first step, it needs to download the Base Model, which is Qwen2.5. The final fine-tuning is based on this model.
In the second step, it processes the materials uploaded by the user. This uses RAG technology, meaning all the documents are split into smaller text chunks, then the text chunks are converted into vector representations, and the basic data preparation and index creation are completed.
In the third step, it extracts deeper structured information. This should be GraphRAG technology. Because through the processing in the second step, we get a large number of text chunks, all of which exist independently and are scattered. But if you look at the whole article, there should be logical relationships between the text chunks, or in other words, between the contexts. Due to the limitations of RAG technology, these logical relationships are lost.
So GraphRAG is needed to identify and extract entities such as people, places, concepts, and the relationships between them, forming a graph about you, about your knowledge system.
In the fourth step, it prepares the dataset used for fine-tuning. Having only the user-uploaded materials is not enough for fine-tuning. Because among those materials, which data needs to be enhanced still depends on the processing from the first two steps. For example, data related to the user’s identity traits obviously needs to be strengthened.
In the fifth step, fine-tuning officially begins. This step is the most resource-intensive and also takes the most time. I waited from the afternoon until evening to get it done.
These five steps are essentially implementing the so-called “hybrid memory” architecture. This architecture includes three layers: L0, L1, and L2.
The RAG in the second step corresponds to L0, the “fast retrieval layer.” It is a simple, fast memory retrieval mechanism. You can think of it as the human brain’s short-term memory.
The GraphRAG in the third step corresponds to L1, the “memory deep processing and structuring layer.” Its goal is to deeply understand memory content and extract structured information. You can think of it as the human brain’s long-term memory.
The first, fourth, and fifth steps correspond to L2, the “model personalization fine-tuning layer.” Its output is that Second Me model, one that has been personally trained, can mimic the user, and can make use of the knowledge base.
All of this is meant to let the three layers work together to achieve a “hybrid memory” system that combines a user’s personal memory, or in other words, personal knowledge base, with the capabilities of a large language model.
As for how to use it, it’s very simple: click “Start Device,” and this project will load the trained model through llama.cpp, then you just chat normally.
Because the materials are all video scripts I wrote myself, I know them all too well. My experience using it is that it doesn’t feel as mechanical as RAG, without that stiff, parroting quality. It seems to understand all the content just like I do. That is the experience brought by “hybrid memory.”
To be honest, a year ago when I was having tea with VC friends, we were already discussing building a personal AI twin by referring to the characteristics of human memory. Now that I see someone has made it happen, I’m genuinely very happy.
And Second Me has even more ideas. For example, you can publish your own twin online for others to use. You can also chat with other people’s online twins. That’s incredibly imaginative, and everyone can think about it carefully.
I recommend that everyone try Second Me if they have the conditions to do so. It’s impressive to be able to use RAG, GraphRAG, and fine-tuning all together, each where it fits best.
OK, that’s it for this episode. If you want to understand AI, want to become a super individual, and want to find like-minded people, come join our newtype community. See you next time!