Original video in Chinese.
Key Takeaway
- RAG has technical limits: traditional chunking severs connections between pieces of knowledge, while GraphRAG uses a graph to solve this but hasn’t gone mainstream (high cost, complex tech, unnecessary for 80% of use cases).
- Implementation difficulty: basic RAG is easy to build (with tools like LangChain/LlamaIndex), but optimization, tuning, advanced techniques, and evaluation are all exhausting and feel kind of mystical.
- Shift in strategy: stop tinkering and use off-the-shelf tools like Cursor (editing + internet access), and shift toward knowledge distillation, like using Gemini to process notes; my advice is to collect enough, then stop and think about the connections, making the leap from information to knowledge.
I haven’t messed around with RAG for a very, very long time.
Actually, my account first got its initial traffic and its first community members because of RAG and personal knowledge base content. But later I gradually stopped paying attention to this technology and the related products. For two reasons.
First, technical limitations.
A year ago, when I came across GraphRAG, I was really impressed, because it solved the serious shortcomings of traditional RAG.
The foundation of traditional RAG is chunking. This cuts off the inherent connections between pieces of knowledge, creating isolated context. For example, suppose there are three chunks. A mentions Zhang San, B mentions Li Si, and C mentions that Zhang San and Li Si are colleagues. Traditional RAG has to retrieve C in order to know the relationship between Zhang San and Li Si.
You can try it yourself: process a novel with traditional RAG, then ask some questions about the characters, and you’ll very clearly see this technical limitation.
So, my knowledge base uses single notes, and when creating and editing notes, I try to make each paragraph express one idea, without too many contextual links or crossovers. All of this is to avoid the limitations of traditional RAG.
GraphRAG is different. Through a knowledge graph, it can connect entities and relationships, and in theory it can perfectly solve the problems of traditional RAG. But after waiting a year, GraphRAG still hasn’t gone mainstream. I think there are probably these three reasons.
First, the cost is too high.
Converting unstructured text into a high-quality knowledge graph is itself a complex NLP task involving entity recognition, relation extraction, and more. This process has very high computational costs, and the accuracy isn’t very stable either, so it still requires a lot of manual validation.
By comparison, traditional RAG just chunks documents and generates vectors, which is much cheaper.
Second, the tech stack is too complex, and implementation is difficult.
Implementing GraphRAG requires too much from the development team. Traditional RAG is relatively easy.
Third, there isn’t enough demand.
Most enterprise and personal knowledge base scenarios are simple Q&A and document lookup. That kind of need probably accounts for 80%, and traditional RAG is enough to handle it. In that case, if you still want to invest in GraphRAG, it’s a bit hard to justify.
So for me, RAG seems to have just stopped at where it was a year and a half ago. The technologies people use today were already things I used back when I was still writing scripts and hand-rolling RAG, like Hybrid Search, Re-ranking, and Multi-Query, all of which are basically patches for this technology. That’s about it.
Second, easy to use is easy; good to use is hard.
Today, whether for companies or individuals, the barrier to building a usable RAG system is actually very low.
If you’re hands-on, the open-source frameworks are already mature, like LangChain and LlamaIndex. They’ve already wrapped the various RAG steps into standardized modules. You just need to call a few functions like building with blocks, and a basic RAG prototype can be up and running.
If you don’t want to tinker, there are also plenty of ready-made tools you can use.
But that’s only the level of usable. If you want to tune the system to the point where it’s actually good to use, there’s still a lot of work to do and a lot of hurdles to clear.
For example, how should your documents be chunked? What size should each chunk be? Should you keep overlap? Which embedding model should you use? All of these can seriously affect retrieval results. And there’s no standard answer, so a lot of experimentation is needed.
To be honest, by the time you’ve tuned it further, it can even start to feel a bit mystical.
Also, if you want better results, you definitely need to use those advanced techniques, like the Hybrid Search I mentioned earlier. That requires a deeper understanding of the whole system, and it’s absolutely not as simple as just calling a few APIs.
Finally, once the system is built, how do you evaluate it? And after all the debugging, did it actually improve or get worse? All of that involves setting up evaluation criteria, and that is also a very big headache.
After I dug deeper into RAG, I felt a strong sense of helplessness. If I really wanted to reach the level I’m personally satisfied with, I’d have to invest a huge amount of effort, and it would keep me from doing a lot of other things.
So, starting this year, on the one hand, I switched to ready-made large tools. For example, Cursor, which I introduced earlier. It can even handle such complicated code relationships, so dealing with my notes is even more of a piece of cake. And Cursor also comes with editing features and internet search, so if I want to write something, it feels very comfortable to use.
On the other hand, compared with information retrieval, I’m now doing more knowledge distillation.
For example, as I shared before, I fed Gemini more than a hundred video scripts, totaling 75,000 characters of content, and had it help me distill the core of my personal IP.
Gemini 2.5 Pro has a large enough context window and strong enough reasoning ability, so it’s very well suited for knowledge distillation. Over the past six months, I’ve used this method a lot, and the gains have been huge.
So I would suggest that once your notes or various materials have accumulated to a certain extent, you should definitely stop and start distilling.
First, the human brain is not a hard drive. Endless collecting only increases cognitive burden.
Second, if you only collect and don’t distill, what you have is just information that stays at the first layer.
The process of self-distillation is forcing yourself to think, to connect, and to create, completing the leap from information to knowledge.
The process of letting AI distill is letting AI help discover blind spots and discover more possible connections.
That’s what I’ve spent the most time doing over the past six months. It’s very useful, and I sincerely recommend it to everyone.
OK, that’s all for this episode. If you want to learn about AI, want to become a super individual, and want to find like-minded people, come join our newtype community. See you next time!