Original video in Chinese.
Key Takeaway
- The essence of technology: The AI revolution comes from fundamental changes in information form (unified into Token sequences, such as standardized text/visual/audio) and interaction methods (from search to semantic summoning/generation), not simply content generation.
- Token unification: Token, like the physical unit of “meters,” achieves the first-ever grand unification of information in history, eliminating format barriers; all knowledge is stored in vector space, and queries are like fishing or magic spells that summon customized content.
- Development potential: The highly abstract drive of Token is fueling the wave; in the future, quantum/adaptive Tokens may handle the uncertainty of reality and push forward more complex knowledge interaction.
Have you ever thought about this:
Why is AI productive? Why is AI a technological wave, the fourth industrial revolution?
Is it because AI automatically generates content? Because AI automatically completes tasks?
Those are all results, not causes.
To find the answer, we first need to know: what is the essence of technological development?
The development of information, and the development of energy.
Printing and the internet belong to the dimension of information development. Steam engines and nuclear energy belong to the dimension of energy. And AI obviously belongs to the former.
So then, what is the essence of information development?
The development of information form, and the development of interaction methods.
Information went from word of mouth, to paper books, and then to full digitization. That is a change in form. Just this kind of change in form alone has brought huge development to human society.
The same is true for information interaction methods. For example, the mobile internet was a transformation in information interaction methods, bringing years of development dividends. Many companies are highly optimistic about VR and AR for the same reason.
Now let’s look at AI. I believe AI can truly be called a technological revolution and a technological wave precisely because it brings fundamental changes in both information form and information interaction methods at the same time. Notice: “fundamental.”
And what makes this possible all comes down to something very, very small. It’s called “Token.”
Hello everyone, welcome back to my channel. Modestly speaking, I’m one of the few bloggers in China who can clearly explain the Why and the How of AI. What I offer is far more valuable than tutorials. Remember to hit follow. If you want to connect with me, come join our newtype community. This community has been running for 600 days, and more than 1,700 people have already paid to join.
If you are a domestic user, you can join through Knowledge Planet. If you are an overseas user, you can join through Substack. My first course, daily Newsletter, and exclusive videos are all available in the community.
Back to today’s topic: the greatest invention of this era — Token.
Let’s start with information form.
Historically, information forms have always been fragmented:
Books are linear text, paintings are visual symbols, and music is temporal waveforms.
The barriers between these forms bring very large costs. For example, if you want to extract knowledge from a video, you need manual transcription; if you want to integrate across disciplines, you have to bridge things by hand.
There were attempts in the past to unify information forms. For example, XML tried to unify data exchange between different systems; Schema tried to unify data storage and querying so different applications could share the same database. But these attempts were limited to specific domains.
That is, until Token appeared. Whether it’s books, blogs, or videos, they can all be transformed into Token sequences. This is very much like standardized units in physics — for example, unifying length into “meters.”
I believe that almost all human knowledge will be transformed into Tokens. This is a grand unification of information form that has never happened in history.
So, after information form is unified, what are the benefits?
Put simply, all knowledge and all information become Tokens. These Tokens are all placed into an ocean called vector space.
And your query is a “hook.” The model uses the hook you give it to “fish out” relevant fragments from this ocean of vector space.
This is what I’ve said many times in the community: in the AI era, the answers are all there — as long as you ask the right question.
Think about it: almost all human knowledge is in vector space. Obviously, the answer you want is definitely there too. And to fish out a specific answer like fishing, you need a specific hook — in other words, you need to ask the right question.
Note: this process is not search, but summoning.
Summoning is much more advanced than search. Search is keyword matching. Summoning is semantic understanding plus generation.
This is the fundamental transformation in information interaction methods:
For the first time in history, information can be freely summoned by humans and custom-displayed according to human intent.
Information is no longer a preset finished product, but entirely new content dynamically reconstructed.
The fishing analogy just now wasn’t very accurate. It’s actually a bit like magic:
Your intent is the spell. Token is the medium. Vector space is the magic circle.
Following this logic, when you look at the various changes today:
A longer context window means more complex knowledge can be summoned. A better memory mechanism can make summoning persistent, no longer limited to a single conversation.
All of this comes from Token. And the essence of Token is the highly abstracted form of all kinds of information. Only with high abstraction can unification be achieved.
Following this logic further, the next step should be an even greater abstraction of Token, such as quantum Token or adaptive Token. Only then can we handle the more uncertain real world.
OK, that’s all for this episode. If you want to understand AI, want to become a super individual, and want to find like-minded people, come join our newtype community. Then see you next episode!