Original video in Chinese.

Key Takeaway

• Multimodal applications are the focus of the AI field. With GPT4-V as the “eyes,” GPT-Turbo as the “brain,” and automation tools as the “hands and feet,” AI can operate browsers and computers like a human.
• AI browser operation uses Puppeteer to control Chrome, enabling page search, recognition, and interaction while avoiding the limitations of traditional scraping.
• The Self-Operating Computer (SOC) project is based on GPT4-V and aims to achieve human-level computer operation, though it still faces challenges such as mouse-click precision.
• This article emphasizes that AI’s multimodal capabilities will bring entirely new AI applications and push entrepreneurs and VCs to pay attention to this area.

Full Content

Building applications around multimodality will be the focus of the AI field in the next six months.

If silicon-based life really exists, then multimodality must be the capability it dreams of. To put it simply:

• GPT-V as the eyes.
• GPT-Turbo as the brain.
• Various automation tools as the hands and feet.

Put these three parts together, and AI can operate a browser on the internet like a human, and it can use a mouse to operate a computer like a human. That’s the logic behind this video.

Two examples:

1. Operating a Browser

With Puppeteer, AI can be used to control Chrome. The general steps are:

1. Based on the request, AI uses Chrome to search for and open the required page.
2. Add red boxes to the locations of buttons and other elements on the page to mark them out.
3. Take a screenshot and send the screenshot to GPT4-V for recognition.
4. GPT4-V returns what it “sees” and answers the user’s request.

In addition to recognizing page information, AI can also click links and open subpages.

If you are a developer of AI applications, you’ll understand that this is completely different from the past logic—previously, the approach was to scrape the page content through scraping. This method has two problems:

1. You need to analyze all the data on the entire webpage, including many things that are completely irrelevant, and then find what you want from them. It’s slow and consumes a lot of tokens.
2. Many websites, especially domestic ones, have all kinds of anti-scraping measures, or they require account login.

Rather than trying to solve the problem at the same level as websites and software, why not raise the level? Stand at the higher level of a human and control everything!

2. Operating a Computer

Self-Operating Computer (hereinafter referred to as “SOC”) is a project on GitHub, and it already has 5.2K stars. It uses GPT4-V as the base model, with the goal of reaching human-level computer operation.

With GPT4 as the brain, SOC’s operations really surprised me. For example, it knows to use the search function in macOS to search for Google Chrome and then open it, instead of going into the applications to look for it.

This project is still relatively early-stage. The biggest challenge is that GPT4-V’s error rate when simulating mouse click positions is a bit high. When I was testing it, I could clearly feel that the AI was really trying over and over, struggling hard to click a certain button.

Once this problem is solved, and if OpenAI can further improve GPT4-V’s response speed, SOC will basically be usable. Interaction and control through voice are also completely feasible.

Half a year ago, I was still thinking that maybe more and more software would open up interfaces for AI to integrate with. Looking at it now, the two are completely not on the same level.

When others can already fly, what’s the use of digging your moat any deeper?

Maybe in three months, we’ll see a batch of brand-new AI applications that fully leverage multimodal capabilities. Entrepreneurs and VCs really need to keep up quickly.