Original video in Chinese.
Key Takeaway
- The main obstacle to using AI to control an iPhone is detection by Chinese apps. The technical connection itself is straightforward. Apps such as WeChat actively look for AI-driven interaction and may crash or even affect your account, so conventional integrations don’t work.
- The most reliable approach for now is to emulate hardware peripherals with a device like the NanoKVM Go. Connect it to the iPhone over USB-C and use an external display plus mouse and keyboard input. The iPhone sees officially supported peripherals, so apps can’t identify the interaction as AI-driven. MCP then lets agents such as Codex take control.
- The real bottleneck is what the agent can do. It must see the screen, interpret the UI, act, and verify the result. The official MCP was slow at first. I added optimizations such as batching actions and taking fewer screenshots, and open-sourced the enhanced MCP. After Computer Use, iPhone Use is the inevitable next step.
For those of us in China, there’s a real need that’s gone unmet for a long time: using AI to control an iPhone.
The hard part isn’t connecting the iPhone to the AI running on your computer. It’s that Chinese apps won’t let AI connect to them.
These apps automatically try to tell whether a person or AI is operating them. If they detect something risky, they may abruptly close, and your account may face consequences I’d rather not get into.
I’ve talked about this before on Knowledge Planet. Take WeChat. If it opened up to AI, its whole ecosystem would fall apart. There’s no way it’s going to let another company’s AI in. If it integrates AI at all, it will be its own.
After experimenting for a while, I’ve found that the most reliable approach for now is to emulate hardware peripherals. One example is the NanoKVM Go I’m testing.
Look at this little box. It has two USB-C ports, one for power and one for connecting to the iPhone. Once you connect it to Wi-Fi, you can open a web page on your computer and see the iPhone screen. Anything you do on the computer is reflected on the phone.
So how do you let AI control it?
Simple: MCP. The NanoKVM team knew people wanted this, so they built an MCP server. Give the link to an agent such as Codex and let it do the setup. Now your iPhone is an iPhone your AI can use.
Look, Codex is operating WeChat right now. I asked it to summarize the conversation in a group chat. It uses the mouse to scroll through the conversation one screen at a time and writes down what it reads.
With this setup, the iPhone opens right up to agents. How does it work, and why does it get past the apps’ checks?
On the display side, the iPhone already supports external displays. The NanoKVM Go just receives the video signal and sends it over the network to the device you’re controlling it from.
On the input side, the iPhone’s AssistiveTouch supports devices such as mice. So the NanoKVM Go turns actions into peripheral input the iPhone can receive. From the iPhone’s point of view, you’ve plugged in a keyboard and mouse. Everything is completely within the rules.
I think this hardware approach is the most reliable. The operating system officially supports both external displays and peripherals. There’s nothing dodgy about it, so Chinese apps don’t really have anything to object to.
There is a downside, though. It asks a lot of the agent, which needs to do at least four things.
First, capture what’s currently on the phone screen.
Second, identify the buttons, text fields, and state of the page.
Third, click, drag, or type.
Fourth, look at the screen again to verify what happened.
This takes multimodal capabilities, tools to perform actions, and so on. Honestly, when I first used the official MCP, Codex ran extremely slowly. I had it go through several rounds of optimization before it got faster.
At first, a single action and screenshot took 10 seconds. That was unbearable. So I turned the model up to Ultra and let Codex figure out how to optimize it. For instance, it can batch actions so it doesn’t have to stop and think after each move. It also doesn’t need to take a screenshot every time. Taking screenshots and interpreting them both take time.
I even asked Codex whether we should use the Jev model. After trying it, it found the gains were limited. The most important thing was to improve the logic for how it performed the actions.
I’ve packaged all the optimizations so far into an MCP server and released it publicly on GitHub. You can search for it directly or get the link from my newtype community on Knowledge Planet. Think of it as an enhanced version of the official MCP. If I make more improvements, I’ll update the repository too.
Today, AI can already control computers freely. After Computer Use, the next step will be iPhone Use, where AI can control phones too. That’s where things are headed, and no manufacturer can stop it.
Once that matures, AI will be fully integrated into most situations and may even take over some of them. A lot of things will look different by then. I’m really looking forward to it.
OK, that’s it for this episode. If you want to understand AI, reclaim your personal sovereignty, and find like-minded people, come join the newtype community. See you next time!