Original video in Chinese.
Key Takeaway
- Local large language models and knowledge bases can be accessed externally through ngrok, balancing data privacy and security with ease of use.
- ngrok is a tool for intranet penetration that can establish secure internal and external channels, enabling external access to an internal network.
- Deploying ngrok requires installing ngrok, adding an authentication token, and mapping a local port to the outside world through a command.
- By accessing a local knowledge base from outside the network, users can interact with their private knowledge base anytime, anywhere to get information or generate content.
- This article emphasizes the trend of deploying AI locally and the value of ngrok in making that trend possible.
Full Content
A local large language model and a local knowledge base can also be accessed through an external network.
To put it simply, you run a large language model on your home computer, and then when you’re outside, you can access it from your phone too. I’ll give you a quick demo first, and then I’ll tell you: 1. what value this has; 2. how to do it specifically.
First I’ll get the local large language model running. Run Ollama and let it sit in the backend. Then run Docker and start this Open WebUI image.
At this point, through the local link, we can use the local large language model on this ChatGPT-style webpage. And it’s not just chatting with the large language model—you can also import documents and build your own local knowledge base.
If you want to know exactly how to install and use it, watch my previous video. If you have any questions, or want to discuss further, you can come find me in Knowledge Planet (Zhishi Xingqiu). I share content there every day, and also answer questions.
As of now, we’re all running locally. On this basis, to enable external access, you only need to run one more command in the terminal. Then send this link to your phone.
I turn off Wi-Fi on my phone and open the link in 5G mode. You can see the same interface, the same large model list, the same documents, which means we’ve connected successfully.
So, after going through all this trouble, what exactly is it for?
For me, the most valuable part is that I can access my private local knowledge base anytime, anywhere.
Actually, this idea is just like NAS.
You have a lot of private documents stored on your home computer. You can’t possibly back up those documents on every computer, and it’s even less possible to upload them to the cloud. So when you’re outside, for example at work or on a business trip, and want to use them, you can access the NAS at home through the external network.
So accessing a local knowledge base from outside the network works the same way. The only difference is that we’re not directly accessing those documents—we’re interacting with the local large language model, letting it tell us what’s in the documents, or letting it generate new content based on the documents.
So all the materials I’ve accumulated over so many years, my entire knowledge system, can be fed to the large language model. They all stay on my home computer, very safely. Then I can access them wherever I go. That’s the part that attracts me the most.
To make this work, you only need to install one thing: ngrok.
If you’re a developer, you should be very familiar with ngrok. It’s both a tool for intranet penetration and a reverse proxy service. Simply put, ngrok establishes a secure channel between the internal and external networks, letting you access the internal network through the external network.
Installing and configuring ngrok only requires running three commands in the terminal. I’m using Windows as an example here. If you’re on MacOS, only the installation command in the first step is different.
First command: choco install ngrok.
This choco install command is specifically used to install and update software packages on Windows systems, somewhat like pip install. But pip install is specifically for the Python environment.
I estimate that most people will run into an error at this step, because Chocolatey isn’t installed on the system. Whenever I encounter an error, I usually ask GitHub Copilot to help me analyze it. You can also use ChatGPT—just copy and paste all the error messages you can’t understand. Then follow the prompts and it’s done.
You see, GitHub Copilot told me to first open PowerShell as an administrator, and then that line of command will automatically download and install Chocolatey, after which you just restart PowerShell.
So it’s completely not difficult. When you see an error, don’t be afraid—just ask the large language model.
Second command: ngrok config add-authtoken <TOKEN>
After registering on the ngrok official website, you’ll get a dedicated authentication token. This command adds your token to the configuration.
Third command: ngrok http http://localhost:3000
This localhost:3000 is the port that Ollama uses locally. You can simply understand it as this command telling ngrok to connect port 3000 to the outside world for me.
When you successfully execute the third command as well, you’ll see in the terminal that ngrok has given you a link—this is the link we can use externally. Just like in the demo I showed a moment ago, send it to yourself, and you can use it from outside too.
If you feel it’s too much trouble to send the link to yourself every time, you can also create a fixed link in the background and configure it with a single command. That way, every time it starts, ngrok will use the same link. Then you can send that webpage to your phone’s home screen, and just tap it each time.
OK, that’s the basic usage of ngrok. The point of this video is to open up a new way of thinking for everyone. Just like I share in Knowledge Planet (Zhishi Xingqiu):
Starting this year, AI will accelerate in adoption. Two paths: 1. move onto terminals; 2. applications will explode.
So, after running large language models on-device and building knowledge bases, if they can also be accessed through the external network, then you can have both data privacy and security as well as ease of use. That’s the value of ngrok.
If you have any suggestions or questions, you can come to the Planet to discuss them. See you in the next episode.