newtype.
ArticleOctober 10, 2025Free to read

Using Open-Source LLMs Like GPT

Tools like LM Studio let users work with open-source LLMs the same way they would with GPT, enhancing and constraining them through Python scripts and frameworks such as LangChain and Llama Index.

Originally published . English translation: . Read the Chinese original.

Original video in Chinese.

Key Takeaway

  • Tools like LM Studio let users work with open-source LLMs the same way they would with GPT, enhancing and constraining them through Python scripts and frameworks such as LangChain and Llama Index.
  • Running open-source LLMs locally can enable knowledge bases, search engines, and other enhancements, and can also constrain the model according to a workflow.
  • LM Studio provides a local server feature that simulates the OpenAI API interface, allowing GPT-based applications to be migrated seamlessly to open-source LLMs.
  • This local solution does not rely on cloud compute and does not require token fees, giving users the freedom to build customized AI applications.
  • The article emphasizes the cost and flexibility advantages of deploying open-source LLMs locally.

Full Content

If I’m running an open-source LLM locally, it’s not very interesting if I’m only using it for simple chat. What I definitely want is to be able to enhance and constrain the LLM like I would with GPT, through Python scripts and frameworks and tools such as LangChain and Llama Index, for example:

  • Enhancement: by connecting a knowledge base or a search engine, improve the timeliness of the LLM’s information and supplement knowledge in a particular domain.
  • Constraining: handle tasks according to a given workflow and chain of thought, rather than letting it improvise freely.

OpenAI provides an API interface, which makes all of this much easier. In fact, with software like LM Studio, you can achieve the same effect when using open-source LLMs.

In the previous video, I introduced the basic usage of LM Studio.

You can simply think of it like a domestic game emulator platform, where the emulator and game library are all bundled together. There’s no need for complicated setup; once you download it, you can play right away.

On top of that, LM Studio also offers an advanced use case:

As a local server, it provides API services similar to OpenAI’s.

The method is simple:

  1. Load a quantized LLM.
  2. Start the local server.
  3. Get the local server endpoint and set it as the base_url in config_list.

If you’ve previously developed applications based on GPT, this code should feel very familiar.

It’s basically just replacing the part that calls the OpenAI API:

  • The api_key does not need to be a real one; you can use “not-needed” instead.
  • For the model, where you used to choose gpt-3.5 or gpt-4, now enter “local-model”

The rest of the script does not need to change. That means all your previous Python scripts can be carried over and used with open-source LLMs.

For example, if you use Microsoft’s AutoGen to configure agents, you only need to make some changes to config_list, and you can still import llm_config as usual.

Not relying on cloud compute, not having to pay token fees, and being able to build a local solution tailored to your own needs based on LM Studio and open-source LLMs — that’s the part that attracts me the most.