LM Studio
The AI Decoded series demystifies AI while introducing new hardware, software, tools, and accelerations for customers of GeForce RTX PCs and NVIDIA RTX workstations.Large language models are causing changes in productivity (LLM). With considerable training on a variety of material, they can produce essays, describe webpages, and give accurate responses to questions about nearly any topic.
Among the emerging generative AI applications that heavily rely on LLMs are digital assistants, conversational avatars, and customer service representatives.
A large number of the latest LLMs may run locally on PCs or workstations. This is beneficial for several reasons: users can use AI without the internet, protect on-device communications and material, or just make use of the powerful NVIDIA GeForce RTX GPUs in their system. Because of their size and complexity, some models cannot fit into the visual memory (VRAM) of the local GPU and must be installed on hardware in large data centers.
However, on a data-center-class model, RTX-powered PCs can locally speed up a portion of a work by using a technique known as GPU offloading. As a result, users are no longer limited by GPU memory while utilizing GPU acceleration.
Comparing Size, Quality, and Performance
Performance, response quality, and model size are all subject to trade-offs. Although they work more slowly, larger machines usually yield greater results. With smaller models, quality declines while performance rises.This trade-off is not always simple to make. Performance could be more important than quality in some circumstances. Accuracy may be more important to some users for use cases like content creation because it can run in the background. Conversely, a conversational assistant needs to react fast and precisely.
The most accurate LLMs are tens of gigabytes in size and may not fit in a GPU's memory because they are designed to run in a data center. This would normally prevent the application from using GPU acceleration.
However, some of the LLM is used on the GPU and some on the CPU when GPU offloading is in place. This makes it possible for users to take full advantage of GPU acceleration, regardless of model size.
Optimize AI Acceleration Using LM Studio and GPU Offloading
LM Studio is a tool that allows users to download and host LLMs on their desktop or laptop computer, and it has an easy-to-use interface that allows for a lot of customization in how those models work. LM Studio is fully compatible with GeForce RTX and NVIDIA RTX graphics cards because it is built on top of llama.cpp.
GPU offloading and LM Studio both leverage GPU acceleration to speed up a locally hosted LLM, even if the model cannot be fully loaded into VRAM.
To aid in GPU offloading, LM Studio divides the model into smaller parts, or "subgraphs," which represent layers of the model architecture. Instead of being permanently stored on the GPU, subgraphs are loaded and unloaded as needed. Users can adjust the number of these layers that are handled by the GPU by adjusting the GPU offloading slider in LM Studio.
Think about using this GPU offloading technique on a big model, like the Gemma 2 27B. An estimate of the memory required to run the model may be found in "27B," which displays the number of parameters in the model.
According to 4-bit quantization, a technique for reducing an LLM without noticeably reducing accuracy, each parameter needs half a byte of memory. The model should therefore require approximately 13.5 billion bytes, or 13.5GB, plus some overhead, which usually ranges from 1 to 5GB.
To properly accelerate this model on the GPU, the GeForce RTX 4090 desktop GPU needs 19GB of VRAM. With GPU offloading, the model can function on a device with a less potent GPU and still achieve acceleration.
In LM Studio, it is possible to compare the performance effects of different GPU offloading levels against CPU-only. The table below shows the results of running the same query at different offloading levels on a GeForce RTX 4090 desktop GPU.
Customers using this particular model can noticeably outperform others who only use CPUs, even if they have an 8GB GPU. Of course, you can always run a smaller model with full GPU acceleration on an 8GB GPU if it fits entirely in the GPU memory.
0 Comments