Plan 9: Local LLM

 

 

Local LLM Inference Engine (LIE) in action

 

It's running in QEMU with 4G memory (atleast 2G needed) on Intel Core i5 14th Generation.

LLM model is QWEN2 coder with tool support (675MB). A model trained for tool support generates tool activation using <tool_call></tool_call> markup. 

lie qwen2.5-coder-0.5b-instruct-q8_0.gguf -t 0 -m chat --softmax-mode 1
 

-t temperature is 0 for repeatable output

-m chat mode (use -i for one shot query)

 

Glossary

  • LLM - Large Language Model
  • GGUF format - a binary format that is optimized for quick loading and saving of LLM models
  • BPE - Byte Pair Encoding is a tokenization algorithm. It converts raw text into numerical token IDs that the neural network can process.  
  • Hugging Face - Open source platform for sharing LLM models
  • LM Studio - Desktop application for running variety of LLM workloads
  • ChatML - Chat Markup Language is a structured format used to organize multi-turn conversations and tool activation.

 

Lessons Learnt

  1. Qwen’s attention scores can sit more than 88 below the max, because of its large-magnitude activations.
  2. /tmp uses ramfs. For large files, ftpfs using /tmp will run out of memory. Solution is to bind a directory location to /tmp before running ftpfs.


Code

 


Comments

Popular posts from this blog

Plan 9 : The Infinity Notebook

Plan 9: Quick Boot with UEFI

Plan 9: Key Visualizer