Plan 9: Local LLM
Local LLM Inference Engine (LIE) in action It's running in QEMU with 4G memory (atleast 2G needed) on Intel Core i5 14th Generation. LLM model is QWEN2 coder with tool support (675MB). A model trained for tool support generates tool activation using <tool_call></tool_call> markup. lie qwen2.5-coder-0.5b-instruct-q8_0.gguf -t 0 -m chat --softmax-mode 1 -t temperature is 0 for repeatable output -m chat mode (use -i for one shot query) Chat Template A Chat template defines what the model accepts as special instructions. ggufinfo -c qwen2.5-coder-0.5b-instruct-q8_0.gguf {%- if tools %} {{- '<|im_start|>system\n' }} {%- if messages[0]['role'] == 'system' %} {{- messages[0]['content'] }} {%- else %} {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }} {%- endif %} {{- "\n\n# Tools...