The journey into agentic coding began with the creation of an LLM Server.

I wanted to understand LLM technology without all the corporatism of the frontier model giants and the dystopia behind them and their monthly plans.

You will understand nothing and be happy.

Leaving the body of human knowledge and its stewardship in the hands of a small group of corrupt corporations is likely not a good model for humanity. Pun intended.

When you compare the frontier model performance to the smaller, superior models being produced in the open source community, the delta is more then disproportional. In particular, the Alibaba Qwen Models are remarkable at only 27B. They seem to be more focused, less corrupt, and likely perform better for agentic tasks then the majors (who always have something to sell).

And they are cheaper.

The dual GPU server discussed below came in at a total cost of < $2500 CAD. Of that $1800 was spent on the GPUs. A Qwen 27B model, on this hardware, can address complex tasks and produce very complex implementation plans for complex software systems, in minutes. And it can build them.

Qwen 27B is a reasoning model. It will think through complex tasks and explore options, often more effectively and certainly more efficiently than most human counterparts.

It is a remarkably productive tool for anyone that can design software.

And there are many benefits to self-hosted agentic coding:

  • The models can be self hosted on modest hardware.
  • The technology can be used honestly, with human dignity, without supporting a dystopian future where kids can’t get access to technology because it is too corrupt for anyone to use safely.
  • Learn how models work and behave and develop skills in hosting inference engines.
  • The output is not skewed or adulturated for the purposes of selling you more
  • There are no token quotas… you and your model can make mistakes and perform rework and it doesn’t cost you more.

I would think that last one would be a sell regardless

The experiment has been very successful and cost effective, well beyond the initial expectations, so it is documented here in case someone else can benefit. Nothing too mysterious or difficult to setup.

Everyone should do it :)

Less customers means less datacenters and less corporate corruption which is good for everyone and the planet. End of PSA.

Qwen 3.6 27B

Once llama.cpp was properly configured and built, the stable and go-to Qwen 3.6 27B NVFP4 model was able to provision over 380k context tokens. A single RTX 5060 Ti 16GB card was unable to squeeze out more then 70k, and this became a constant limitation as the codebase grew in size. Analysis activities would frequently hit out of context errors.

With the 2nd RTX context was no longer a problem. However, on large analysis tasks the model would often stop mid way. There were no out of context errors, but it turned out it was related to max_tokens sent with the request, and the --n-predict value passed to the llama.cpp. Once adjusted, the model could complete long and complex analyses.

Qwen 3.6 Systemd

Description=Llama.cpp Blackwell Reason Model Service
After=dev-nvidia0.device
Wants=dev-nvidia0.device

[Service]
Type=simple
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu/workspace/llama.cpp/build/bin

# Add these environment variables
Environment="PATH=/usr/local/cuda/bin:/usr/local/cuda/bin:$PATH"
Environment="LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH"
Environment="CUDA_VISIBLE_DEVICES=0,1"

ExecStart=/home/ubuntu/workspace/llama.cpp/build/bin/llama-server -m /mnt/models/Qwen3.6-27B-NVFP4-SMALL-MTP-GGUF/Qwen3.6-27B-NVFP4-SMALL-MTP-GGUF.gguf \
  --metrics\
  --fit off\
  --ctx-size 284142 \
  --n-predict 16384 \
  --device CUDA0,CUDA1 \
  --split-mode layer \
  --tensor-split 1,1 \
  --gpu-layers all \
  --flash-attn true \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --reasoning-preserve \
  --n-gpu-layers -1 \
  --host 0.0.0.0 \
  --port 9000

Restart=on-failure
RestartSec=30s

[Install]
WantedBy=multi-user.target

Qwen 3.8 27B

Prior to the 2nd RTX it was not possible to run Qwen 3.8 27B at all. It is currently running and completing the complex tasks listed above. There is less room for context, but so far no ceilings have been hit.

Qwen 3.8 27B w/ MTP enabled performed at an amazing 33-40 t/s in task generation results. The fastest speed yet recorded, on the newest and most capable model.

Qwen 3.8 llama.cpp

The following Qwen 3.8 model and parameters were used for the work performed in the links above:

/home/ubuntu/workspace/llama.cpp/build/bin/llama-server \
  --model /mnt/models/qwen38-27b/Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf \
  --mmproj /mnt/models/qwen38-27b/mmproj-BF16.gguf \
  --ctx-size 128384 \
  -np 1 \
  --flash-attn on \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-min 0.75 \
  --temp 0.7 --top-p 0.95 --top-k 20 \
  --device CUDA0,CUDA1 \
  --split-mode layer \
  --tensor-split 1,1 \
  --gpu-layers all \
  --reasoning-preserve \
  --n-gpu-layers -1 \
  --host 0.0.0.0 \
  --port 9000

Even with MTP enabled all processing remained GPU resident, which was never possible with the single RTX.

It is all local.

There are no monthly charges.

There is no loss of privacy, control or worries of intellectual property.

I’m not using any services I don’t host. Or trust.

And computing and creating software is fun… yet again.

It should always be fun… and free. The alternative is a dystopia.

The Future

ChatGPT created this excellent depiction of the current state of affairs from a rather abstract prompt… maybe it knows something. It does have a lot of parameters.