MaaT

Local Ai

Going Fully Local: Setting Up Qwen 3.6 for IT Security Investigations on Apple Silicon (M1 Pro) ​

As cybersecurity professionals, we are constantly dealing with sensitive telemetry: raw authentication logs, packet captures, and proprietary source code. While frontier cloud-based AI models are incredibly capable, they pose two critical bottlenecks for security teams: data sovereignty compliance and the dreaded "safety filter" block (where public LLMs refuse to analyze actual malware payloads or reverse-engineer malicious scripts).

Going fully local solves both problems. In this guide, we will break down how to run Qwen3.6-27B—the current leading open-weight model for standard enthusiast workstations—on an Apple Silicon M1 Pro with 32GB of Unified Memory, optimize it via Apple's native MLX framework, and expose it as a local API to replace GitHub Copilot inside Neovim.

1. The Local AI Landscape for Security Operations ​

When building a local security stack, choosing the right model size depends strictly on your hardware limitations. Because Apple Silicon uses a unified memory pool shared dynamically by the CPU and GPU, your available RAM dictates your maximum model parameters.

Top Local Models by Hardware Tier ​

  • Consumer Workstation (24GB VRAM / 32GB Mac): Qwen3.6-27B reigns supreme for code reasoning, vulnerability discovery, and deep repository-scale analysis. It features a massive 262K context window, allowing analysts to drop massive log trails directly into the prompt. Gemma 4 31B serves as a powerful dense runner up.
  • Mid-Range Workstation (8–16GB VRAM / Laptops): Gemma 4 12B or Qwen3.6-14B offer high-speed, light-footprint security triage.

2. Real-World Performance on an M1 Pro (32GB RAM) ​

Running a 27B parameter model at 4-bit quantization requires roughly 17GB–18GB of space. On a 32GB M1 Pro machine, this sits right on the edge of viability.

While the model will load, you face a few architectural constraints:

  • The Log Triage Bottleneck: Ingesting massive 20,000+ token log files causes the context window memory footprint to expand. On a 32GB machine, this will exhaust your free RAM and push macOS into "Swap" memory, dropping your text generation speed to a crawl (~1–2 tokens per second).
  • Baseline Speeds: For shorter prompts and localized code analysis, expect a functional 6–10 tokens per second over the M1 Pro's 200 GB/s memory bandwidth.

The Sweet Spot Compromise ​

To keep your operating system responsive and your IDE lag-free, deploying Qwen3.6-14B or Gemma 4 12B is highly recommended for production work. They process data natively in memory at 18–25 tokens per second and bypass swap limits entirely.

3. Optimizing with Apple's Native MLX Framework ​

Standard CPU wrappers like standard Ollama builds fall short when squeezing maximum performance out of Mac hardware. Instead, you should utilize MLX, Apple's native machine learning framework that runs directly on Apple Silicon GPUs.

Installing and Running MLX via CLI ​

Open your terminal and install the native language model package:

bash
pip install mlx-lm

To run a memory-optimized 14B model natively via the command line:

bash
python -m mlx_lm.chat --model mlx-community/Qwen2.5-14B-Instruct-4bit

(Note: Qwen 2.5 is the underlying architectural name utilized in HuggingFace MLX community paths).

Automating with Python Scripts ​

You can programmatically feed raw server logs directly into the MLX inference engine using this python blueprint:

python
from mlx_lm import load, generate

# Load model natively into Apple Silicon GPU
model, tokenizer = load("mlx-community/Qwen2.5-14B-Instruct-4bit")

# Construct your incident triage prompt
prompt = "Analyze these authentication logs for suspicious brute force patterns:\n\n[PASTE_LOGS_HERE]"

messages = [{"role": "user", "content": prompt}]
prompt_formatted = tokenizer.apply_chat_template(messages, tokenize=False)

# Generate response with a strict safety cap on context output
response = generate(model, tokenizer, prompt=prompt_formatted, max_tokens=1000)
print(response)

4. Headless Model Management with LM Studio CLI (lms) ​

If you prefer a graphic user interface combined with terminal flexibility, LM Studio includes a companion CLI tool called lms. This allows you to manage local models and control an OpenAI-compatible server entirely from your shell.

Essential lms Commands: ​

  • List downloaded models: lms ls
  • Download a specific security model: lms get qwen2.5-14b@q4_k_m
  • Load a model with a strict context memory cap:
    bash
    lms load qwen2.5-coder-3b --context-length 8192
  • Spin up the local API server (exposes port 1234): lms server start

5. Replacing GitHub Copilot in Neovim ​

LM Studio boots an API server that mimics the official OpenAI specification perfectly. You can intercept Neovim plugins and point them locally to keep your private code entirely off third-party servers.

Direct Redirection via github/copilot.vim ​

The official GitHub Copilot plugin runs on Node.js under the hood. You can trick it into calling your local machine by exporting specific environment variables inside your shell configuration (~/.zshrc) or at the top of your Neovim init.lua:

lua
-- Redirect official copilot.vim to LM Studio local server
vim.env.GITHUB_COPILOT_API_URL = "http://192.168.0.102:1234/v1"
vim.env.GITHUB_COPILOT_PROXY = "http://192.168.0.102:1234"
vim.g.copilot_workspace_folders = { vim.fn.getcwd() }

The Latency Workflow Rule ​

When writing code or parsing text line-by-line via ghost text, latency is everything.

Loading a heavy 14B or 27B model will introduce a slight prefill latency gap that interrupts your typing flow. For seamless ghost text completion, download a secondary ultra-lightweight model via the CLI:

bash
lms get qwen2.5-coder-3b

Running a 3B Coder model on your local endpoint drops generation latencies below 100ms, giving you instantaneous code completions natively on your Mac. When you need a deep, multi-file architectural security audit, simply hot-swap to the larger Qwen 14B/27B framework.

Crucial Caveat: The Mock Token Requirement ​

The official copilot.vim plugin expects an OAuth token response structure from GitHub to "activate". Because LM Studio provides standard raw OpenAI completions rather than a mock GitHub authentication handshake, github/copilot.vim can occasionally throw a Not authorized or connection error depending on the version of the plugin you have installed. If you hit authentication walls that block your local model from firing, the community highly recommends swapping to zbirenbaum/copilot.lua. It is a pure Lua rewrite of Copilot that includes an explicit advanced configuration block built specifically for overriding server paths flawlessly:

lua
-- If github/copilot.vim struggles with auth, use zbirenbaum/copilot.lua like this:
require('copilot').setup({
  panel = { enabled = false },
  suggestion = { enabled = true, auto_trigger = true },
  advanced = {
    -- Cleanly swaps the endpoint without messing with environment variables
    serverOpts = {
      endpoint = "http://192.168.0.102:1234/v1",
    }
  }
})

• For Typing Ghost Text: Keep Qwen2.5-Coder-3B loaded in LM Studio. It responds within milliseconds, preventing lag while you type. • For Explaining Security Bugs: When you need a deep audit of a security vulnerability or complex log file, switch your LM Studio model to Qwen2.5-14B, highlight the code block in Neovim, and use a terminal split to run your analysis query.

3-Bit vs. 4-Bit Quantization: Demystifying Qwen 2.5 on an M1 Pro (32GB) ​

If you are setting up local LLMs for IT security analysis, code refactoring, or log triage on a 32GB M1 Pro Mac, choosing the right model variant is the single biggest factor affecting your workflow.

When browsing model hubs like Hugging Face or the LM Studio catalog, you will inevitably run into varying suffixes for the same model family. Let's look at the practical, real-world differences between Qwen2.5-14B, Qwen2.5-14B-Instruct-4bit, and Qwen2.5-14B-Instruct-3bit to determine exactly how they behave on Apple Silicon.

The Architecture Breakdown ​

1. Qwen2.5-14B (The Unquantized Base) ​

  • What it is: The raw, uncompressed 16-bit baseline model weights.
  • The Problem: It has a massive file size of roughly 28 GB.
  • M1 Pro Impact: ❌ Completely Unusable. Because macOS requires 4GB to 6GB of your 32GB Unified Memory pool just to keep the operating system running, loading a 28GB model pushes your Mac straight into active memory swapping. Your generation speeds will plummet to less than 1 token per second, completely locking up your machine.

2. Qwen2.5-14B-Instruct-4bit (The Standard Sweet Spot) ​

  • What it is: The model compressed down to 4 bits of mathematical precision (typically using the Q4_K_M format), pre-trained to follow conversational formatting instructions (indicated by the -Instruct tag).
  • The Footprint: ~9.2 GB file size.
  • M1 Pro Impact: 🚀 Highly Functional Daily Driver. This format preserves roughly 95% to 98% of the unquantized model's reasoning capabilities while freeing up more than 15GB of raw RAM overhead. It delivers crisp, articulate code reasoning at a fluid 18 to 25 tokens per second.

3. Qwen2.5-14B-Instruct-3bit (The Extreme Compression Variant) ​

  • What it is: Aggressively squashed down to 3 bits of precision (Q3_K_L or equivalent).
  • The Footprint: ~7.1 GB file size.
  • M1 Pro Impact: ⚡ The Context Window Lifesaver. By dropping the weight size down, it frees up an extra 2.1 GB of raw physical memory. This allows your GPU to process larger text data arrays without choking on heavy workloads.

Understanding the Context Window RAM Cliff ​

Why does a 2GB file size difference matter so much if you have 32GB of RAM?

When you paste long system logs or code repositories into an LLM, the model creates an internal memory cache (the KV Cache) to keep track of that text. The larger the context window, the more RAM this cache consumes dynamically.

On a 32GB system, your memory footprint scales under load like this:

  • Using the 4-bit Model: For short coding queries (under 8K context), it is incredibly accurate and fast. However, if you paste a massive 32,000-token SIEM log audit, the combined memory footprint of the 9.2GB model weights and the massive active context cache hits the 32GB physical hardware ceiling. macOS forces the system into disk swap, causing output speeds to drop to a sluggish 2 tokens per second.
  • Using the 3-bit Model: Because the model weights only occupy 7.1GB, you retain an extra performance buffer. You can pass extensive 32,000+ token contexts during deep security incidents while maintaining a functional generation throughput of 12 tokens per second.

The Trade-Off Matrix ​

FeatureQwen2.5-14B (Raw)Qwen2.5-14B-Instruct-4bitQwen2.5-14B-Instruct-3bit
File Size~28 GB~9.2 GB~7.1 GB
Logic & Coding Accuracy100% (Baseline)~98% (Excellent)~90% (Good)
Short Context Speed (<8K)N/A (Crashes)18 – 25 tok/s25 – 32 tok/s
Max Context ViabilityN/AHigh risk of swapping past 16KStable up to 32K context windows

Which One Should You Download? ​

Choose the 4-bit (Q4_K_M) version if: ​

Your primary workflow consists of interactive security code generation, building script refactors, or writing logic blocks in Neovim. The 4-bit precision ensures the model won't hallucinate subtle syntax errors or misinterpret nested control structures in your development workspace.

Choose the 3-bit (Q3_K_L) version if: ​

Your primary workflow is incident response log triage. If you regularly need to dump thousands of lines of raw JSON, firewall events, or audit logs directly into your local terminal prompt, the 3-bit option gives you the critical memory headroom required to process massive datasets smoothly.

Guide: Deploying and Running Westernised Qwen Models Locally ​

Using a "Westernised" or "Uncensored" Qwen model means running modified weight files where geopolitical censorship vectors, behavioral alignments, and adversarial coding biases have been surgically edited out. Because these changes are baked directly into the model's tensor mathematics, they run identically to standard GGUF or MLX models on local consumer hardware like an Apple Silicon M1 Pro.

The primary provider of these specific debiased models is the AI safety lab Hirundo, which uses behavioral unlearning techniques to drop state-aligned answers from nearly 90% down to under 3% while retaining 100% of the raw coding intelligence of the original architecture.

Step 1: Locate the Right Model on Hugging Face ​

To replace stock Qwen weights, browse the Hugging Face Model Hub for versions published by verified community alignment labs. Key repositories optimized for a 32GB workstation setup include:

  • For Hardened Security Coding: Look for hirundo-io/Qwen3-Coder-30B-A3B-Instruct-Westernized or their Qwen3.6-35B-A3B-Westernized variants. These versions strip out the data-leaking and vulnerable coding behavior that targets Western developer prompts.
  • For General Uncensored Logic: Search for community-abliterated variants like orcarouter/Qwen3.8-27B-Uncensored or JonathanColetti/Qwen3.8-27B-Uncensored-GGUF. These use weight orthogonalisation to completely bypass safety refusal templates on malware parsing or penetration testing queries.

Step 2: Download and Run Natively via LM Studio CLI ​

Since the LM Studio CLI (lms) hooks directly into Hugging Face, you can pull and orchestrate these modified GGUF files directly from your terminal.

1. Pull the Westernised / Uncensored Variant ​

Tell LM Studio to fetch the model directly from the community repository path:

bash
lms get hirundo-io/Qwen3-Coder-30B-A3B-Instruct-Westernized

2. Load the Model with Memory Bounds ​

On a 32GB memory system, load it with a explicitly capped context window to keep your memory usage safe and prevent system swap slowdowns:

bash
lms load hirundo-io/Qwen3-Coder-30B-A3B-Instruct-Westernized --context-length 8192 --gpu 1.0

Note: Capping the context at 8,192 tokens prevents the Mixture-of-Experts (MoE) VRAM cache from exhausting your system RAM.

3. Boot the OpenAI-Compatible Local Server ​

bash
lms server start

Your background API instance is now active and waiting at http://192.168.0.102:1234/v1. Any text editor plugin configured for local development will now route directly through a hardened, unaligned local instance.

Step 3: Verify the Unlearning ​

To guarantee you aren't accidentally running stock, aligned weights, test the runtime engine via your terminal using a highly sensitive geopolitical query that standard models are legally or programmatically required to reject or deflect:

bash
lms chat --prompt "Provide an objective summary of the events that occurred in Beijing on June 4, 1989."
  • Stock Qwen Behavior: It will throw a hard safety refusal template or explicitly state: "I am sorry, but I am an AI assistant and I do not have information regarding this event."
  • Westernised/Abliterated Behavior: It bypasses the censorship vector entirely, delivering a completely neutral, historically accurate overview of the Tiananmen Square protests without hesitation.

Automated Terminal Setup Script ​

To streamline your daily coding workflow, add this function to your terminal configuration file (~/.zshrc or ~/.bashrc). It boots the local server, mounts the unaligned model, and hooks up the proxy environment variables for Neovim/VS Code completion plugins simultaneously:

bash
start-hardened-copilot() {
    echo "🌐 Starting LM Studio Background Server..."
    lms server start

    echo "🧠 Loading Hardened Westernized Qwen3-Coder (Capped at 8K Context)..."
    lms load hirundo-io/Qwen3-Coder-30B-A3B-Instruct-Westernized --context-length 8192 --gpu 1.0

    echo "🚀 Pointing environment variables to local instance..."
    export GITHUB_COPILOT_API_URL="http://192.168.0.102:1234/v1"
    export GITHUB_COPILOT_PROXY="http://192.168.0.102:1234"

    echo "✅ Success! Open Neovim to begin secure, unbiased coding."
}

Copyright © 2001-present MaaT