What it is
AppleNeuralEngine-Kit is my toolkit for running large language models on Apple silicon through the Neural Engine, not only the GPU. It covers conversion, inference, and a small interface layer for macOS and iOS.
The README calls out architecture-aware optimization (Llama, Qwen, Mistral, and similar), a SwiftUI chat UI, a visual converter with progress and ETA, KV-cache-oriented prefill models, and multi-function chunks aimed at keeping conversion memory in check.
Why it matters
On-device LLMs on Apple hardware are a different constraint problem than server GPUs. The ANE has its own op support, memory limits, and conversion quirks. I have been poking at that path since the Core ML conversion scripts I published in 2024.
How it works
Conversion analyzes the model, splits embeddings / FFN / LM head, applies ANE-oriented optimizations, chunks the graph, and quantizes. Python scripts (convert_hf_to_coreml.py) and Swift CLIs (ANEModelConverter, ANEToolCLI) share the job; ANEChat is the chat front end.
Related public conversion notes live beside it: Stateful GPT-2 KV cache, Llama Core ML, OpenELM, and the older ane-prototypes tree.
Status
WIP, public at antmikinka/AppleNeuralEngine-Kit. Swift 5.9+, coremltools on the Python side.
Related
- StatefulGPT2CoreML — GPT-2 KV-cache Core ML experiments, with a model on Hugging Face
- LlamaCoreMLOptimized — Llama conversion notes aimed at iPhone-class devices
- OpenELMCoreMLOptimizations — OpenELM conversion and chunking scripts
- ane-prototypes — earlier ANE prototype scripts
- Hugging Face — converted / edge-oriented models