Add VibeASR.cpp release to News (#596 )

Revamp README with structured layout, News section, and Model Releases (#590 )
Update README for BitNet Embedding 0.6B/270M release (#589 )
2026-07-27 13:52:05 +08:00 · 2026-07-21 18:39:52 +08:00 · 2026-07-20 16:49:27 +08:00 · 2026-07-17 11:43:50 +08:00 · 2026-07-16 17:29:40 +02:00 · 2026-07-16 16:01:37 +08:00
41 changed files with 6108 additions and 572 deletions
@@ -1,4 +1,4 @@
 [submodule "3rdparty/llama.cpp"]
 	path = 3rdparty/llama.cpp
-	url = https://github.com/Eddie-Wang1120/llama.cpp.git
-	branch = merge-dev
+	url = https://github.com/isHuangXin/llama.cpp.git
+	branch = release-bitnet-embedding-0.6b-270m
@@ -1,43 +1,105 @@
+<div align="center">
+
 # bitnet.cpp
+
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 ![version](https://img.shields.io/badge/version-1.0-blue)
+[![Hugging Face](https://img.shields.io/badge/HuggingFace-Collection-orange?logo=huggingface)](https://huggingface.co/collections/microsoft/bitnet)
+[![Technical Report](https://img.shields.io/badge/Technical-Report-red?logo=arxiv)](https://arxiv.org/abs/2502.11880)
+[![Demo](https://img.shields.io/badge/Online-Demo-green?logo=microsoft)](https://demo-bitnet-h0h8hcfqeqhrf5gf.canadacentral-01.azurewebsites.net/)
+[![GPU Kernel](https://img.shields.io/badge/GPU-Kernel-6F42C1?logo=github)](https://github.com/microsoft/BitNet/blob/main/gpu/README.md)

-[<img src="./assets/header_model_release.png" alt="BitNet Model on Hugging Face" width="800"/>](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T)
+</div>

-Try it out via this [demo](https://bitnet-demo.azurewebsites.net/), or build and run it on your own [CPU](https://github.com/microsoft/BitNet?tab=readme-ov-file#build-from-source) or [GPU](https://github.com/microsoft/BitNet/blob/main/gpu/README.md).
+<div align="left">

-bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support **fast** and **lossless** inference of 1.58-bit models on CPU and GPU (NPU support will coming next).
+<h3>📰 News</h3>

-The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of **1.37x** to **5.07x** on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by **55.4%** to **70.0%**, further boosting overall efficiency. On x86 CPUs, speedups range from **2.37x** to **6.17x** with energy reductions between **71.9%** to **82.2%**. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. Please refer to the [technical report](https://arxiv.org/abs/2410.16144) for more details.
+<strong>07/23/2026:</strong> 📣 We released <a href="https://github.com/microsoft/VibeASR.cpp"><strong>VibeASR.cpp</strong></a> — a real-time multilingual ASR inference engine on CPU using BitNet I2_S quantization, achieving RTF < 1 with very few threads on x86 (AVX2) and ARM (NEON) platforms. [<a href="https://github.com/microsoft/VibeASR.cpp">Code</a>] [<a href="https://huggingface.co/microsoft/VibeVoice-ASR-BitNet">Models</a>] [<a href="https://arxiv.org/abs/2607.21075">Report</a>] ![NEW](https://img.shields.io/badge/NEW-red)

-<img src="./assets/m2_performance.jpg" alt="m2_performance" width="800"/>
-<img src="./assets/intel_performance.jpg" alt="m2_performance" width="800"/>
+<strong>07/20/2026:</strong> 📣 We released <a href="https://huggingface.co/microsoft/BitNet-embedding-0.6B"><strong>BitNet-embedding-0.6B</strong></a> and <a href="https://huggingface.co/microsoft/BitNet-embedding-270M"><strong>BitNet-embedding-270M</strong></a> on Hugging Face — the first 1-bit embedding models that deliver competitive embedding quality with significantly faster inference on CPUs.
+- **1.42x to 2.28x speedup** over F16 on BitNet-embedding-0.6B prefill (8 threads)
+- **1.32x to 1.74x speedup** over F16 on BitNet-embedding-270M prefill (8 threads)
+- Supports I2_S conversion with optimized kernels on x86 CPUs
+- Lossless inference with 2 bits per weight

->The tested models are dummy setups used in a research context to demonstrate the inference performance of bitnet.cpp.
+07/16/2026: 📣 Released [BitNet Embeddings 0.6B/270M: I2_S Conversion and Inference Optimization](docs/bitnet-embeddings-i2s-guide.md) — detailed guide for converting and running BitNet embedding models with optimized I2_S kernels.

-## Demo
+01/15/2026: 📣 Released [BitNet CPU Inference Optimization](https://github.com/microsoft/BitNet/blob/main/src/README.md) — parallel kernel implementations with configurable tiling and embedding quantization support, achieving **1.15x to 2.1x** additional speedup over the original implementation.

-A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:
+05/20/2025: 📣 Released [BitNet Official GPU inference kernel](https://github.com/microsoft/BitNet/blob/main/gpu/README.md) — extending 1-bit inference beyond CPUs.

-https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1
+04/14/2025: 📣 Released [BitNet Official 2B Parameter Model](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) on Hugging Face — the first official BitNet b1.58 model trained with 4T tokens.

-## What's New:
- 05/20/2025 [BitNet Official GPU inference kernel](https://github.com/microsoft/BitNet/blob/main/gpu/README.md) ![NEW](https://img.shields.io/badge/NEW-red)
- 04/14/2025 [BitNet Official 2B Parameter Model on Hugging Face](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T)
- 02/18/2025 [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://arxiv.org/abs/2502.11880)
- 11/08/2024 [BitNet a4.8: 4-bit Activations for 1-bit LLMs](https://arxiv.org/abs/2411.04965)
- 10/21/2024 [1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs](https://arxiv.org/abs/2410.16144)
- 10/17/2024 bitnet.cpp 1.0 released.
- 03/21/2024 [The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf)
- 02/27/2024 [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764)
- 10/17/2023 [BitNet: Scaling 1-bit Transformers for Large Language Models](https://arxiv.org/abs/2310.11453)
+02/18/2025: 📑 [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://arxiv.org/abs/2502.11880) — system-level paper on bitnet.cpp's architecture and design.

-## Acknowledgements
+11/08/2024: 📑 [BitNet a4.8: 4-bit Activations for 1-bit LLMs](https://arxiv.org/abs/2411.04965) — enabling 4-bit activations for further efficiency gains.
+
+10/21/2024: 📑 [1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs](https://arxiv.org/abs/2410.16144) — the technical report behind bitnet.cpp.
+
+10/17/2024: 📣 bitnet.cpp 1.0 released.
+
+03/21/2024: 📑 [The-Era-of-1-bit-LLMs: Training Tips, Code, FAQ](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf)
+
+02/27/2024: 📑 [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764) — the foundational paper introducing BitNet b1.58.
+
+10/17/2023: 📑 [BitNet: Scaling 1-bit Transformers for Large Language Models](https://arxiv.org/abs/2310.11453) — the original BitNet paper.
+
+</div>
+
+## Overview
+
+bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels that support **fast** and **lossless** inference of 1.58-bit models on **CPU** and **GPU** (NPU support coming next).
+
+Try it out via this [online demo](https://demo-bitnet-h0h8hcfqeqhrf5gf.canadacentral-01.azurewebsites.net/), or build and run it on your own [CPU](https://github.com/microsoft/BitNet?tab=readme-ov-file#build-from-source) or [GPU](https://github.com/microsoft/BitNet/blob/main/gpu/README.md).
+
+bitnet.cpp achieves speedups of **1.37x** to **5.07x** on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by **55.4%** to **70.0%**, further boosting overall efficiency. On x86 CPUs, speedups range from **2.37x** to **6.17x** with energy reductions between **71.9%** to **82.2%**. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. Please refer to the [technical report](https://arxiv.org/abs/2410.16144) for more details.
+
+<img src="./assets/performance.png" alt="performance_comparison" width="800"/>
+
+## Model Releases
+
+### 1. [BitNet-b1.58-2B-4T](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) - 1-bit Large Language Model
+
+**BitNet-b1.58-2B-4T** is the first official BitNet b1.58 model with **2.4B parameters**, trained on **4 trillion tokens**. It is a ternary (1.58-bit) language model that delivers competitive performance with full-precision models of similar size while enabling significantly faster and more energy-efficient inference.
+
+- **Fast CPU Inference**: Achieves up to **6.17x speedup** on x86 CPUs and **5.07x** on ARM CPUs compared to full-precision models.
+- **Energy Efficient**: Reduces energy consumption by up to **82.2%** on x86 and **70.0%** on ARM.
+- **GPU Support**: Official GPU inference kernel available for accelerated deployment.
+- **Chat-Ready**: Supports conversational mode for interactive use.
+
+[🤗 Hugging Face](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) | [🔗 Online Demo](https://demo-bitnet-h0h8hcfqeqhrf5gf.canadacentral-01.azurewebsites.net/) | [📄 Technical Report](https://arxiv.org/abs/2410.16144)
+
+<img src="./assets/bitnet_b1.58_2b_benchmark.png" alt="BitNet b1.58 2B Benchmark" width="600"/>
+
+### 2. [BitNet-embedding-0.6B](https://huggingface.co/microsoft/BitNet-embedding-0.6B) - 1-bit Embedding Model
+
+**BitNet-embedding-0.6B** is a **0.6B-parameter** 1-bit embedding model that achieves competitive embedding quality with significantly faster CPU inference. It is the first model to demonstrate that ternary weights can deliver strong performance on embedding tasks.
+
+- **1.42x to 2.28x speedup** over F16 on prefill (8 threads, x86)
+- **Lossless Quality**: Competitive embedding quality with 2 bits per weight
+- **I2_S Kernel**: Supports optimized I2_S conversion on x86 CPUs
+
+[🤗 Hugging Face](https://huggingface.co/microsoft/BitNet-embedding-0.6B) | [📄 I2_S Guide](docs/bitnet-embeddings-i2s-guide.md)
+
+<img src="./assets/embedding_prefill_0.6B.png" alt="BitNet Embedding 0.6B Prefill Performance" width="600"/>
+
+### 3. [BitNet-embedding-270M](https://huggingface.co/microsoft/BitNet-embedding-270M) - Lightweight 1-bit Embedding Model
+
+**BitNet-embedding-270M** is a compact **270M-parameter** 1-bit embedding model designed for resource-constrained environments, offering fast inference with minimal memory footprint.
+
+- **1.32x to 1.74x speedup** over F16 on prefill (8 threads, x86)
+- **Lossless Quality**: Competitive embedding quality with 2 bits per weight
+- **Lightweight**: Only 270M parameters for edge deployment scenarios
+
+[🤗 Hugging Face](https://huggingface.co/microsoft/BitNet-embedding-270M) | [📄 I2_S Guide](docs/bitnet-embeddings-i2s-guide.md)
+
+<img src="./assets/embedding_prefill_270M.png" alt="BitNet Embedding 270M Prefill Performance" width="600"/>
+
+
+## Supported Models

-This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp) framework. We would like to thank all the authors for their contributions to the open-source community. Also, bitnet.cpp's kernels are built on top of the Lookup Table methodologies pioneered in [T-MAC](https://github.com/microsoft/T-MAC/). For inference of general low-bit LLMs beyond ternary models, we recommend using T-MAC.
-## Official Models
 <table>
-    </tr>
    <tr>
        <th rowspan="2">Model</th>
        <th rowspan="2">Parameters</th>
@@ -49,6 +111,9 @@ This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp)
        <th>TL1</th>
        <th>TL2</th>
    </tr>
+    <tr>
+        <th colspan="6" style="text-align:left;">Official Models</th>
+    </tr>
    <tr>
        <td rowspan="2"><a href="https://huggingface.co/microsoft/BitNet-b1.58-2B-4T">BitNet-b1.58-2B-4T</a></td>
        <td rowspan="2">2.4B</td>
@@ -63,23 +128,36 @@ This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp)
        <td>&#9989;</td>
        <td>&#10060;</td>
    </tr>
-</table>
-
-## Supported Models
-❗️**We use existing 1-bit LLMs available on [Hugging Face](https://huggingface.co/) to demonstrate the inference capabilities of bitnet.cpp. We hope the release of bitnet.cpp will inspire the development of 1-bit LLMs in large-scale settings in terms of model size and training tokens.**
-
-<table>
+    <tr>
+        <td rowspan="2"><a href="https://huggingface.co/microsoft/BitNet-embedding-0.6B">BitNet-embedding-0.6B</a></td>
+        <td rowspan="2">0.6B</td>
+        <td>x86</td>
+        <td>&#9989;</td>
+        <td>&#10060;</td>
+        <td>&#10060;</td>
    </tr>
    <tr>
-        <th rowspan="2">Model</th>
-        <th rowspan="2">Parameters</th>
-        <th rowspan="2">CPU</th>
-        <th colspan="3">Kernel</th>
+        <td>ARM</td>
+        <td>&#10060;</td>
+        <td>&#10060;</td>
+        <td>&#10060;</td>
    </tr>
    <tr>
-        <th>I2_S</th>
-        <th>TL1</th>
-        <th>TL2</th>
+        <td rowspan="2"><a href="https://huggingface.co/microsoft/BitNet-embedding-270M">BitNet-embedding-270M</a></td>
+        <td rowspan="2">270M</td>
+        <td>x86</td>
+        <td>&#9989;</td>
+        <td>&#10060;</td>
+        <td>&#10060;</td>
+    </tr>
+    <tr>
+        <td>ARM</td>
+        <td>&#10060;</td>
+        <td>&#10060;</td>
+        <td>&#10060;</td>
+    </tr>
+    <tr>
+        <th colspan="6" style="text-align:left;">Community Models</th>
    </tr>
    <tr>
        <td rowspan="2"><a href="https://huggingface.co/1bitLLM/bitnet_b1_58-large">bitnet_b1_58-large</a></td>
@@ -153,12 +231,12 @@ This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp)
    </tr>
 </table>

-
+❗️**We use existing 1-bit LLMs available on [Hugging Face](https://huggingface.co/) to demonstrate the inference capabilities of bitnet.cpp. We hope the release of bitnet.cpp will inspire the development of 1-bit LLMs in large-scale settings in terms of model size and training tokens.**

 ## Installation

 ### Requirements
- python>=3.9
+- python>=3.10
 - cmake>=3.22
 - clang>=18
    - For Windows users, install [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/). In the installer, toggle on at least the following options(this also automatically installs the required additional tools like CMake):
@@ -185,7 +263,7 @@ cd BitNet
 2. Install the dependencies
 ```bash
 # (Recommended) Create a new conda environment
-conda create -n bitnet-cpp python=3.9
+conda create -n bitnet-cpp python=3.10
 conda activate bitnet-cpp

 pip install -r requirements.txt
@@ -216,6 +294,7 @@ optional arguments:
  --quant-embd          Quantize the embeddings to f16
  --use-pretuned, -p    Use the pretuned kernel parameters
 </pre>
+
 ## Usage
 ### Basic usage
 ```bash
@@ -245,6 +324,12 @@ optional arguments:
                        (When this option is turned on, the prompt specified by -p will be used as the system prompt.)
 </pre>

+### Demo
+
+A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:
+
+https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1
+
 ### Benchmark
 We provide scripts to run the inference benchmark providing a model.

@@ -303,6 +388,10 @@ huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 --local-dir ./models/
 python ./utils/convert-helper-bitnet.py ./models/bitnet-b1.58-2B-4T-bf16
 ```

+## Acknowledgements
+
+This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp) framework. We would like to thank all the authors for their contributions to the open-source community. Also, bitnet.cpp's kernels are built on top of the Lookup Table methodologies pioneered in [T-MAC](https://github.com/microsoft/T-MAC/). For inference of general low-bit LLMs beyond ternary models, we recommend using T-MAC.
+
 ### FAQ (Frequently Asked Questions)📌 

 #### Q1: The build dies with errors building llama.cpp due to issues with std::chrono in log.cpp?
@@ -0,0 +1,536 @@
+# BitNet-Embeddings-0.6B/270M: I2_S Conversion and Inference Optimization Guide
+
+## 1. Model Overview
+
+BitNet-Embeddings is a family of multilingual text embedding models developed by Microsoft BitNet team.
+The models use decoder-only architecture with last-token pooling and L2 normalization to produce dense text embeddings.
+They can be applied to a wide range of tasks, including text retrieval, clustering, semantic similarity, classification, bitext mining, and reranking.
+They achieve competitive performance on public benchmarks while maintaining excellent inference and storage efficiency.
+
+- **Developed by:** BitNet Team, Microsoft Research
+- **Model type:** BitNet b1.58 based Text Embeddings
+- **Language(s):** Multilingual
+- **License:** MIT License
+
+### Model Sources
+
+- **Repository:** [https://github.com/microsoft/BitNet](https://github.com/microsoft/BitNet)
+- **Paper:** [The Era of 1-bit LLMs: BitNet b1.58 and its Inference Optimization](https://arxiv.org/abs/2402.17764)
+- **Paper:** [Multilingual E5 Text Embeddings: A Technical Report](https://arxiv.org/abs/2402.05672)
+
+| Model | Weights | Parameters | Embedding Dimension | Max Tokens | MTEB v2 Mean |
+|---|---|---|---|---|---|
+| [bitnet-embeddings-270m](https://huggingface.co/microsoft/bitnet-embedding-270m) | 1.58-bit | 270M | 640 | 32,768 | 66.26 |
+| [harrier-oss-v1-270m](https://huggingface.co/microsoft/harrier-oss-v1-270m) | bf16 | 270M | 640 | 32,768 | 66.5 |
+| [bitnet-embeddings-0.6b](https://huggingface.co/microsoft/bitnet-embedding-0.6b) | 1.58-bit | 0.6B | 1,024 | 32,768 | 67.49 |
+| [harrier-oss-v1-0.6b](https://huggingface.co/microsoft/harrier-oss-v1-0.6b) | bf16 | 0.6B | 1,024 | 32,768 | 69.0 |
+
+---
+
+## 2. Model Details
+
+- **Architecture**: Transformer-based, modified with BitLinear layers (BitNet framework).
+  - Uses Rotary Position Embeddings (RoPE).
+  - Employs SubLN (sub-layer normalization) for training stabilization under quantization.
+  - No bias terms in linear or normalization layers.
+- **Quantization**: Native 1.58-bit weights and 8-bit activations (W1.58A8).
+  - Weights are quantized to ternary values {-1, 0, +1} using absmean quantization.
+  - Activations are quantized to 8-bit integers using absmax quantization (per-token).
+  - Trained from scratch with this quantization scheme, not post-training quantized.
+- **Context Length**: 32,768 tokens.
+- **Pooling Strategy**: Last-token (EOS) pooling followed by L2 normalization.
+- **Training Pipeline**:
+  1. **BitNet Conversion**: Convert backbone into a BitNet-style encoder with ternary weights, quantized activations, and SubLN normalization.
+  2. **Continual Contrastive Pre-training**: Trained on 1B text pairs with InfoNCE loss.
+  3. **Distillation-based Supervised Fine-tuning**: Contrastive loss + similarity-distribution distillation + attention-relation distillation from FP16 teacher.
+
+| Model | [bitnet-embedding-0.6B](https://huggingface.co/microsoft/bitnet-embedding-0.6b) | [bitnet-embedding-270M](https://huggingface.co/microsoft/bitnet-embedding-270m) |
+|---|---|---|
+| Backbone | Qwen3-0.6B | Gemma3 |
+| Parameters | ~0.6B | ~270M |
+| Embedding Dimension | 1,024 | 640 |
+| Hidden Layers | 28 | 18 |
+| Attention Heads (KV) | 16 (8) | 4 (1) |
+| head_dim | 128 | 256 |
+| Intermediate Size | 3,072 | 2,048 |
+| Activation | SiLU | GELU |
+| Tokenizer | Qwen3 (151,936) | Gemma (262,144) |
+| Post-attn/FFW norms | No | Yes |
+| Embedding scaling | No | sqrt(hidden_size) |
+
+### MTEB v2 Evaluation Scores (16-bit embeddings)
+
+| Model | Weights | Bitext | Classification | Clustering | Pair Class. | Reranking | Retrieval | STS | **Mean** |
+|---|---|---|---|---|---|---|---|---|---|
+| bitnet-embeddings-270m | 1.58-bit | 80.47 | 71.09 | 52.37 | 79.72 | 60.50 | 66.71 | 74.35 | **66.26** |
+| bitnet-embeddings-0.6b | 1.58-bit | 81.47 | 72.65 | 53.06 | 80.47 | 62.12 | 68.33 | 74.97 | **67.49** |
+
+### Embedding Quantization
+
+The output embeddings can be quantized to 8, 4, 2, or even 1 bit, allowing users to flexibly trade off between storage cost and retrieval performance based on their application needs.
+
+![Embedding Quantization — Mean MTEB v2 Score](fig1_quant_per_task.png)
+
+### Training
+
+The models are trained with contrastive learning objectives on a large-scale mixture of multilingual datasets covering diverse tasks.
+Knowledge distillation from larger embedding models is used during training.
+The BitNet quantization is applied to all linear layers, resulting in 1.58-bit ternary weights while keeping activations in higher precision.
+
+### MMTEB (eng, v2) — BitNet 0.6B vs FP16 Teacher
+
+| Model | Cls. | Clust. | PairCls. | Rerank. | Retr. | STS | Summ. | Avg. | Speed (t/s) |
+|-------|------|--------|----------|--------|-------|-----|-------|------|-------------|
+| FP16 Teacher | 86.37 | 55.48 | 82.56 | 43.89 | 55.34 | 81.15 | 31.87 | 67.95 | 382.15 |
+| **BitNet Embedding 0.6B** | **86.49** | **55.42** | **82.30** | **43.41** | **54.03** | **81.15** | **32.06** | **67.60** | **870.90** |
+
+The model achieves **67.60** average score on MMTEB (eng, v2), only **0.35 points** below the FP16 teacher, while delivering **2.28x** higher CPU throughput.
+
+---
+
+## 3. I2_S GGUF Conversion
+
+### 3.1 Background
+
+BitNet embedding models apply per-projection RMSNorm (`BitLinear`) before each linear projection (q/k/v/o/gate/up/down). Each projection has a `.norm.weight` that applies RMSNorm to the input **before** the matmul:
+
+```
+x → RMSNorm(x, norm.weight) → activation_quant(8bit) → matmul(weight_quant(ternary))
+```
+
+This pattern does **not** exist in any standard llama.cpp architecture:
+- Standard Qwen3/Gemma3: no per-projection norms
+- Standard BitNet: has `attn_sub_norm`/`ffn_sub_norm` at different positions (after attention/gate*up, not before each projection)
+
+Currently two base architectures are supported (see [§2. Model Details](#2-model-details) for general architecture comparison). Key conversion-relevant parameters:
+
+| | bitnet-embeddings-0.6b (Qwen3) | bitnet-embeddings-270m (Gemma3) |
+|---|---|---|
+| Architecture (`model_type`) | `qwen3` | `gemma3_text` |
+| head_dim | 128 (note: != hidden_size/num_heads = 64) | 256 (note: != hidden_size/num_heads = 160) |
+| rope_theta | 1000000 | 10000.0 |
+| rms_norm_eps | 1e-06 | 1e-06 |
+| query_pre_attn_scalar | N/A | 256 |
+| tie_word_embeddings | true | true |
+
+#### Per-Layer Tensors (7 extra norm tensors per layer)
+
+| Tensor | Qwen3 Shape | Gemma3 Shape |
+|--------|-------------|--------------|
+| `self_attn.q_proj.norm.weight` | [1024] | [640] |
+| `self_attn.k_proj.norm.weight` | [1024] | [640] |
+| `self_attn.v_proj.norm.weight` | [1024] | [640] |
+| `self_attn.o_proj.norm.weight` | [2048] | [1024] |
+| `mlp.gate_proj.norm.weight` | [1024] | [640] |
+| `mlp.up_proj.norm.weight` | [1024] | [640] |
+| `mlp.down_proj.norm.weight` | [3072] | [2048] |
+
+
+### 3.2 GGUF Tensor Name Mapping
+
+#### Common Tensors (both architectures)
+
+| HF Name | GGUF Name | Notes |
+|----------|-----------|-------|
+| `embed_tokens.weight` | `token_embd.weight` | |
+| `norm.weight` | `output_norm.weight` | |
+| `layers.{i}.input_layernorm.weight` | `blk.{i}.attn_norm.weight` | |
+| `layers.{i}.self_attn.q_proj.weight` | `blk.{i}.attn_q.weight` | |
+| `layers.{i}.self_attn.k_proj.weight` | `blk.{i}.attn_k.weight` | |
+| `layers.{i}.self_attn.v_proj.weight` | `blk.{i}.attn_v.weight` | |
+| `layers.{i}.self_attn.o_proj.weight` | `blk.{i}.attn_output.weight` | |
+| `layers.{i}.self_attn.q_norm.weight` | `blk.{i}.attn_q_norm.weight` | QK head norm |
+| `layers.{i}.self_attn.k_norm.weight` | `blk.{i}.attn_k_norm.weight` | QK head norm |
+| `layers.{i}.self_attn.q_proj.norm.weight` | `blk.{i}.attn_q_norm_in.weight` | BitNet per-projection |
+| `layers.{i}.self_attn.k_proj.norm.weight` | `blk.{i}.attn_k_norm_in.weight` | BitNet per-projection |
+| `layers.{i}.self_attn.v_proj.norm.weight` | `blk.{i}.attn_v_norm_in.weight` | BitNet per-projection |
+| `layers.{i}.self_attn.o_proj.norm.weight` | `blk.{i}.attn_output_norm_in.weight` | BitNet per-projection |
+| `layers.{i}.mlp.gate_proj.weight` | `blk.{i}.ffn_gate.weight` | |
+| `layers.{i}.mlp.up_proj.weight` | `blk.{i}.ffn_up.weight` | |
+| `layers.{i}.mlp.down_proj.weight` | `blk.{i}.ffn_down.weight` | |
+| `layers.{i}.mlp.gate_proj.norm.weight` | `blk.{i}.ffn_gate_norm_in.weight` | BitNet per-projection |
+| `layers.{i}.mlp.up_proj.norm.weight` | `blk.{i}.ffn_up_norm_in.weight` | BitNet per-projection |
+| `layers.{i}.mlp.down_proj.norm.weight` | `blk.{i}.ffn_down_norm_in.weight` | BitNet per-projection |
+
+#### Architecture-Specific Tensors
+
+The two architectures differ in norm tensor naming, which affects the BF16→F16→GGUF mapping:
+
+- **Qwen3**: `post_attention_layernorm` maps directly to `ffn_norm`
+- **Gemma3**: `post_attention_layernorm` maps to `post_attention_norm` (different semantics), and has a separate `pre_feedforward_layernorm` → `ffn_norm`; also has `post_feedforward_layernorm` → `post_ffw_norm`
+
+Additional conversion differences:
+- **EOS token**: Qwen3 requires explicit override (`<|endoftext|>` id 151643); Gemma3 auto-detects from `tokenizer_config.json`
+- **Embedding scaling**: Gemma3 applies `sqrt(n_embd)` scaling (written as GGUF metadata)
+
+**Qwen3:**
+
+| HF Name | GGUF Name |
+|----------|-----------|
+| `layers.{i}.post_attention_layernorm.weight` | `blk.{i}.ffn_norm.weight` |
+
+**Gemma3:**
+
+| HF Name | GGUF Name |
+|----------|-----------|
+| `layers.{i}.post_attention_layernorm.weight` | `blk.{i}.post_attention_norm.weight` |
+| `layers.{i}.pre_feedforward_layernorm.weight` | `blk.{i}.ffn_norm.weight` |
+| `layers.{i}.post_feedforward_layernorm.weight` | `blk.{i}.post_ffw_norm.weight` |
+
+
+### 3.3 Conversion Script
+
+#### `utils/convert-bitnet-embedding-to-gguf.py`
+
+Unified standalone conversion script (safetensors → GGUF) that **auto-detects** the model architecture from `config.json`'s `model_type` field (`qwen3` or `gemma3_text`). Key features:
+
+- Hardcoded HF→GGUF tensor name mapping (no dependency on llama.cpp's Python converter)
+- Auto-detection of architecture and GGUF arch string (`qwen3` / `gemma3`)
+- Supports three output types:
+  - `--outtype f32`: all weights in float32
+  - `--outtype f16`: 2D weights and embeddings as float16, norms as float16
+  - `--outtype i2_s`: ternary weights packed in I2_S layout, non-ternary weights as float16
+- Writes `key_length` and `value_length` metadata for correct head_dim (critical: head_dim != hidden_size/num_heads for both models, default calculation would give wrong values)
+- BPE tokenizer handling with per-architecture pre-tokenizer hash verification:
+  - Qwen3: GPT-2 BPE tokenizer
+  - Gemma3: GemmaTokenizerFast (BPE)
+- Pooling type auto-detection from `modules.json` / `1_Pooling/config.json` (sentence-transformers convention)
+- Architecture-specific tokenizer handling:
+  - Qwen3: EOS token override (`<|endoftext|>` 151643) + `add_eos_token(True)` for last-token pooling
+  - Gemma3: EOS token auto-set by SpecialVocab from tokenizer_config.json (eos_token_id=1)
+- Gemma3: writes `query_pre_attn_scalar = 256` for correct attention scaling
+
+#### I2_S Ternary Packing
+
+The I2_S format packs ternary weights {-1, 0, +1} into 2-bit representation:
+
+- Quantization: `scale = 1/mean(|w|)`, `q = round(w * scale).clamp(-1, 1)`
+- Encoding: `-1 → 0`, `0 → 1`, `+1 → 2`
+- Every 128 values form a block, packed into 32 bytes
+- Each byte stores 4 values: `byte = (c0 << 6) | (c1 << 4) | (c2 << 2) | c3`
+- Scale (float32) is appended at the end of the packed data buffer
+
+#### Tensor Type Assignment
+
+| Tensor Type | f16 mode | i2_s mode |
+|-------------|----------|-----------|
+| 2D linear weights | float16 | I2_S ternary packed |
+| Embedding weights | float16 | float16 |
+| Norm weights (1D) | float16 | float16 |
+
+Note: `output.weight` (lm_head) is skipped for embedding models — it is not needed (no token generation).
+
+#### Example Usage
+
+```bash
+# I2_S conversion (requires BitNet natively-trained models with ternary weights)
+# Source: https://huggingface.co/microsoft/bitnet-embedding-0.6b
+# Output: ~699 MiB (~50% of F16 size for 0.6B)
+python3 utils/convert-bitnet-embedding-to-gguf.py \
+  /path/to/bitnet-embeddings-0.6b \
+  --outtype i2_s \
+  --outfile bitnet-embeddings-0.6b-i2_s.gguf
+
+# Source: https://huggingface.co/microsoft/bitnet-embedding-270m
+python3 utils/convert-bitnet-embedding-to-gguf.py \
+  /path/to/bitnet-embeddings-270m \
+  --outtype i2_s \
+  --outfile bitnet-embeddings-270m-i2_s.gguf
+
+# F16 conversion (for baseline comparison, does NOT require BitNet-trained models)
+# Can use standard FP16/BF16 teacher models directly
+# Output: ~1.11 GiB for 0.6B (595.78M params)
+python3 utils/convert-bitnet-embedding-to-gguf.py \
+  /path/to/multilingual-e5-0.6b-260311 \
+  --outtype f16 \
+  --outfile multilingual-e5-0.6b-f16.gguf
+
+python3 utils/convert-bitnet-embedding-to-gguf.py \
+  /path/to/multilingual-e5-270m-260311 \
+  --outtype f16 \
+  --outfile multilingual-e5-270m-f16.gguf
+```
+
+> **Note:** `multilingual-e5-*` is the **teacher/baseline model** with standard float weights, used as the F16 performance reference. `bitnet-embeddings-*` is the **1-bit quantized student model** with ternary weights, converted to I2_S for efficient CPU inference. Benchmarking compares both to measure the throughput gain and quality trade-off.
+
+#### Tensor Type Summary
+
+| Tensor | F16 (baseline) | I2_S (BitNet) |
+|--------|----------------|---------------|
+| Linear projections (q/k/v/o/gate/up/down) | float16 | I2_S (2-bit packed + float32 scale) |
+| Embedding (`token_embd.weight`) | float16 | float16 |
+| Per-projection norms (`*_norm_in`) | N/A (not present) | float16 |
+| Layer norms (attn_norm, ffn_norm, etc.) | float16 | float16 |
+| QK head norms (`attn_q_norm`, `attn_k_norm`) | float16 | float16 |
+| `output.weight` (lm_head) | skipped | skipped |
+
+### 3.4 Accuracy Verification
+
+After conversion, verify that the I2_S GGUF model maintains accuracy compared to the original safetensors and F16 GGUF baselines.
+
+#### Accuracy Test Script
+
+```bash
+#!/bin/bash
+set -e
+
+# Evaluate models on MTEB multilingual v2 benchmark
+# Compares: safetensors (GPU) vs F16 GGUF (CPU) vs I2_S GGUF (CPU)
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+SCRIPT="${SCRIPT_DIR}/eval_mmteb_v2.py"
+BUILD_DIR="/path/to/BitNet/build"
+MODEL_BASE="/path/to/models"
+OUTPUT_DIR="${SCRIPT_DIR}/eval_results"
+LOG_DIR="${OUTPUT_DIR}/log"
+
+mkdir -p "$OUTPUT_DIR" "$LOG_DIR"
+
+# Group 1: F16 baseline (multilingual-e5 teacher vs f16 GGUF)
+echo "Starting Group 1: multilingual-e5-0.6b (safetensors vs f16 GGUF)"
+nohup python "$SCRIPT" \
+    --model-dir "$MODEL_BASE/multilingual-e5-0.6b-260311" \
+    --f16-gguf "$MODEL_BASE/multilingual-e5-0.6b-260311/embeddings-0.6b-f16.gguf" \
+    --build-dir "$BUILD_DIR" \
+    --output-dir "$OUTPUT_DIR/multilingual-e5-0.6b" \
+    --model-name "multilingual-e5-0.6b" \
+    --model-type all \
+    --gpu 0 \
+    > "$LOG_DIR/eval_f16.log" 2>&1 &
+
+# Group 2: I2_S (bitnet-embeddings safetensors vs i2s GGUF)
+echo "Starting Group 2: bitnet-embeddings-0.6b (safetensors vs i2s GGUF)"
+nohup python "$SCRIPT" \
+    --model-dir "$MODEL_BASE/bitnet-embeddings-0.6b" \
+    --i2s-gguf "$MODEL_BASE/bitnet-embeddings-0.6b/bitnet-embeddings-0.6b-i2_s.gguf" \
+    --build-dir "$BUILD_DIR" \
+    --output-dir "$OUTPUT_DIR/bitnet-embeddings-0.6b" \
+    --model-name "bitnet-embeddings-0.6b" \
+    --model-type i2s \
+    --gpu 1 \
+    > "$LOG_DIR/eval_i2s.log" 2>&1 &
+
+echo "Both tasks running in background."
+```
+
+#### Accuracy Results
+
+**bitnet-embeddings-0.6B:**
+
+| Task | Safetensors | F16.gguf | I2_S.gguf |
+|------|-------------|----------|-----------|
+| BornholmBitextMining | 0.5727 | 0.5893 | 0.5610 |
+| FinancialPhrasebankClassification | 0.8792 | 0.8788 | 0.8781 |
+| KorHateSpeechMLClassification | 0.1027 | 0.1164 | 0.0987 |
+| KorSarcasmClassification | 0.7034 | 0.7016 | 0.7034 |
+| PoemSentimentClassification | 0.8321 | 0.8283 | 0.8269 |
+| SICK-R | 0.8218 | 0.8218 | 0.8216 |
+| STS17 | 0.8482 | 0.8482 | 0.8481 |
+| STSBenchmark | 0.8606 | 0.8606 | 0.8603 |
+| **AVERAGE** | **0.7188** | **0.7212** | **0.7180** |
+
+> I2_S.gguf achieves **0.7180** average, only **0.0008** below the original safetensors (0.7188) and **0.0032** below F16.gguf (0.7212) — negligible accuracy loss.
+
+**bitnet-embeddings-270M:**
+
+Note: F16.gguf is converted from `multilingual-e5-270m-260311`, the original bf16 model without BitNet training, serving as the baseline. The Safetensors and I2_S.gguf columns are from the same BitNet-trained model.
+
+| Task | Safetensors | F16.gguf | I2_S.gguf |
+|------|-------------|----------|-----------|
+| BornholmBitextMining | 0.6286 | 0.6637 | 0.6545 |
+| FinancialPhrasebankClassification | 0.8135 | 0.7180 | 0.7178 |
+| KorHateSpeechMLClassification | 0.6771 | 0.7790 | 0.7790 |
+| KorSarcasmClassification | 0.5579 | 0.5949 | 0.5871 |
+| PoemSentimentClassification | 0.0947 | 0.0873 | 0.0897 |
+| SICK-R | 0.8102 | 0.8108 | 0.8111 |
+| STS17 | 0.8568 | 0.8527 | 0.8519 |
+| STSBenchmark | 0.7998 | 0.7942 | 0.7947 |
+| **AVERAGE** | **0.7998** | **0.6626** | **0.6607** |
+
+> For 270M, Safetensors vs I2_S.gguf are from the same BitNet model — I2_S conversion preserves accuracy faithfully (**0.6607** vs Safetensors **0.7998** difference is due to different evaluation setup, not conversion loss). F16.gguf vs I2_S.gguf differ by only **0.0019**.
+
+
+---
+
+---
+
+## 4. Quick Start Example
+
+> **Note on build flags:** The build examples below use `-DGGML_NATIVE=ON`, which auto-detects and enables the best instruction set supported by the host CPU (e.g., AVX, AVX2, AVX-VNNI, FMA, F16C). This yields optimal performance. To target only AVX2 (e.g., for portable binaries), set `-DGGML_NATIVE=OFF` and manually specify:
+> ```
+> -DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON
+> -DGGML_AVX512=OFF -DGGML_AVX512_VBMI=OFF -DGGML_AVX512_VNNI=OFF -DGGML_AVX512_BF16=OFF
+> ```
+
+### Option 1: Using setup_env.py (recommended)
+
+```bash
+git clone --recursive https://github.com/microsoft/BitNet.git
+cd BitNet
+cd 3rdparty/llama.cpp && git checkout release-bitnet-embedding-0.6b-270m && cd ../..
+python setup_env.py -hr microsoft/bitnet-embedding-0.6b -md /path/to/save/model
+```
+
+### Option 2: Using CMake directly
+
+```bash
+git clone --recursive https://github.com/microsoft/BitNet.git
+cd BitNet
+cd 3rdparty/llama.cpp && git checkout release-bitnet-embedding-0.6b-270m && cd ../..
+cmake -S . -B build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_C_COMPILER=clang \
+  -DCMAKE_CXX_COMPILER=clang++ \
+  -DGGML_NATIVE=ON \
+  -DGGML_OPENMP=OFF \
+  -DLLAMA_BUILD_COMMON=ON \
+  -DLLAMA_BUILD_TOOLS=ON \
+  -DLLAMA_BUILD_EXAMPLES=ON
+cmake --build build --target llama-embedding llama-bench -j$(nproc)
+```
+
+### Run Inference
+
+```bash
+./build/bin/llama-embedding \
+  -m /path/to/save/model/bitnet-embedding-0.6b/ggml-model-i2_s.gguf \
+  -p "query: What is BitNet?" \
+  --embd-normalize 2 \
+  --embd-output-format array
+```
+
+**Example output** (1024-dimensional L2-normalized embedding, truncated):
+
+```json
+[[0.0239517, 0.6826404, -0.0000000, -0.0644535, 0.0613754, 0.0473094, 0.0114330, ...]]
+```
+
+---
+
+## 5. Inference Performance (CPU, 8 threads)
+
+Performance on **Intel Xeon Platinum 8573C** with 8 threads, Clang/Clang++ (no OpenMP), GGML_NATIVE=ON. All results in tokens/second (mean ± std over 3 runs).
+
+### Benchmark Script
+
+```bash
+#!/bin/bash
+# Benchmark: F16 vs I2_S
+set -e
+
+BENCH="./build/bin/llama-bench"
+THREADS=${1:-8}
+
+GGUF_F16="/path/to/models/multilingual-e5-0.6b/embeddings-0.6b-f16.gguf"
+GGUF_I2S="/path/to/models/bitnet-embeddings-0.6b/bitnet-embeddings-0.6b-i2_s.gguf"
+
+BENCH_ARGS="-t $THREADS -p 128,256,512,1024,2048,4096 -n 32,64 -r 3 -ngl 0"
+
+echo "========================================================"
+echo "  Benchmark: F16 vs I2_S"
+echo "  Threads: $THREADS"
+echo "========================================================"
+
+echo
+echo "--- F16 ---"
+$BENCH -m "$GGUF_F16" $BENCH_ARGS
+
+echo
+echo "--- I2_S ---"
+$BENCH -m "$GGUF_I2S" $BENCH_ARGS
+
+echo
+echo "Done."
+```
+
+### Results & Summary
+- **0.6B model**: I2_S achieves **1.42x–2.28x** speedup over F16, with the largest gain at short sequences (pp128). The speedup decreases at longer sequences due to the increasing dominance of attention computation (which is not quantized).
+- **270M model**: I2_S achieves **1.32x–1.74x** speedup over F16. The smaller speedup compared to 0.6B is expected — the 270M model has fewer linear projection parameters relative to other operations, so the benefit of ternary weight quantization is proportionally smaller.
+- **General trend**: Speedup is highest at short prompt lengths where matmul (weight-bound) dominates, and decreases at longer prompts where attention (compute-bound) takes over.
+
+#### bitnet-embedding-0.6B
+
+| Test | F16.gguf (t/s) | **I2_S.gguf (t/s)** | Speedup |
+|------|---------------|-----------------|---------|
+| pp128 | 382.15 | **870.90** | **2.28x** |
+| pp256 | 373.95 | **827.75** | **2.21x** |
+| pp512 | 371.86 | **716.27** | **1.93x** |
+| pp1024 | 341.55 | **620.58** | **1.82x** |
+| pp2048 | 298.21 | **481.14** | **1.61x** |
+| pp4096 | 236.76 | **336.32** | **1.42x** |
+
+#### bitnet-embedding-270m
+
+| Test | F16.gguf (t/s) | **I2_S.gguf (t/s)** | Speedup |
+|------|---------------|-----------------|---------|
+| pp128 | 1212.68 | **2019.59** | **1.67x** |
+| pp256 | 1221.28 | **2119.50** | **1.74x** |
+| pp512 | 1394.99 | **2181.23** | **1.56x** |
+| pp1024 | 1265.22 | **2086.46** | **1.65x** |
+| pp2048 | 1024.47 | **1471.60** | **1.44x** |
+| pp4096 | 785.54 | **1033.46** | **1.32x** |
+
+---
+
+## 6. FAQ
+
+**1. Do I need to add instructions to the query?**
+
+Yes, this is how the model is trained, otherwise you will see a performance degradation.
+The task definition should be a one-sentence instruction that describes the task.
+This is a way to customize text embeddings for different scenarios through natural language instructions.
+
+On the other hand, there is no need to add instructions to the document side.
+
+**2. Why are my reproduced results slightly different from reported in the model card?**
+
+Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences.
+
+**3. What pooling strategy does this model use?**
+
+The model uses **last-token pooling** — the embedding of the last non-padding token is used as the sentence representation.
+The embedding is then L2-normalized.
+
+---
+
+## 7. Uses and Limitations
+
+### Direct Use
+
+- Efficient information retrieval for RAG, web search, enterprise search, and question answering applications.
+- Text clustering, classification, and bitext mining based on dense text embeddings.
+
+### Out-of-Scope Use
+
+- BitNet-Embeddings does not generate any human-readable texts. It maps input texts into dense embedding vectors.
+- **Limited Training Data Representation:** Performance in low-resource languages may be significantly limited.
+- **Domain-Specific Limitations:** Specific or niche domains such as legal, medical, or scientific literature may not be adequately represented.
+- **Use in High-Risk Applications:** Not recommended for commercial or real-world applications without further testing and development.
+
+---
+
+## 8. Citation
+
+```bibtex
+@article{bitnet2024,
+  title={The Era of 1-bit LLMs: BitNet b1.58 and its Inference Optimization},
+  author={Ma, Shuming and Wang, Hongyu and others},
+  journal={arXiv preprint arXiv:2402.17764},
+  year={2024}
+}
+
+@inproceedings{wang2025bitnet,
+  title={BitNet.cpp: Efficient Edge Inference for Ternary LLMs},
+  author={Wang, Jinheng and Zhou, Hansong and Song, Ting and Cao, Shijie and Xia, Yan and Cao, Ting and Wei, Jianyu and Ma, Shuming and Wang, Hongyu and Wei, Furu},
+  booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
+  pages={9305--9322},
+  year={2025}
+}
+
+@article{wang2024multilingual,
+  title={Multilingual E5 Text Embeddings: A Technical Report},
+  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
+  journal={arXiv preprint arXiv:2402.05672},
+  year={2024}
+}
+```
@@ -73,7 +73,9 @@ It significantly improves GEMV throughput when processing quantized weights and

 ## Performance

-Kernel performance (tested on NVIDIA A100 40GB GPU):
+### Kernel Benchmarks
+
+Tested on NVIDIA A100 40GB GPU, our custom W2A8 kernel shows significant speedups over standard BF16 implementations:

 | Shape (N×K)         | W2A8 Latency (us) | BF16 Latency (us) | Speedup Ratio        |
 |---------------------|-------------------|-------------------|----------------------|
@@ -86,8 +88,20 @@ Kernel performance (tested on NVIDIA A100 40GB GPU):
 | 3200 × 10240        | 19.64             | 60.79             |   3.10               |
 | 20480 × 3200        | 30.99             | 112.39            |   3.63               |

-Generation throughput:
+### End-to-End Generation Latency

-| BF16 (tokens/s) | W2A8 (tokens/s) | Speedup Ratio |
-|---|---|---|
-| 10.9 | 213.3 | 19.6 |
+Compared to a similarly-sized BF16 model (Gemma-2-2B using vLLM), BitNet-b1.58-2B with our kernel achieves consistent speedups across workloads:
+
+| Input Length | Output Length | BF16 Latency (ms) | W2A8 Latency (ms) | Speedup Ratio |
+| --- | --- | --- | --- | --- |
+| 64 | 16 | 187.64 | 57.40 | 3.27 |
+| 64 | 32 | 353.50 | 112.22 | 3.15 |
+| 64 | 64 | 683.23 | 221.08 | 3.09 |
+| 256 | 16 | 183.14 | 61.24 | 2.99 |
+| 256 | 32 | 353.14 | 115.47 | 3.06 |
+| 256 | 64 | 684.24 | 224.16 | 3.05 |
+| 512 | 16 | 208.99 | 68.06 | 3.07 |
+| 512 | 32 | 354.33 | 122.72 | 2.89 |
+| 512 | 64 | 709.65 | 231.82 | 3.06 |
+
+*Note: Comparison uses equivalent-sized models (2B parameters) on NVIDIA A100 40GB GPU.*
@@ -34,7 +34,7 @@ def convert_ts_checkpoint(
    def convert_int8_to_int2(weight):
        return convert_weight_int8_to_int2(weight)

-    merged_result = torch.load(input_path, map_location="cpu", mmap=True)
+    merged_result = torch.load(input_path, map_location="cpu", mmap=True, weights_only=True)
    int2_result = {}
    fp16_result = {}
    zero = torch.zeros(1).to(torch.bfloat16)
@@ -64,9 +64,9 @@ class FastGen:
        decode_model = fast.Transformer(model_args_decode)

        fp16_ckpt_path = str(Path(ckpt_dir) / "model_state_fp16.pt")
-        fp16_checkpoint = torch.load(fp16_ckpt_path, map_location="cpu")
+        fp16_checkpoint = torch.load(fp16_ckpt_path, map_location="cpu", weights_only=True)
        int2_ckpt_path = str(Path(ckpt_dir) / "model_state_int2.pt")
-        int2_checkpoint = torch.load(int2_ckpt_path, map_location="cpu")
+        int2_checkpoint = torch.load(int2_ckpt_path, map_location="cpu", weights_only=True)
        prefill_model.load_state_dict(fp16_checkpoint, strict=True)
        decode_model.load_state_dict(int2_checkpoint, strict=True)

@@ -0,0 +1,35 @@
+#define ACT_PARALLEL
+#if defined(__AVX__) || defined(__AVX2__) || defined(__AVX512F__) || defined(__SSSE3__)
+#if defined(ACT_PARALLEL)
+    #define ROW_BLOCK_SIZE 4
+    #define COL_BLOCK_SIZE 128
+    #define PARALLEL_SIZE 4
+#else
+    #define ROW_BLOCK_SIZE 128
+    #define COL_BLOCK_SIZE 32
+    #define PARALLEL_SIZE 8
+#endif // ACT_PARALLEL
+#elif defined(__ARM_NEON)
+#if defined(__ARM_FEATURE_DOTPROD)
+#if defined(ACT_PARALLEL)
+    #define ROW_BLOCK_SIZE 8
+    #define COL_BLOCK_SIZE 256
+    #define PARALLEL_SIZE 8
+#else
+    #define ROW_BLOCK_SIZE 64
+    #define COL_BLOCK_SIZE 16
+    #define PARALLEL_SIZE 2
+#endif // ACT_PARALLEL
+#else
+#if defined(ACT_PARALLEL)
+    #define ROW_BLOCK_SIZE 8
+    #define COL_BLOCK_SIZE 256
+    #define PARALLEL_SIZE 4
+#else
+    #define ROW_BLOCK_SIZE 128
+    #define COL_BLOCK_SIZE 32
+    #define PARALLEL_SIZE 4
+#endif // ACT_PARALLEL
+#endif // __ARM_FEATURE_DOTPROD
+#endif // __AVX__
+
@@ -14,6 +14,8 @@ typedef float bitnet_float_type;
 extern "C" {
 #endif

+struct ggml_compute_params;
+
 struct bitnet_tensor_extra {
    int lut_scales_size;
    int BK;
@@ -33,6 +35,7 @@ GGML_API size_t ggml_bitnet_mul_mat_get_wsize(const struct ggml_tensor * src0, c
 GGML_API void ggml_bitnet_mul_mat_task_init(void * src1, void * qlut, void * lut_scales, void * lut_biases, int n, int k, int m, int bits);
 GGML_API void ggml_bitnet_mul_mat_task_compute(void * src0, void * scales, void * qlut, void * lut_scales, void * lut_biases, void * dst, int n, int k, int m, int bits);
 GGML_API void ggml_bitnet_transform_tensor(struct ggml_tensor * tensor);
+GGML_API void ggml_bitnet_mul_mat(const struct ggml_compute_params * params, struct ggml_tensor * dst);
 GGML_API int ggml_bitnet_get_type_bits(enum ggml_type type);
 GGML_API void ggml_bitnet_set_n_threads(int n_threads);
 #if defined(GGML_BITNET_ARM_TL1)
@@ -0,0 +1,21 @@
+[Kernels_0]
+m = 3200
+k = 8640
+bm = 160
+bk = 96
+bmm = 32
+
+[Kernels_1]
+m = 3200
+k = 3200
+bm = 320
+bk = 96
+bmm = 32
+
+[Kernels_2]
+m = 8640
+k = 3200
+bm = 320
+bk = 96
+bmm = 32
+
@@ -30,7 +30,6 @@ def run_inference():
        '-ngl', '0',
        '-c', str(args.ctx_size),
        '--temp', str(args.temperature),
-        "-b", "1",
    ]
    if args.conversation:
        command.append("-cnv")
@@ -56,6 +56,12 @@ SUPPORTED_HF_MODELS = {
    "tiiuae/Falcon-E-1B-Base": {
        "model_name": "Falcon-E-1B-Base",
    },
+    "microsoft/bitnet-embedding-0.6b": {
+        "model_name": "bitnet-embedding-0.6b",
+    },
+    "microsoft/bitnet-embedding-270m": {
+        "model_name": "bitnet-embedding-270m",
+    },
 }

 SUPPORTED_QUANT_TYPES = {
@@ -64,8 +70,8 @@ SUPPORTED_QUANT_TYPES = {
 }

 COMPILER_EXTRA_ARGS = {
-    "arm64": ["-DBITNET_ARM_TL1=ON"],
-    "x86_64": ["-DBITNET_X86_TL2=ON"]
+    "arm64": ["-DBITNET_ARM_TL1=OFF"],
+    "x86_64": ["-DBITNET_X86_TL2=OFF"]
 }

 OS_EXTRA_ARGS = {
@@ -211,7 +217,7 @@ def compile():
        logging.error(f"Arch {arch} is not supported yet")
        exit(0)
    logging.info("Compiling the code using CMake.")
-    run_command(["cmake", "-B", "build", *COMPILER_EXTRA_ARGS[arch], *OS_EXTRA_ARGS.get(platform.system(), []), "-DCMAKE_C_COMPILER=clang", "-DCMAKE_CXX_COMPILER=clang++"], log_step="generate_build_files")
+    run_command(["cmake", "-B", "build", *COMPILER_EXTRA_ARGS[arch], *OS_EXTRA_ARGS.get(platform.system(), []), "-DCMAKE_C_COMPILER=clang", "-DCMAKE_CXX_COMPILER=clang++", "-DLLAMA_BUILD_TOOLS=ON", "-DLLAMA_BUILD_EXAMPLES=ON", "-DLLAMA_BUILD_COMMON=ON", "-DLLAMA_BUILD_SERVER=ON"], log_step="generate_build_files")
    # run_command(["cmake", "--build", "build", "--target", "llama-cli", "--config", "Release"])
    run_command(["cmake", "--build", "build", "--config", "Release"], log_step="compile")

@@ -0,0 +1,205 @@
+# BitNet CPU Inference Optimization
+
+This update provides significant performance improvements for BitNet inference on CPU through paralleled kernel implementations, native I2_S GEMM/GEMV support, configurable tiling block size and embedding quantization.
+
+## Update
+
+- **Parallel Weight & Activation Computation**  
+  Implemented parallel processing of weights and activations in the W2A8 vet_dot kernel, achieving improved throughput on both x86 and ARM architectures.
+
+- **Native I2_S GEMM & GEMV Support**  
+  Integrated I2_S GEMM and GEMV operations into ggml library, making them fully compatible with the llama.cpp architecture. This enables seamless integration with existing inference pipelines.
+
+- **Configurable Tiling & Parallelism**  
+  Introduced configurable GEMM & GEMV block sizes and parallelism levels, allowing performance fine-tuning for different CPU architectures.
+
+- **Embedding Quantization**  
+  Added support for embedding layer quantization with Q6_K format, reducing memory footprint and improving inference speed while maintaining high accuracy.
+
+## Usage
+
+### Configuration Options
+
+The `include/gemm-config.h` file controls kernel behavior:
+
+```c
+#define ROW_BLOCK_SIZE 4
+#define COL_BLOCK_SIZE 128
+#define PARALLEL_SIZE 4
+```
+
+Modify these values based on your CPU cache size and architecture for optimal performance. Users can fine-tune performance on their machine through `include/gemm-config.h`.
+
+### Enabling Embedding Quantization
+
+To use embedding quantization for additional speedup:
+
+**Using setup_env.py:**
+```bash
+python setup_env.py --quant-embd
+```
+This automatically converts embeddings to Q6_K format.
+
+**Manual conversion:**
+```bash
+build/bin/llama-quantize --token-embedding-type Q6_K models/BitNet-b1.58-2B-4T/ggml-model-f32.gguf models/BitNet-b1.58-2B-4T/ggml-model-i2_s-embed-q6_k.gguf I2_S 1 1
+```
+
+## Optimizations
+
+### 1. Weight & Activation Parallelism
+
+The kernel implements two parallelization strategies:
+
+- **Weight Parallel:** Processes multiple weight rows/columns in a single kernel call, reducing kernel launch overhead.
+
+- **Activation Parallel:** Built on top of weight parallel, amortizes the I2_S weight unpacking cost across multiple activation elements.
+
+**Recommendation:** For I2_S quantization format, activation parallel is recommended due to the unpack operation benefits. The current kernel defaults to activation parallel.
+
+**Kernel Performance Comparison:**
+
+<div align="center">
+
+Test configuration: AMD EPYC 7V13 (x86), 1 threads, time in milliseconds (mean±std)
+
+| Matrix Size | No Parallel | Weight Parallel | Activation Parallel |
+|:---:|:---:|:---:|:---:|
+| [1, 2048] × [2048, 2048] | 0.075±0.012 | **0.058±0.007** | 0.076±0.011 |
+| [32, 2048] × [2048, 2048] | 2.400±0.041 | 1.599±0.020 | **1.202±0.018** |
+| [128, 2048] × [2048, 2048] | 10.820±0.039 | 6.458±0.168 | **5.805±0.039** |
+| [256, 2048] × [2048, 2048] | 21.669±0.080 | 12.739±0.183 | **11.882±0.040** |
+| [512, 2048] × [2048, 2048] | 43.257±0.083 | 25.680±0.335 | **23.342±0.082** |
+| [2048, 2048] × [2048, 2048] | 173.175±0.214 | 103.112±0.552 | **93.276±0.612** |
+| [128, 2048] × [2048, 8192] | 43.345±0.090 | 25.541±0.239 | **23.528±0.052** |
+| [128, 8192] × [8192, 2048] | 38.085±0.162 | 23.866±0.096 | **22.569±0.132** |
+
+</div>
+
+### 2. GEMM/GEMV Integration with llama.cpp
+
+Integrated I2_S quantization format into llama.cpp's compute graph:
+
+- **GEMV Operations:** Optimized matrix-vector multiplication for token generation.
+- **GEMM Operations:** Efficient matrix-matrix multiplication for prompt processing.
+- **Tiling Strategy:** Configurable block sizes for optimal cache utilization.
+
+### 3. Configuration Fine-tuning
+
+Fine-tuning kernel parameters for optimal performance on specific hardware:
+
+**Example Configuration (x86, AMD EPYC 7V13):**
+- Method: Activation Parallel
+- Threads: 8
+- Workload: 128 prompt tokens (pp128)
+
+**Fine-tuning Parameters:**
+- **Parallelism Degree:** [2, 4, 8]
+- **Row Block Size:** [2, 4, 8, 16, 32]
+- **Column Block Size:** [32, 64, 128, 256, 512, 1024]
+
+**Fine-tuning Results:**
+
+<div align="center">
+
+<img src="./assets/fine_tuning_result.png" alt="fine_tune_result" width="800"/>
+
+*Shows throughput (tokens/s) for various configurations.*
+
+</div>
+
+**Optimal Configuration:** Under this setup (x86, 8 threads, pp128), the best performance is achieved with parallelism degree = 4, row block size = 4, and column block size = 128.
+
+### 4. Embedding Quantization
+
+Evaluated multiple embedding quantization formats to balance memory usage, model quality, and inference speed:
+
+**Perplexity Comparison:**
+
+<div align="center">
+
+Test configuration: BitNet-b1.58-2B-4T, TG128
+
+| Embedding Type | Wikitext | PTB | LAMBADA | IMDB | AG NEWS |
+|:---:|:---:|:---:|:---:|:---:|:---:|
+| **F32** | 17.1090±0.1278 | 33.0858±0.4886 | 43.2850±0.6363 | 29.3016±0.2890 | 36.7686±0.3920 |
+| **F16** | 17.1090±0.1278 | 33.0858±0.4886 | 43.2850±0.6363 | 29.3016±0.2890 | 36.7686±0.3920 |
+| **Q8_0** | 17.1197±0.1280 | 33.1181±0.4893 | 43.2891±0.6364 | 29.3133±0.2892 | 36.7740±0.3920 |
+| **Q6_K** | 17.1487±0.1282 | 33.2203±0.4914 | 43.3046±0.6362 | 29.3491±0.2897 | 36.7972±0.3921 |
+| **Q5_0** | 17.2379±0.1288 | 33.2439±0.4907 | 43.4631±0.6379 | 29.5481±0.2920 | 36.8539±0.3924 |
+| **Q4_0** | 17.3529±0.1300 | 33.7754±0.5001 | 44.4552±0.6559 | 30.1044±0.2978 | 37.3985±0.3997 |
+| **Q3_K** | 17.6434±0.1320 | 34.3914±0.5089 | 45.4591±0.6735 | 30.8476±0.3069 | 39.5692±0.4259 |
+| **I2_S** | N/A | N/A | N/A | N/A | N/A |
+
+**N/A indicates model failure due to extreme quantization.*
+
+</div>
+
+**Inference Speed Comparison:**
+
+<div align="center">
+
+<img src="./assets/embedding_throughput.png" alt="embedding_throughput" width="800"/>
+
+*Token generation throughput (tg128) for different embedding quantization types.*
+
+</div>
+
+**Recommendation:** Based on comprehensive evaluation of memory footprint, perplexity preservation, and inference speed, **Q6_K** is selected as the optimal embedding quantization format.
+
+## Performance
+
+Comparison of optimized parallel kernels vs. original implementation:
+
+**Test Configuration:**
+- Model: BitNet-b1.58-2B-4T
+- Hardware: AMD EPYC 7V13
+- Threads: 1 / 2 / 4 / 8 / 12 / 16
+- Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
+- Method: Activation Parallel
+
+<div align="center">
+
+<img src="./assets/performance_comparison_amd_epyc.png" alt="performance_comparison_amd_epyc" width="800"/>
+
+</div>
+
+**Test Configuration:**
+- Model: BitNet-b1.58-2B-4T
+- Hardware: Intel i7-13800H
+- Threads: 1 / 2 / 4 / 6
+- Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
+- Method: Activation Parallel
+
+<div align="center">
+
+<img src="./assets/performance_comparison_i7-13800h.png" alt="performance_comparison_i7-13800h" width="800"/>
+
+</div>
+
+**Test Configuration:**
+- Model: BitNet-b1.58-2B-4T
+- Hardware: Cobalt 100
+- Threads: 1 / 2 / 4 / 8
+- Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
+- Method: Activation Parallel
+
+<div align="center">
+
+<img src="./assets/performance_comparison_cobalt100_dotprod.png" alt="performance_comparison_cobalt100_dotprod" width="800"/>
+
+</div>
+
+## Technical Details
+
+### Key Files Modified
+
+- `src/ggml-bitnet-mad.cpp`: Parallel kernel implementations
+- `3rdparty/llama.cpp/ggml/src/ggml.c`: GEMM/GEMV integration
+- `include/gemm-config.h`: Configuration file
+
+### Supported Architectures
+
+- ✅ x86-64 with AVX2
+- ✅ ARM with NEON
+- ✅ ARM with DOTPROD extension
@@ -5,23 +5,26 @@
 #include <stdio.h>
 #include <stdlib.h>

+#ifdef __x86_64__
+#include <immintrin.h>
+#endif
+
 #include "ggml-bitnet.h"
 #include "ggml-quants.h"
+#include "ggml-cpu-impl.h"
+
+#if defined(GGML_BITNET_ARM_TL1) || defined(GGML_BITNET_X86_TL2)
 #include "bitnet-lut-kernels.h"
+#endif

 #if defined(GGML_BITNET_ARM_TL1)

 void ggml_bitnet_init(void) {
-    // LOG(INFO) << "ggml_bitnet_init";
-
    if (initialized) {
        return;
    }
    initialized = true;

-    // if (wrapper == nullptr) {
-    //     wrapper = new BITNET::BITNETGeMMWrapper<bitnet_bitnet_float_type>();
-    // }
    if (bitnet_tensor_extras == nullptr) {
        bitnet_tensor_extras = new bitnet_tensor_extra[GGML_BITNET_MAX_NODES];
    }
@@ -29,26 +32,17 @@ void ggml_bitnet_init(void) {
 }

 void ggml_bitnet_free(void) {
-    // LOG(INFO) << "ggml_bitnet_free";
-
    if (!initialized) {
        return;
    }
    initialized = false;

-    // delete wrapper;
-    // wrapper = nullptr;
-    for (size_t i = 0; i < bitnet_tensor_extras_index; i++) {
-        // aligned_free(bitnet_tensor_extras[i].qweights);
-        // aligned_free(bitnet_tensor_extras[i].scales);
-    }
    delete[] bitnet_tensor_extras;
    bitnet_tensor_extras = nullptr;
 }

 static bool do_permutate(enum ggml_type type) {
    if (type == GGML_TYPE_TL1) {
-        // Add additional args to decide if permuted I2 or naive I2
        return false;
    } else {
        return true;
@@ -58,8 +52,7 @@ static bool do_permutate(enum ggml_type type) {
 bool ggml_bitnet_can_mul_mat(const struct ggml_tensor * src0, const struct ggml_tensor * src1, const struct ggml_tensor * dst) {
    if ((is_type_supported(src0->type)) &&
        src1->type == GGML_TYPE_F32 &&
-        dst->type == GGML_TYPE_F32 &&
-        src0->backend == GGML_BACKEND_TYPE_CPU) {
+        dst->type == GGML_TYPE_F32) {
        if (src1->ne[1] <= 1) {
            return true;
        }
@@ -72,10 +65,9 @@ size_t ggml_bitnet_mul_mat_get_wsize(const struct ggml_tensor * src0, const stru
    const size_t ne10 = src1->ne[0];
    const size_t ne11 = src1->ne[1];
    const int bits = ggml_bitnet_get_type_bits(src0->type);
-    
+
    size_t wsize = ne10 * ne11 * 15 * sizeof(int8_t) + 1 * ne11 * 2 * sizeof(bitnet_float_type);
    if (sizeof(bitnet_float_type) == 2) {
-        // Need fp32 to fp16 conversion
        wsize += std::max(ne10, ne01) * ne11 * sizeof(bitnet_float_type);
    }
    wsize = ((wsize - 1) / 64 + 1) * 64;
@@ -93,19 +85,61 @@ int ggml_bitnet_get_type_bits(enum ggml_type type) {
    }
 }

+void ggml_bitnet_mul_mat(const struct ggml_compute_params * params, struct ggml_tensor * dst) {
+    const struct ggml_tensor * src0 = dst->src[0];
+    const struct ggml_tensor * src1 = dst->src[1];
+
+    const size_t ne00 = src0->ne[0];
+    const size_t ne01 = src0->ne[1];
+    const size_t ne10 = src1->ne[0];
+    const size_t ne11 = src1->ne[1];
+
+    const int ith = params->ith;
+    const int nth = params->nth;
+    const int bits = ggml_bitnet_get_type_bits(src0->type);
+
+    struct bitnet_tensor_extra * extra = (struct bitnet_tensor_extra *)src0->extra;
+    GGML_ASSERT(extra != nullptr);
+
+    char * wdata = (char *)params->wdata;
+    const size_t wsize_per_thread = ggml_bitnet_mul_mat_get_wsize(src0, src1, dst);
+
+    int8_t  * qlut  = (int8_t  *)(wdata);
+    bitnet_float_type * lut_scales = (bitnet_float_type *)(qlut + ne10 * ne11 * 15);
+    bitnet_float_type * lut_biases = (bitnet_float_type *)(lut_scales + ne11);
+
+    if (ith == 0) {
+        ggml_bitnet_mul_mat_task_init(
+            (void *)((char *)src1->data),
+            (void *)qlut,
+            (void *)lut_scales,
+            (void *)lut_biases,
+            ne10, ne00, ne11, bits);
+    }
+
+    // barrier
+    if (nth > 1) {
+        ggml_barrier(params->threadpool);
+    }
+
+    ggml_bitnet_mul_mat_task_compute(
+        (void *)extra->qweights,
+        (void *)extra->scales,
+        (void *)qlut,
+        (void *)lut_scales,
+        (void *)lut_biases,
+        (void *)((char *)dst->data),
+        ne10, ne00, ne11, bits);
+}
+
 #endif
 #if defined(GGML_BITNET_X86_TL2)
 void ggml_bitnet_init(void) {
-    // LOG(INFO) << "ggml_bitnet_init";
-
    if (initialized) {
        return;
    }
    initialized = true;

-    // if (wrapper == nullptr) {
-    //     wrapper = new BITNET::BITNETGeMMWrapper<bitnet_bitnet_float_type>();
-    // }
    if (bitnet_tensor_extras == nullptr) {
        bitnet_tensor_extras = new bitnet_tensor_extra[GGML_BITNET_MAX_NODES];
    }
@@ -113,19 +147,11 @@ void ggml_bitnet_init(void) {
 }

 void ggml_bitnet_free(void) {
-    // LOG(INFO) << "ggml_bitnet_free";
-
    if (!initialized) {
        return;
    }
    initialized = false;

-    // delete wrapper;
-    // wrapper = nullptr;
-    for (size_t i = 0; i < bitnet_tensor_extras_index; i++) {
-        // aligned_free(bitnet_tensor_extras[i].qweights);
-        // aligned_free(bitnet_tensor_extras[i].scales);
-    }
    delete[] bitnet_tensor_extras;
    bitnet_tensor_extras = nullptr;
 }
@@ -133,8 +159,7 @@ void ggml_bitnet_free(void) {
 bool ggml_bitnet_can_mul_mat(const struct ggml_tensor * src0, const struct ggml_tensor * src1, const struct ggml_tensor * dst) {
    if ((is_type_supported(src0->type)) &&
        src1->type == GGML_TYPE_F32 &&
-        dst->type == GGML_TYPE_F32 &&
-        src0->backend == GGML_BACKEND_TYPE_CPU) {
+        dst->type == GGML_TYPE_F32) {
        return true;
    }
    return false;
@@ -144,10 +169,9 @@ size_t ggml_bitnet_mul_mat_get_wsize(const struct ggml_tensor * src0, const stru
    const size_t ne01 = src0->ne[1];
    const size_t ne10 = src1->ne[0];
    const size_t ne11 = src1->ne[1];
-    
+
    size_t wsize = ne10 * ne11 * 11 * sizeof(int8_t) + 2 * ne11 * 2 * sizeof(bitnet_float_type);
    if (sizeof(bitnet_float_type) == 2) {
-        // Need fp32 to fp16 conversion
        wsize += std::max(ne10, ne01) * ne11 * sizeof(bitnet_float_type);
    }
    wsize = ((wsize - 1) / 64 + 1) * 64;
@@ -164,4 +188,5 @@ int ggml_bitnet_get_type_bits(enum ggml_type type) {
            return 0;
    }
 }
-#endif
+
+#endif
@@ -0,0 +1,646 @@
+#!/usr/bin/env python3
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import os
+import sys
+from hashlib import sha256
+from pathlib import Path
+from typing import Any, Iterator
+
+import numpy as np
+import torch
+
+# Allow using the local gguf-py if present
+if "NO_LOCAL_GGUF" not in os.environ:
+    _local_gguf = Path(__file__).parent / "gguf-py"
+    if _local_gguf.exists():
+        sys.path.insert(1, str(_local_gguf))
+import gguf
+
+logger = logging.getLogger("convert-bitnet-embedding")
+
+# Supported architectures: model_type -> gguf arch name
+SUPPORTED_ARCHS = {
+    "qwen3":       "qwen3",
+    "gemma3_text": "gemma3",
+}
+
+# ---------------------------------------------------------------------------
+# Tensor name mapping: HuggingFace -> GGUF
+# ---------------------------------------------------------------------------
+
+def build_tensor_name_map(n_layers: int, arch: str) -> dict[str, str]:
+    """Build HF tensor name -> GGUF tensor name mapping."""
+    mapping: dict[str, str] = {
+        "embed_tokens.weight": "token_embd.weight",
+        "norm.weight": "output_norm.weight",
+    }
+
+    for i in range(n_layers):
+        pfx = f"layers.{i}"
+        blk = f"blk.{i}"
+
+        mapping.update({
+            # Layer norms
+            f"{pfx}.input_layernorm.weight":           f"{blk}.attn_norm.weight",
+
+            # Self-attention projections
+            f"{pfx}.self_attn.q_proj.weight":           f"{blk}.attn_q.weight",
+            f"{pfx}.self_attn.k_proj.weight":           f"{blk}.attn_k.weight",
+            f"{pfx}.self_attn.v_proj.weight":           f"{blk}.attn_v.weight",
+            f"{pfx}.self_attn.o_proj.weight":           f"{blk}.attn_output.weight",
+
+            # QK head norms
+            f"{pfx}.self_attn.q_norm.weight":           f"{blk}.attn_q_norm.weight",
+            f"{pfx}.self_attn.k_norm.weight":           f"{blk}.attn_k_norm.weight",
+
+            # Per-projection input norms (BitNet-specific)
+            f"{pfx}.self_attn.q_proj.norm.weight":      f"{blk}.attn_q_norm_in.weight",
+            f"{pfx}.self_attn.k_proj.norm.weight":      f"{blk}.attn_k_norm_in.weight",
+            f"{pfx}.self_attn.v_proj.norm.weight":      f"{blk}.attn_v_norm_in.weight",
+            f"{pfx}.self_attn.o_proj.norm.weight":      f"{blk}.attn_output_norm_in.weight",
+
+            # MLP projections
+            f"{pfx}.mlp.gate_proj.weight":              f"{blk}.ffn_gate.weight",
+            f"{pfx}.mlp.up_proj.weight":                f"{blk}.ffn_up.weight",
+            f"{pfx}.mlp.down_proj.weight":              f"{blk}.ffn_down.weight",
+
+            # Per-projection input norms for MLP (BitNet-specific)
+            f"{pfx}.mlp.gate_proj.norm.weight":         f"{blk}.ffn_gate_norm_in.weight",
+            f"{pfx}.mlp.up_proj.norm.weight":           f"{blk}.ffn_up_norm_in.weight",
+            f"{pfx}.mlp.down_proj.norm.weight":         f"{blk}.ffn_down_norm_in.weight",
+        })
+
+        if arch == "qwen3":
+            mapping[f"{pfx}.post_attention_layernorm.weight"] = f"{blk}.ffn_norm.weight"
+        elif arch == "gemma3_text":
+            mapping.update({
+                f"{pfx}.post_attention_layernorm.weight":   f"{blk}.post_attention_norm.weight",
+                f"{pfx}.pre_feedforward_layernorm.weight":  f"{blk}.ffn_norm.weight",
+                f"{pfx}.post_feedforward_layernorm.weight": f"{blk}.post_ffw_norm.weight",
+            })
+
+    return mapping
+
+
+# ---------------------------------------------------------------------------
+# Tokenizer handling
+# ---------------------------------------------------------------------------
+
+def get_vocab_base_pre(tokenizer, arch: str) -> str:
+    # encoding this string and hashing the resulting tokens would (hopefully) give us a unique identifier that
+    # is specific for the BPE pre-tokenizer used by the model
+    # we will use this unique identifier to write a "tokenizer.ggml.pre" entry in the GGUF file which we can
+    # use in llama.cpp to implement the same pre-tokenizer
+
+    chktxt = '\n \n\n \n\n\n \t \t\t \t\n  \n   \n    \n     \n\U0001f680 (normal) \U0001f636‍\U0001f32b️ (multiple emojis concatenated) ✅ \U0001f999\U0001f999 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច\U0001f601 ?我想在apple工作1314151天～ ------======= нещо на Български \'\'\'\'\'\'```````""""""......!!!!!!?????? I\'ve been \'told he\'s there, \'RE you sure? \'M not sure I\'ll make it, \'D you like some tea? We\'Ve a\'lL'
+
+    chktok = tokenizer.encode(chktxt)
+    chkhsh = sha256(str(chktok).encode()).hexdigest()
+
+    logger.debug(f"chktok: {chktok}")
+    logger.debug(f"chkhsh: {chkhsh}")
+
+    res = None
+
+    if arch == "qwen3":
+        # NOTE: if you get an error here, you need to update the convert_hf_to_gguf_update.py script
+        #       or pull the latest version of the model from Huggingface
+        #       don't edit the hashes manually!
+        if chkhsh == "0ef9807a4087ebef797fc749390439009c3b9eda9ad1a097abbe738f486c01e5":
+            # ref: https://huggingface.co/meta-llama/Meta-Llama-3-8B
+            res = "llama-bpe"
+        if chkhsh == "049ecf7629871e3041641907f3de7c733e4dbfdc736f57d882ba0b0845599754":
+            # ref: https://huggingface.co/deepseek-ai/deepseek-llm-7b-base
+            res = "deepseek-llm"
+        if chkhsh == "347715f544604f9118bb75ed199f68779f423cabb20db6de6f31b908d04d7821":
+            # ref: https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base
+            res = "deepseek-coder"
+        if chkhsh == "8aeee3860c56296a157a1fe2fad249ec40aa59b1bb5709f4ade11c4e6fe652ed":
+            # ref: https://huggingface.co/tiiuae/falcon-7b
+            res = "falcon"
+        if chkhsh == "3ce83efda5659b07b1ad37ca97ca5797ea4285d9b9ab0dc679e4a720c9da7454":
+            # ref: https://huggingface.co/openai-community/gpt2
+            res = "gpt-2"
+        if chkhsh == "d4540891389ea895b53b399da6ac824becc30f2fba0e9ddbb98f92e55ca0e97c":
+            # ref: https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
+            res = "qwen2"
+        if chkhsh == "855d9fb74bb0b28ce2305e9cd037ff6d8c798f18d19381ddfc14bea3dc9c002f":
+            # ref: multilingual-e5-0.6b-260311 (Qwen3 tokenizer variant)
+            res = "qwen2"
+    elif arch == "gemma3_text":
+        if chkhsh == "fcb6bf9f20f6c40fa4aa4f7f99607bd6c106ca2348efdacacdca8152e59dcfe9":
+            # ref: multilingual-e5-270m-260311 (Gemma3 tokenizer)
+            res = "default"
+        if chkhsh == "a8594e3edff7c29c003940395316294b2c623571571fc8d3d2d6571f5571cbe6":
+            # ref: google/gemma-2-9b
+            res = "default"
+
+    if res is None:
+        logger.warning("\n")
+        logger.warning("**************************************************************************************")
+        logger.warning("** WARNING: The BPE pre-tokenizer was not recognized!")
+        logger.warning("**          There are 2 possible reasons for this:")
+        logger.warning("**          - the model has not been added to convert_hf_to_gguf_update.py yet")
+        logger.warning("**          - the pre-tokenization config has changed upstream")
+        logger.warning("**          Check your model files and convert_hf_to_gguf_update.py and update them accordingly.")
+        logger.warning("** ref:     https://github.com/ggml-org/llama.cpp/pull/6920")
+        logger.warning("**")
+        logger.warning(f"** chkhsh:  {chkhsh}")
+        logger.warning("**************************************************************************************")
+        logger.warning("\n")
+        raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
+
+    logger.debug(f"tokenizer.ggml.pre: {repr(res)}")
+    logger.debug(f"chkhsh: {chkhsh}")
+
+    return res
+
+
+def _does_token_look_special(token: str) -> bool:
+    """Check if a token looks like a special token (e.g., <|...|>, <...>)."""
+    if not token:
+        return False
+    # Matches patterns like <|endoftext|>, <s>, </s>, [CLS], [SEP], etc.
+    if token.startswith(("<|", "<", "[")) and token.endswith(("|>", ">", "]")):
+        return True
+    return False
+
+
+def set_vocab(gguf_writer: gguf.GGUFWriter, dir_model: Path, hparams: dict, arch: str):
+    """Set tokenizer vocab.
+
+    - Qwen3: BPE tokenizer (tokenizer.ggml.model = "gpt2")
+    - Gemma3: SPM-compatible tokenizer from tokenizer.json (tokenizer.ggml.model = "llama")
+      Gemma uses SentencePiece-style tokenization with ▁ space prefix and byte fallback.
+      Using "llama" model type ensures llama.cpp uses the correct SPM pre-tokenizer
+      instead of the BPE regex-based pre-tokenizer which breaks CJK tokenization.
+    """
+    from transformers import AutoTokenizer
+    tokenizer = AutoTokenizer.from_pretrained(dir_model)
+    vocab_size = hparams.get("vocab_size", len(tokenizer.vocab))
+
+    if arch == "gemma3_text":
+        _set_vocab_gemma3(gguf_writer, dir_model, tokenizer, vocab_size)
+    else:
+        _set_vocab_bpe(gguf_writer, dir_model, tokenizer, vocab_size, arch)
+
+
+def _set_vocab_bpe(gguf_writer: gguf.GGUFWriter, dir_model: Path,
+                   tokenizer, vocab_size: int, arch: str):
+    """Set BPE vocab (for Qwen3)."""
+    tokpre = get_vocab_base_pre(tokenizer, arch)
+
+    tokens: list[str] = []
+    toktypes: list[int] = []
+
+    reverse_vocab = {id_: tok for tok, id_ in tokenizer.vocab.items()}
+    added_vocab = tokenizer.get_added_vocab()
+
+    added_tokens_decoder = tokenizer.added_tokens_decoder
+
+    for i in range(vocab_size):
+        if i not in reverse_vocab:
+            tokens.append(f"[PAD{i}]")
+            toktypes.append(gguf.TokenType.UNUSED)
+        elif reverse_vocab[i] in added_vocab:
+            token = reverse_vocab[i]
+
+            # Only encode-decode non-normalized tokens (matching llama.cpp upstream)
+            if not added_tokens_decoder[i].normalized:
+                token = tokenizer.decode(tokenizer.encode(token, add_special_tokens=False))
+
+            if added_tokens_decoder[i].special or _does_token_look_special(token):
+                toktypes.append(gguf.TokenType.CONTROL)
+            else:
+                token = token.replace(b"\xe2\x96\x81".decode("utf-8"), " ")
+                toktypes.append(gguf.TokenType.USER_DEFINED)
+
+            tokens.append(token)
+        else:
+            tokens.append(reverse_vocab[i])
+            toktypes.append(gguf.TokenType.NORMAL)
+
+    gguf_writer.add_tokenizer_model("gpt2")
+    gguf_writer.add_tokenizer_pre(tokpre)
+    gguf_writer.add_token_list(tokens)
+    gguf_writer.add_token_types(toktypes)
+
+    special_vocab = gguf.SpecialVocab(dir_model, load_merges=True)
+
+    if arch == "qwen3":
+        # Override EOS token: PyTorch tokenizer appends <|endoftext|> (151643) as the
+        # sentence-end marker, not <|im_end|> (151645). For last-token pooling to work
+        # correctly, llama.cpp must append the same token.
+        special_vocab.special_token_ids["eos"] = 151643
+
+    special_vocab.add_to_gguf(gguf_writer)
+
+    if arch == "qwen3":
+        # Embedding models need EOS token appended for last-token pooling
+        gguf_writer.add_add_eos_token(True)
+
+
+def _set_vocab_gemma3(gguf_writer: gguf.GGUFWriter, dir_model: Path,
+                      tokenizer, vocab_size: int):
+    """Set SPM-compatible vocab for Gemma3.
+
+    Gemma's tokenizer is SentencePiece-based (BPE variant with ▁ space prefix
+    and byte fallback). We read tokenizer.json to extract vocab and compute
+    BPE merge scores, then write as tokenizer.ggml.model = "llama" so llama.cpp
+    uses the SPM code path (correct pre-tokenizer behavior for CJK etc.).
+
+    Score assignment:
+      - BPE merge results get scores derived from merge rank (lower rank = higher score)
+      - Single-char / byte tokens get score 0
+      - Special / added tokens get score -1000
+    """
+    tokenizer_json_file = dir_model / "tokenizer.json"
+    if not tokenizer_json_file.exists():
+        raise FileNotFoundError(f"tokenizer.json not found in {dir_model}")
+
+    with open(tokenizer_json_file, encoding="utf-8") as f:
+        tokenizer_json = json.load(f)
+
+    bpe_vocab = tokenizer_json["model"]["vocab"]  # token_str -> token_id
+    bpe_merges = tokenizer_json["model"].get("merges", [])
+
+    # Build merge result -> rank mapping for score computation
+    # merge_scores[result_token] = -rank (lower rank = earlier merge = higher priority)
+    merge_scores: dict[str, float] = {}
+    for rank, merge in enumerate(bpe_merges):
+        if isinstance(merge, list):
+            result = "".join(merge)
+        else:
+            parts = merge.split(" ", 1)
+            result = "".join(parts)
+        if result not in merge_scores:
+            merge_scores[result] = -float(rank)
+
+    # Build token arrays
+    reverse_vocab = {v: k for k, v in bpe_vocab.items()}
+    added_tokens_decoder = tokenizer.added_tokens_decoder
+
+    tokens: list[bytes] = []
+    scores: list[float] = []
+    toktypes: list[int] = []
+
+    for i in range(vocab_size):
+        if i not in reverse_vocab:
+            tokens.append(f"[PAD{i}]".encode("utf-8"))
+            scores.append(-10000.0)
+            toktypes.append(gguf.TokenType.UNUSED)
+            continue
+
+        token_str = reverse_vocab[i]
+        token_bytes = token_str.encode("utf-8")
+
+        # Determine token type
+        if i in added_tokens_decoder:
+            tok_data = added_tokens_decoder[i]
+            if tok_data.special or _does_token_look_special(token_str):
+                toktypes.append(gguf.TokenType.CONTROL)
+            else:
+                toktypes.append(gguf.TokenType.USER_DEFINED)
+            scores.append(-1000.0)
+        elif token_str.startswith("<0x") and token_str.endswith(">") and len(token_str) == 6:
+            # Byte token: <0xHH>
+            toktypes.append(gguf.TokenType.BYTE)
+            scores.append(0.0)
+        elif token_str == "<unk>":
+            toktypes.append(gguf.TokenType.UNKNOWN)
+            scores.append(0.0)
+        else:
+            toktypes.append(gguf.TokenType.NORMAL)
+            # Score from merge rank, or 0 for single-char tokens
+            scores.append(merge_scores.get(token_str, 0.0))
+
+        tokens.append(token_bytes)
+
+    gguf_writer.add_tokenizer_model("llama")
+    gguf_writer.add_tokenizer_pre("default")
+    gguf_writer.add_token_list(tokens)
+    gguf_writer.add_token_scores(scores)
+    gguf_writer.add_token_types(toktypes)
+    gguf_writer.add_add_space_prefix(False)
+
+    special_vocab = gguf.SpecialVocab(dir_model, load_merges=False)
+    special_vocab.add_to_gguf(gguf_writer)
+
+
+# ---------------------------------------------------------------------------
+# GGUF metadata
+# ---------------------------------------------------------------------------
+
+def set_gguf_parameters(gguf_writer: gguf.GGUFWriter, hparams: dict, dir_model: Path, ftype: int):
+    gguf_writer.add_name(dir_model.name)
+
+    n_layers = hparams["num_hidden_layers"]
+    n_embd = hparams["hidden_size"]
+    n_head = hparams["num_attention_heads"]
+    n_head_kv = hparams.get("num_key_value_heads", n_head)
+    n_ff = hparams["intermediate_size"]
+
+    gguf_writer.add_block_count(n_layers)
+    gguf_writer.add_context_length(hparams.get("max_position_embeddings", 32768))
+    gguf_writer.add_embedding_length(n_embd)
+    gguf_writer.add_feed_forward_length(n_ff)
+    gguf_writer.add_head_count(n_head)
+    gguf_writer.add_head_count_kv(n_head_kv)
+    gguf_writer.add_vocab_size(hparams["vocab_size"])
+
+    head_dim = hparams.get("head_dim", n_embd // n_head)
+    gguf_writer.add_rope_dimension_count(head_dim)
+    gguf_writer.add_key_length(head_dim)
+    gguf_writer.add_value_length(head_dim)
+
+    if hparams.get("rope_theta") is not None:
+        gguf_writer.add_rope_freq_base(hparams["rope_theta"])
+    if hparams.get("rms_norm_eps") is not None:
+        gguf_writer.add_layer_norm_rms_eps(hparams["rms_norm_eps"])
+
+    gguf_writer.add_file_type(ftype)
+
+    # Pooling type for embedding models
+    # Try to read from modules.json / 1_Pooling/config.json (sentence-transformers convention)
+    pooling_type = None
+    module_path = dir_model / "modules.json"
+    if module_path.is_file():
+        with open(module_path, encoding="utf-8") as f:
+            modules = json.load(f)
+        for mod in modules:
+            if mod["type"].endswith("Pooling"):
+                pooling_path = dir_model / mod["path"] / "config.json"
+                if pooling_path.is_file():
+                    with open(pooling_path, encoding="utf-8") as f:
+                        pooling = json.load(f)
+                    if pooling.get("pooling_mode_mean_tokens"):
+                        pooling_type = gguf.PoolingType.MEAN
+                    elif pooling.get("pooling_mode_cls_token"):
+                        pooling_type = gguf.PoolingType.CLS
+                    elif pooling.get("pooling_mode_lasttoken"):
+                        pooling_type = gguf.PoolingType.LAST
+                break
+    if pooling_type is None:
+        # Default to MEAN pooling for embedding models
+        logger.info("  No pooling config found, defaulting to MEAN pooling")
+        pooling_type = gguf.PoolingType.MEAN
+    gguf_writer.add_pooling_type(pooling_type)
+
+    logger.info(f"  n_layers={n_layers}, n_embd={n_embd}, n_head={n_head}, n_head_kv={n_head_kv}, n_ff={n_ff}, head_dim={head_dim}")
+
+
+# ---------------------------------------------------------------------------
+# Tensor iteration from safetensors
+# ---------------------------------------------------------------------------
+
+def iter_tensors(dir_model: Path) -> Iterator[tuple[str, torch.Tensor]]:
+    """Yield (name, tensor) from safetensors files."""
+    from safetensors import safe_open
+
+    safetensor_files = sorted(dir_model.glob("*.safetensors"))
+    if not safetensor_files:
+        raise FileNotFoundError(f"No .safetensors files in {dir_model}")
+
+    for sf_path in safetensor_files:
+        logger.info(f"Loading {sf_path.name}")
+        with safe_open(str(sf_path), framework="pt", device="cpu") as f:
+            for name in f.keys():
+                yield name, f.get_tensor(name)
+
+
+# ---------------------------------------------------------------------------
+# I2_S ternary packing (platform-independent)
+# ---------------------------------------------------------------------------
+#
+# I2_S format (from dequantize_row_i2_s in ggml-quants.c):
+#   - Every 128 values form a block, packed into 32 bytes
+#   - Each byte stores 4 values at positions [0*32+gp, 1*32+gp, 2*32+gp, 3*32+gp]
+#     where gp is the byte index within the 32-byte group
+#   - Encoding per byte: c0=(b>>6)&3, c1=(b>>4)&3, c2=(b>>2)&3, c3=(b>>0)&3
+#   - Value mapping: 0 -> -1, 1 -> 0, 2 -> +1, 3 -> 0
+#   - Scale is stored as a separate tensor (tensor_name + "_scale")
+
+def quantize_to_i2_s(w: np.ndarray) -> np.ndarray:
+    """Quantize float weights to ternary and pack into I2_S layout.
+
+    Uses the same quantization as BitLinear weight_quant_minmax():
+        scale = 1.0 / mean(|w|)
+        q = round(w * scale).clamp(-1, 1)
+        dequant = q / scale = q * mean(|w|)
+
+    The I2_S format is self-contained: packed ternary bytes followed by a f32 scale
+    appended at the end of the data buffer.
+
+    Args:
+        w: float weight tensor of shape (M, K)
+
+    Returns:
+        packed_data: uint8 array containing I2_S packed bytes + scale (as 4 trailing bytes)
+    """
+    M, K = w.shape
+    n = M * K
+    w_flat = w.flatten().astype(np.float32)
+
+    # BitLinear weight_quant_minmax: scale = 1/mean(|w|), then round & clamp
+    abs_mean = np.mean(np.abs(w_flat))
+    abs_mean = max(abs_mean, 1e-5)
+    inv_scale = 1.0 / abs_mean
+    q_float = np.round(w_flat * inv_scale).clip(-1, 1)  # ternary: {-1, 0, 1}
+
+    # scale for dequantization = abs_mean (i.e., dequant = q * abs_mean)
+    scale = np.float32(abs_mean)
+
+    # Map ternary {-1, 0, 1} -> I2_S encoding {0, 1, 2}
+    #   -1 -> 0,  0 -> 1,  +1 -> 2
+    q = np.ones(n, dtype=np.uint8)  # default to 1 (zero)
+    q[q_float > 0.5] = 2    # +1 -> 2
+    q[q_float < -0.5] = 0   # -1 -> 0
+
+    # Pack into I2_S layout: 128-value blocks, interleaved into 32 bytes
+    # Pad to multiple of 128
+    pad_len = (128 - n % 128) % 128
+    if pad_len:
+        q = np.pad(q, (0, pad_len), constant_values=1)
+
+    n_padded = len(q)
+    n_blocks = n_padded // 128
+
+    q = q.reshape(n_blocks, 4, 32)
+
+    # Pack: byte = (c0 << 6) | (c1 << 4) | (c2 << 2) | c3
+    packed = (q[:, 0, :].astype(np.uint8) << 6) | \
+             (q[:, 1, :].astype(np.uint8) << 4) | \
+             (q[:, 2, :].astype(np.uint8) << 2) | \
+             (q[:, 3, :].astype(np.uint8))
+
+    packed = packed.reshape(-1).astype(np.uint8)
+
+    # I2_S format: packed_bytes + 32-byte aligned tail (scale in first 4 bytes of tail)
+    # Total size = n_elements / 4 + 32  (as defined in ggml.c)
+    packed_size = n // 4
+    total_size = packed_size + 32
+    result = np.zeros(total_size, dtype=np.uint8)
+    result[:len(packed)] = packed[:packed_size]
+    # Write scale as float32 at offset packed_size
+    result[packed_size:packed_size+4] = np.frombuffer(scale.tobytes(), dtype=np.uint8)
+
+    return result
+
+
+# ---------------------------------------------------------------------------
+# Main conversion
+# ---------------------------------------------------------------------------
+
+def main():
+    parser = argparse.ArgumentParser(description="Convert bitnet-embeddings (Qwen3/Gemma3) to GGUF")
+    parser.add_argument("model", type=Path, help="Model directory")
+    parser.add_argument("--outfile", type=Path, default=None, help="Output GGUF file")
+    parser.add_argument("--outtype", choices=["f32", "f16", "i2_s"], default="f16",
+                        help="Output type: f32, f16, or i2_s (ternary quantized)")
+    parser.add_argument("--verbose", action="store_true")
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO)
+
+    dir_model = args.model
+    if not dir_model.is_dir():
+        logger.error(f"{dir_model} is not a directory")
+        sys.exit(1)
+
+    # Default output filename
+    if args.outfile is None:
+        suffix = {"f32": "-f32", "f16": "-f16", "i2_s": "-f16-new-i2_s"}[args.outtype]
+        args.outfile = dir_model / f"{dir_model.name}{suffix}.gguf"
+
+    # Load config
+    with open(dir_model / "config.json") as f:
+        hparams = json.load(f)
+
+    arch = hparams.get("model_type", "")
+    if arch not in SUPPORTED_ARCHS:
+        logger.error(f"Unsupported model_type '{arch}'. Supported: {list(SUPPORTED_ARCHS.keys())}")
+        sys.exit(1)
+
+    gguf_arch = SUPPORTED_ARCHS[arch]
+    n_layers = hparams["num_hidden_layers"]
+
+    # Determine ftype
+    if args.outtype == "f32":
+        ftype = 0  # GGML F32
+    elif args.outtype == "f16":
+        ftype = 1  # GGML F16
+    else:  # i2_s
+        ftype = 40  # LLAMA_FTYPE_MOSTLY_I2_S
+
+    logger.info(f"Converting {dir_model.name} (arch={arch}) to GGUF ({args.outtype})")
+
+    # Create GGUF writer
+    gguf_writer = gguf.GGUFWriter(str(args.outfile), gguf_arch)
+
+    # Set parameters
+    set_gguf_parameters(gguf_writer, hparams, dir_model, ftype)
+
+    # Set vocab
+    logger.info("Setting tokenizer/vocab...")
+    set_vocab(gguf_writer, dir_model, hparams, arch)
+
+    # Build tensor name map
+    tensor_map = build_tensor_name_map(n_layers, arch)
+
+    # Process tensors
+    logger.info("Processing tensors...")
+    tensor_count = 0
+    for hf_name, data_torch in iter_tensors(dir_model):
+        # Skip tensors we don't need
+        if hf_name.endswith((".attention.masked_bias", ".attention.bias", ".rotary_emb.inv_freq")):
+            continue
+
+        # Strip "model." prefix if present
+        name = hf_name
+        if name.startswith("model."):
+            name = name[len("model."):]
+
+        # Look up GGUF name
+        gguf_name = tensor_map.get(name)
+        if gguf_name is None:
+            logger.warning(f"Skipping unmapped tensor: {hf_name}")
+            continue
+
+        old_dtype = data_torch.dtype
+
+        # Convert bf16 -> f32 first (bf16 not directly supported by gguf)
+        if data_torch.dtype == torch.bfloat16:
+            data_torch = data_torch.to(torch.float32)
+
+        data = data_torch.squeeze().numpy()
+        n_dims = len(data.shape)
+        data_shape = data.shape
+
+        # Determine if this is a linear weight suitable for ternary quantization
+        is_norm = gguf_name.endswith("_norm.weight") or gguf_name.endswith("_norm_in.weight")
+        is_embed = gguf_name == "token_embd.weight"
+        is_linear_weight = n_dims == 2 and not is_norm and not is_embed
+        suit_i2 = is_linear_weight
+
+        if args.outtype == "i2_s" and suit_i2:
+            # --- I2_S ternary packing (scale embedded in data) ---
+            packed = quantize_to_i2_s(data)
+            data_qtype = gguf.GGMLQuantizationType.I2_S
+
+            shape_str = f"{{{', '.join(str(n) for n in reversed(data_shape))}}}"
+            logger.info(f"  {gguf_name}: {list(data_shape)} {old_dtype} -> I2_S, shape = {shape_str}")
+
+            gguf_writer.add_tensor(gguf_name, packed, raw_shape=data_shape, raw_dtype=data_qtype)
+            tensor_count += 1
+
+        elif args.outtype in ("f16", "i2_s") and (is_linear_weight or is_embed):
+            # 2D weight tensors (linear + embedding) -> f16
+            data = data.astype(np.float16)
+            logger.info(f"  {gguf_name}: {list(data_torch.shape)} {old_dtype} -> float16")
+            gguf_writer.add_tensor(gguf_name, data)
+            tensor_count += 1
+
+        else:
+            # norms, 1D tensors
+            # Gemma3 RMSNorm uses (1+w)*x instead of w*x; preprocess w -> w+1
+            # so llama.cpp's standard RMSNorm produces correct results.
+            # NOTE: *_norm_in weights are BitLinear standard RMSNorm (initialized ~1.0),
+            # NOT Gemma3RMSNorm (initialized ~0.0), so they must NOT get +1.
+            is_gemma3_native_norm = (arch == "gemma3_text" and is_norm
+                                     and not gguf_name.endswith("_norm_in.weight"))
+            if is_gemma3_native_norm:
+                data = data.astype(np.float32) + 1.0
+                logger.info(f"    [Gemma3 norm offset] {gguf_name}: applied w = w + 1")
+
+            if args.outtype in ("f16", "i2_s"):
+                data = data.astype(np.float16)
+                logger.info(f"  {gguf_name}: {list(data_torch.shape)} {old_dtype} -> float16")
+            else:
+                if data.dtype != np.float32:
+                    data = data.astype(np.float32)
+                logger.info(f"  {gguf_name}: {list(data_torch.shape)} {old_dtype} -> float32")
+            gguf_writer.add_tensor(gguf_name, data)
+            tensor_count += 1
+
+    logger.info(f"Total tensors written: {tensor_count}")
+
+    # Note: output.weight (lm_head) is skipped for embedding models —
+    # it is not needed (no token generation) and saves ~297MB for this model.
+
+    # Write GGUF
+    logger.info(f"Writing to {args.outfile}...")
+    gguf_writer.write_header_to_file()
+    gguf_writer.write_kv_data_to_file()
+    gguf_writer.write_tensors_to_file()
+    gguf_writer.close()
+
+    logger.info("Done!")
+
+
+if __name__ == "__main__":
+    main()
@@ -109,12 +109,12 @@ def main():
            except OSError as e:
                print(f"Warning: Could not remove {preprocessed_output_file}: {e}")
        
-        if gguf_f32_output.exists():
-            print(f"Removing f32 GGUF: {gguf_f32_output}")
-            try:
-                gguf_f32_output.unlink()
-            except OSError as e:
-                print(f"Warning: Could not remove {gguf_f32_output}: {e}")
+        # if gguf_f32_output.exists():
+        #     print(f"Removing f32 GGUF: {gguf_f32_output}")
+        #     try:
+        #         gguf_f32_output.unlink()
+        #     except OSError as e:
+        #         print(f"Warning: Could not remove {gguf_f32_output}: {e}")
        
        if input_backup_file.exists():
            if not input_file.exists():
@@ -153,8 +153,12 @@ class Model(ABC):
            self.gguf_writer.add_expert_used_count(n_experts_used)
            logger.info(f"gguf: experts used count = {n_experts_used}")

-        self.gguf_writer.add_file_type(self.ftype)
-        logger.info(f"gguf: file type = {self.ftype}")
+        # Map ggml tensor type to llama ftype for general.file_type metadata
+        ftype_val = self.ftype
+        if self.ftype == gguf.GGMLQuantizationType.I2_S:
+            ftype_val = 40  # LLAMA_FTYPE_MOSTLY_I2_S (matches official model)
+        self.gguf_writer.add_file_type(ftype_val)
+        logger.info(f"gguf: file type = {ftype_val}")

    def write_tensors(self):
        block_count = self.hparams.get("n_layers", self.hparams.get("num_hidden_layers", self.hparams.get("n_layer")))
@@ -274,7 +278,7 @@ class Model(ABC):
            elif reverse_vocab[i] in added_vocab:
                # We need to manually encode and decode the added tokens in case special characters
                # used for `\n` / `\t` have been manually added in the added tokens
-                encoded_decoded_token = tokenizer.decode(tokenizer.encode(reverse_vocab[i]))
+                encoded_decoded_token = tokenizer.decode(tokenizer.encode(reverse_vocab[i], add_special_tokens=False))
                tokens.append(encoded_decoded_token)
                if tokenizer.added_tokens_decoder[i].special:
                    toktypes.append(gguf.TokenType.CONTROL)
@@ -659,6 +663,70 @@ def preprocess_weights_tl2(
                             weight.shape[0]), mode='constant', constant_values=0)
    return weight

+def quantize_to_i2_s(w: np.ndarray, override_scale: float = None) -> np.ndarray:
+    """Quantize a float weight matrix to I2_S ternary format.
+
+    I2_S format: packed ternary bytes (4 values per byte) + 32-byte tail with f32 scale.
+    Dequantization: y = scale * ternary, where ternary in {-1, 0, +1}.
+
+    Args:
+        w: float weight tensor of shape (M, K)
+        override_scale: if provided, use this as the I2_S scale instead of computing from data.
+                       For offline-quantized BitNet models, this should be the weight_scale value.
+    """
+    M, K = w.shape
+    n = M * K
+    w_flat = w.flatten().astype(np.float32)
+
+    # Compute scale for I2_S dequantization
+    if override_scale is not None:
+        # override_scale is weight_scale from offline-quantized models ≈ mean(|original_bf16_weights|)
+        # Use it directly as the I2_S scale
+        scale = np.float32(override_scale)
+        # Weights are already ternary {-1, 0, 1}, use directly
+        q_float = w_flat
+    else:
+        # w_flat comes from weight_quant: values are ±scale or 0
+        # Use the first nonzero absolute value as scale (matches C quantize_i2_s)
+        nonzero = np.abs(w_flat[np.abs(w_flat) > 1e-6])
+        if len(nonzero) > 0:
+            scale = np.float32(nonzero[0])
+        else:
+            scale = np.float32(1e-5)
+        # Quantize to ternary {-1, 0, 1}
+        inv_scale = 1.0 / scale
+        q_float = np.round(w_flat * inv_scale).clip(-1, 1)
+
+    # Map ternary {-1, 0, 1} -> I2_S encoding {0, 1, 2}
+    q = np.ones(n, dtype=np.uint8)  # default to 1 (zero)
+    q[q_float > 0.5] = 2    # +1 -> 2
+    q[q_float < -0.5] = 0   # -1 -> 0
+
+    # Pack into I2_S layout: 128-value blocks, interleaved into 32 bytes
+    pad_len = (128 - n % 128) % 128
+    if pad_len:
+        q = np.pad(q, (0, pad_len), constant_values=1)
+
+    n_padded = len(q)
+    n_blocks = n_padded // 128
+    q = q.reshape(n_blocks, 4, 32)
+
+    packed = (q[:, 0, :].astype(np.uint8) << 6) | \
+             (q[:, 1, :].astype(np.uint8) << 4) | \
+             (q[:, 2, :].astype(np.uint8) << 2) | \
+             (q[:, 3, :].astype(np.uint8))
+    packed = packed.reshape(-1).astype(np.uint8)
+
+    # I2_S format: packed_bytes + 32-byte aligned tail (scale in first 4 bytes)
+    packed_size = n // 4
+    total_size = packed_size + 32
+    result = np.zeros(total_size, dtype=np.uint8)
+    result[:len(packed)] = packed[:packed_size]
+    result[packed_size:packed_size+4] = np.frombuffer(scale.tobytes(), dtype=np.uint8)
+
+    return result
+
+
 def transform_to_tl1(x: np.ndarray):
    scale = np.max(np.abs(x))
    # res = np.round(x / scale + 2).astype(np.uint8)
@@ -813,10 +881,17 @@ class LlamaModel(Model):
                            data = data.astype(np.float16)
                        data_qtype = gguf.GGMLQuantizationType.F16

-                if data_qtype is None:  # by default, convert to float32
-                    if data_dtype != np.float32:
-                        data = data.astype(np.float32)
-                    data_qtype = gguf.GGMLQuantizationType.F32
+                if data_qtype is None:  # by default
+                    # For I2_S/TL models, keep non-quantized 2D weights (e.g. embed) as F16 instead of F32
+                    if self.ftype in (gguf.GGMLQuantizationType.I2_S, gguf.GGMLQuantizationType.TL1, gguf.GGMLQuantizationType.TL2) \
+                       and n_dims >= 2 and not new_name.endswith("_norm.weight"):
+                        if data_dtype != np.float16:
+                            data = data.astype(np.float16)
+                        data_qtype = gguf.GGMLQuantizationType.F16
+                    else:
+                        if data_dtype != np.float32:
+                            data = data.astype(np.float32)
+                        data_qtype = gguf.GGMLQuantizationType.F32

                shape = data_shape
                # shape = gguf.quant_shape_from_byte_shape(data.shape, data_qtype) if data.dtype == np.uint8 else data.shape
@@ -952,20 +1027,30 @@ class LlamaModel(Model):
                raise ValueError(f"Unprocessed experts: {experts}")


-@Model.register("BitnetForCausalLM")
+@Model.register("BitnetForCausalLM", "BitNetForCausalLM")
 class BitnetModel(Model):
-    model_arch = gguf.MODEL_ARCH.BITNET
+    model_arch = gguf.MODEL_ARCH.BITNET_B158

    def set_vocab(self):
-        self._set_vocab_sentencepiece()
+        try:
+            self._set_vocab_sentencepiece()
+        except FileNotFoundError:
+            try:
+                self._set_vocab_llama_hf()
+            except (FileNotFoundError, TypeError):
+                self._set_vocab_gpt2()
        
    def set_gguf_parameters(self):
        super().set_gguf_parameters()

        self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])

-        self.gguf_writer.add_rope_scaling_type(gguf.RopeScalingType.LINEAR)
-        self.gguf_writer.add_rope_scaling_factor(1.0)
+        # rope dimension count (required for correct positional encoding)
+        if "head_dim" in self.hparams:
+            rope_dim = self.hparams["head_dim"]
+        else:
+            rope_dim = self.hparams["hidden_size"] // self.hparams["num_attention_heads"]
+        self.gguf_writer.add_rope_dimension_count(rope_dim)

    def weight_quant(self, weight):
        dtype = weight.dtype
@@ -976,7 +1061,10 @@ class BitnetModel(Model):

    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
        # quant weight to i2 (in fp16)
-        if name.endswith(("q_proj.weight", "k_proj.weight", "v_proj.weight", 
+        # Skip weight_quant for offline-quantized models — weights are already ternary {-1,0,1}
+        # weight_quant would scale them to small floats (e.g. ±0.39), breaking quantize_to_i2_s
+        if not getattr(self, '_has_offline_quant', False) and \
+           name.endswith(("q_proj.weight", "k_proj.weight", "v_proj.weight",
                          "down_proj.weight", "up_proj.weight", "gate_proj.weight",
                          "o_proj.weight")):
            data_torch = self.weight_quant(data_torch)
@@ -986,15 +1074,40 @@ class BitnetModel(Model):
    def write_tensors(self):
        max_name_len = max(len(s) for _, s in self.tensor_map.mapping.values()) + len(".weight,")

+        # First pass: collect weight_scale tensors for offline-quantized models
+        scale_map = dict()
        for name, data_torch in self.get_tensors():
+            if name.endswith("weight_scale"):
+                data_torch = data_torch.to(torch.float32)
+                name = name.replace(".weight_scale", "")
+                scale_map[name] = data_torch
+
+        self._has_offline_quant = len(scale_map) > 0
+
+        for name, data_torch in self.get_tensors():
+            # skip weight_scale tensors
+            if name.endswith("weight_scale"):
+                continue
            # we don't need these
            if name.endswith((".attention.masked_bias", ".attention.bias", ".rotary_emb.inv_freq")):
                continue

            old_dtype = data_torch.dtype

+            # Handle offline-quantized weights (uint8 packed with weight_scale)
+            if name.replace(".weight", "") in scale_map:
+                data_torch = data_torch.to(torch.uint8)
+                origin_shape = data_torch.shape
+                shift = torch.tensor([0, 2, 4, 6], dtype=torch.uint8).reshape((4, *(1 for _ in range(len(origin_shape)))))
+                data_torch = data_torch.unsqueeze(0).expand((4, *origin_shape)) >> shift
+                data_torch = data_torch & 3
+                data_torch = (data_torch.float() - 1).reshape((origin_shape[0] * 4, *origin_shape[1:]))
+                # For F16/F32 output: divide by weight_scale to get full float values
+                # For I2_S output: keep as ternary {-1,0,1}, scale is passed separately to quantize_to_i2_s
+                if self.ftype not in (gguf.GGMLQuantizationType.I2_S, gguf.GGMLQuantizationType.TL1, gguf.GGMLQuantizationType.TL2):
+                    data_torch = data_torch / scale_map[name.replace(".weight", "")].float()
            # convert any unsupported data types to float32
-            if data_torch.dtype not in (torch.float16, torch.float32):
+            elif data_torch.dtype not in (torch.float16, torch.float32):
                data_torch = data_torch.to(torch.float32)

            # use the first number-like part of the tensor name as the block id
@@ -1048,7 +1161,13 @@ class BitnetModel(Model):

                i2_scale = None
                if self.ftype != gguf.GGMLQuantizationType.F32 and extra_f16 and not extra_f32:
-                    if self.ftype == gguf.GGMLQuantizationType.TL1 and suit_i2:
+                    if self.ftype == gguf.GGMLQuantizationType.I2_S and suit_i2:
+                        data_qtype = gguf.GGMLQuantizationType.I2_S
+                        # Use original weight_scale if available (offline-quantized models)
+                        orig_scale = scale_map.get(name.replace(".weight", ""))
+                        override_scale = orig_scale.item() if orig_scale is not None else None
+                        data = quantize_to_i2_s(data, override_scale=override_scale)
+                    elif self.ftype == gguf.GGMLQuantizationType.TL1 and suit_i2:
                        data, i2_scale = transform_to_tl1(data)
                        assert data.dtype == np.uint8
                        assert i2_scale.dtype == np.float32
@@ -1063,10 +1182,17 @@ class BitnetModel(Model):
                            data = data.astype(np.float16)
                        data_qtype = gguf.GGMLQuantizationType.F16

-                if data_qtype is None:  # by default, convert to float32
-                    if data_dtype != np.float32:
-                        data = data.astype(np.float32)
-                    data_qtype = gguf.GGMLQuantizationType.F32
+                if data_qtype is None:  # by default
+                    # For I2_S/TL models, keep non-quantized 2D weights (e.g. embed) as F16 instead of F32
+                    if self.ftype in (gguf.GGMLQuantizationType.I2_S, gguf.GGMLQuantizationType.TL1, gguf.GGMLQuantizationType.TL2) \
+                       and n_dims >= 2 and not new_name.endswith("_norm.weight"):
+                        if data_dtype != np.float16:
+                            data = data.astype(np.float16)
+                        data_qtype = gguf.GGMLQuantizationType.F16
+                    else:
+                        if data_dtype != np.float32:
+                            data = data.astype(np.float32)
+                        data_qtype = gguf.GGMLQuantizationType.F32

                shape = data_shape
                # shape = gguf.quant_shape_from_byte_shape(data.shape, data_qtype) if data.dtype == np.uint8 else data.shape
@@ -1087,6 +1213,7 @@ class BitnetModel(Model):
 ftype_map = {
    "f32": gguf.GGMLQuantizationType.F32,
    "f16": gguf.GGMLQuantizationType.F16,
+    "i2_s": gguf.GGMLQuantizationType.I2_S,
    "tl1" : gguf.GGMLQuantizationType.TL1,
    "tl2" : gguf.GGMLQuantizationType.TL2,
 }
@@ -12,14 +12,12 @@ import json
 import math
 import mmap
 import os
-import pickle
 import re
 import signal
 import struct
 import sys
 import textwrap
 import time
-import zipfile
 from abc import ABC, abstractmethod
 from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
 from dataclasses import dataclass
@@ -945,7 +943,6 @@ def part_lazy(lazy_tensor: LazyTensor, n_part: int) -> LazyTensor:

 import torch

-@torch.compile
 def forward_t(x):
    dtype = x.dtype
    x = x.float()
@@ -956,7 +953,8 @@ def forward_t(x):
 def weight_quant(weight):
    weight = torch.tensor(weight, dtype=torch.float32)
    weight = forward_t(weight)
-    weight = weight.numpy().astype(np.float32)
+    # Use tolist() then convert to numpy to avoid PyTorch-NumPy compatibility issues
+    weight = np.array(weight.tolist(), dtype=np.float32)
    return weight

 def part_lazy_q(lazy_tensor: LazyTensor, n_part: int) -> LazyTensor:
@@ -1028,98 +1026,6 @@ def pack_experts_lazy(lazy_tensors: list[LazyTensor]) -> LazyTensor:
    return LazyTensor(load, s, lazy_tensors[0].data_type, 'pack_experts ' + ' | '.join(lt.description for lt in lazy_tensors))


-# Functionality that simulates `torch.load` but where individual tensors are
-# only loaded into memory on demand, not all at once.
-# PyTorch can't do this natively as of time of writing:
-# - https://github.com/pytorch/pytorch/issues/64327
-# This allows us to de-shard without multiplying RAM usage, and also
-# conveniently drops the PyTorch dependency (though we still need numpy).
-
-
-@dataclass
-class LazyStorageKind:
-    data_type: DataType
-
-
-@dataclass
-class LazyStorage:
-    load: Callable[[int, int], NDArray]
-    kind: LazyStorageKind
-    description: str
-
-
-class LazyUnpickler(pickle.Unpickler):
-    def __init__(self, fp: IO[bytes], data_base_path: str, zip_file: zipfile.ZipFile):
-        super().__init__(fp)
-        self.data_base_path = data_base_path
-        self.zip_file = zip_file
-
-    def persistent_load(self, pid: Any) -> Any:
-        assert pid[0] == 'storage'
-        assert isinstance(pid[1], LazyStorageKind)
-        data_type = pid[1].data_type
-        filename_stem = pid[2]
-        filename = f'{self.data_base_path}/{filename_stem}'
-        info = self.zip_file.getinfo(filename)
-
-        def load(offset: int, elm_count: int) -> NDArray:
-            dtype = data_type.dtype
-            with self.zip_file.open(info) as fp:
-                fp.seek(offset * dtype.itemsize)
-                size = elm_count * dtype.itemsize
-                data = fp.read(size)
-            assert len(data) == size
-            return np.frombuffer(data, dtype)
-        description = f'storage data_type={data_type} path-in-zip={filename} path={self.zip_file.filename}'
-        return LazyStorage(load=load, kind=pid[1], description=description)
-
-    @staticmethod
-    def lazy_rebuild_tensor_v2(storage: Any, storage_offset: Any, size: Any, stride: Any,
-                               requires_grad: Any, backward_hooks: Any, metadata: Any = None) -> LazyTensor:
-        assert isinstance(storage, LazyStorage)
-
-        def load() -> UnquantizedTensor:
-            elm_count = stride[0] * size[0]
-            return UnquantizedTensor(storage.load(storage_offset, elm_count).reshape(size))
-        description = f'pickled storage_offset={storage_offset} in {storage.description}'
-        return LazyTensor(load, list(size), storage.kind.data_type, description)
-
-    @staticmethod
-    def rebuild_from_type_v2(func, new_type, args, state):
-        return func(*args)
-
-    CLASSES = {
-        # getattr used here as a workaround for mypy not being smart enough to determine
-        # the staticmethods have a __func__ attribute.
-        ('torch._tensor', '_rebuild_from_type_v2'): getattr(rebuild_from_type_v2, '__func__'),
-        ('torch._utils', '_rebuild_tensor_v2'): getattr(lazy_rebuild_tensor_v2, '__func__'),
-        ('torch', 'BFloat16Storage'): LazyStorageKind(DT_BF16),
-        ('torch', 'HalfStorage'): LazyStorageKind(DT_F16),
-        ('torch', 'FloatStorage'): LazyStorageKind(DT_F32),
-        ('torch', 'IntStorage'): LazyStorageKind(DT_I32),
-        ('torch', 'Tensor'): LazyTensor,
-    }
-
-    def find_class(self, module: str, name: str) -> Any:
-        if not module.startswith('torch'):
-            return super().find_class(module, name)
-        return self.CLASSES[(module, name)]
-
-
-def lazy_load_torch_file(outer_fp: IO[bytes], path: Path) -> ModelPlus:
-    zf = zipfile.ZipFile(outer_fp)
-    pickle_paths = [name for name in zf.namelist() if name.endswith('.pkl')]
-    assert len(pickle_paths) == 1, pickle_paths
-    pickle_fp = zf.open(pickle_paths[0], 'r')
-    unpickler = LazyUnpickler(pickle_fp,
-                              data_base_path=pickle_paths[0][:-4],
-                              zip_file=zf)
-    model = unpickler.load()
-    if 'model' in model: model = model['model']
-    as_dict = dict(model.items())
-    return ModelPlus(model=as_dict, paths=[path], format='torch', vocab=None)
-
-
 def lazy_load_safetensors_file(fp: IO[bytes], path: Path) -> ModelPlus:
    header_size, = struct.unpack('<Q', fp.read(8))
    header: dict[str, dict[str, Any]] = json.loads(fp.read(header_size))
@@ -1156,14 +1062,11 @@ def lazy_load_file(path: Path) -> ModelPlus:
    fp = open(path, 'rb')
    first8 = fp.read(8)
    fp.seek(0)
-    if first8[:2] == b'PK':
-        # A zip file, i.e. PyTorch format
-        return lazy_load_torch_file(fp, path)
-    elif struct.unpack('<Q', first8)[0] < 16 * 1024 * 1024:
-        # Probably safetensors
+    if struct.unpack('<Q', first8)[0] < 16 * 1024 * 1024:
+        # Safetensors format
        return lazy_load_safetensors_file(fp, path)
    else:
-        raise ValueError(f"unknown format: {path}")
+        raise ValueError(f"unknown format: {path}. Only safetensors format is supported.")


 In = TypeVar('In')
@@ -1491,7 +1394,8 @@ def convert_model_names(model: LazyModel, params: Params, skip_unknown: bool) ->
    #             tmp[f"layers.{i_l}.feed_forward.experts.w{w}.weight"] = pack_experts_lazy(experts)
    # tmp[f"rope.freqs"] = part_lazy_rope(1.0 / (torch.tensor(500000) ** (torch.arange(0, 128, 2).float().to("cpu") / 128)))
    # 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
-    rope_ndarray = (1.0 / (torch.tensor(500000.0) ** (torch.arange(0, 128, 2).float() / 128))).numpy().astype(np.float32)
+    # Use pure NumPy instead of torch to avoid NumPy compatibility issues
+    rope_ndarray = (1.0 / (np.float32(500000.0) ** (np.arange(0, 128, 2, dtype=np.float32) / 128))).astype(np.float32)
    # print(rope_ndarray)
    

@@ -1583,7 +1487,7 @@ def convert_model_names(model: LazyModel, params: Params, skip_unknown: bool) ->

    out: LazyModel = {}
    for name, lazy_tensor in model.items():
-        tensor_type, name_new = tmap.get_type_and_name(name, try_suffixes = (".weight", ".bias")) or (None, None)
+        tensor_type, name_new = tmap.get_type_and_name(name, try_suffixes = (".weight", ".bias", ".weight_scale")) or (None, None)
        if name_new is None:
            if skip_unknown:
                logger.info(f"Unexpected tensor name: {name} - skipping")
@@ -1644,15 +1548,11 @@ def load_some_model(path: Path) -> ModelPlus:
    '''Load a model of any supported format.'''
    # Be extra-friendly and accept either a file or a directory:
    if path.is_dir():
-        # Check if it's a set of safetensors files first
-        globs = ["model-00001-of-*.safetensors", "model.safetensors", "consolidated.safetensors", "model-int2.pth"]
+        # Check if it's a set of safetensors files
+        globs = ["model-00001-of-*.safetensors", "model.safetensors", "consolidated.safetensors"]
        files = [file for glob in globs for file in path.glob(glob)]
        if not files:
-            # Try the PyTorch patterns too, with lower priority
-            globs = ["consolidated.00.pth", "pytorch_model-00001-of-*.bin", "*.pt", "pytorch_model.bin"]
-            files = [file for glob in globs for file in path.glob(glob)]
-        if not files:
-            raise FileNotFoundError(f"Can't find model in directory {path}")
+            raise FileNotFoundError(f"Can't find safetensors model in directory {path}")
        if len(files) > 1:
            raise ValueError(f"Found multiple models in {path}, not sure which to pick: {files}")
        path = files[0]
@@ -1744,7 +1644,7 @@ def do_dump_model(model_plus: ModelPlus) -> None:

 def main(args_in: list[str] | None = None) -> None:
    output_choices = ["f32", "f16", "i2"]
-    if np.uint32(1) == np.uint32(1).newbyteorder("<"):
+    if sys.byteorder == "little":
        # We currently only support Q8_0 output on little endian systems.
        output_choices.append("q8_0")
    parser = argparse.ArgumentParser(description="Convert a LLaMA model to a GGML compatible file")
@@ -1852,4 +1752,4 @@ def main(args_in: list[str] | None = None) -> None:


 if __name__ == '__main__':
-    main()
+    main()
@@ -12,14 +12,12 @@ import json
 import math
 import mmap
 import os
-import pickle
 import re
 import signal
 import struct
 import sys
 import textwrap
 import time
-import zipfile
 from abc import ABC, abstractmethod
 from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
 from dataclasses import dataclass
@@ -954,98 +952,6 @@ def pack_experts_lazy(lazy_tensors: list[LazyTensor]) -> LazyTensor:
    return LazyTensor(load, s, lazy_tensors[0].data_type, 'pack_experts ' + ' | '.join(lt.description for lt in lazy_tensors))


-# Functionality that simulates `torch.load` but where individual tensors are
-# only loaded into memory on demand, not all at once.
-# PyTorch can't do this natively as of time of writing:
-# - https://github.com/pytorch/pytorch/issues/64327
-# This allows us to de-shard without multiplying RAM usage, and also
-# conveniently drops the PyTorch dependency (though we still need numpy).
-
-
-@dataclass
-class LazyStorageKind:
-    data_type: DataType
-
-
-@dataclass
-class LazyStorage:
-    load: Callable[[int, int], NDArray]
-    kind: LazyStorageKind
-    description: str
-
-
-class LazyUnpickler(pickle.Unpickler):
-    def __init__(self, fp: IO[bytes], data_base_path: str, zip_file: zipfile.ZipFile):
-        super().__init__(fp)
-        self.data_base_path = data_base_path
-        self.zip_file = zip_file
-
-    def persistent_load(self, pid: Any) -> Any:
-        assert pid[0] == 'storage'
-        assert isinstance(pid[1], LazyStorageKind)
-        data_type = pid[1].data_type
-        filename_stem = pid[2]
-        filename = f'{self.data_base_path}/{filename_stem}'
-        info = self.zip_file.getinfo(filename)
-
-        def load(offset: int, elm_count: int) -> NDArray:
-            dtype = data_type.dtype
-            with self.zip_file.open(info) as fp:
-                fp.seek(offset * dtype.itemsize)
-                size = elm_count * dtype.itemsize
-                data = fp.read(size)
-            assert len(data) == size
-            return np.frombuffer(data, dtype)
-        description = f'storage data_type={data_type} path-in-zip={filename} path={self.zip_file.filename}'
-        return LazyStorage(load=load, kind=pid[1], description=description)
-
-    @staticmethod
-    def lazy_rebuild_tensor_v2(storage: Any, storage_offset: Any, size: Any, stride: Any,
-                               requires_grad: Any, backward_hooks: Any, metadata: Any = None) -> LazyTensor:
-        assert isinstance(storage, LazyStorage)
-
-        def load() -> UnquantizedTensor:
-            elm_count = stride[0] * size[0]
-            return UnquantizedTensor(storage.load(storage_offset, elm_count).reshape(size))
-        description = f'pickled storage_offset={storage_offset} in {storage.description}'
-        return LazyTensor(load, list(size), storage.kind.data_type, description)
-
-    @staticmethod
-    def rebuild_from_type_v2(func, new_type, args, state):
-        return func(*args)
-
-    CLASSES = {
-        # getattr used here as a workaround for mypy not being smart enough to determine
-        # the staticmethods have a __func__ attribute.
-        ('torch._tensor', '_rebuild_from_type_v2'): getattr(rebuild_from_type_v2, '__func__'),
-        ('torch._utils', '_rebuild_tensor_v2'): getattr(lazy_rebuild_tensor_v2, '__func__'),
-        ('torch', 'BFloat16Storage'): LazyStorageKind(DT_BF16),
-        ('torch', 'HalfStorage'): LazyStorageKind(DT_F16),
-        ('torch', 'FloatStorage'): LazyStorageKind(DT_F32),
-        ('torch', 'IntStorage'): LazyStorageKind(DT_I32),
-        ('torch', 'Tensor'): LazyTensor,
-    }
-
-    def find_class(self, module: str, name: str) -> Any:
-        if not module.startswith('torch'):
-            return super().find_class(module, name)
-        return self.CLASSES[(module, name)]
-
-
-def lazy_load_torch_file(outer_fp: IO[bytes], path: Path) -> ModelPlus:
-    zf = zipfile.ZipFile(outer_fp)
-    pickle_paths = [name for name in zf.namelist() if name.endswith('.pkl')]
-    assert len(pickle_paths) == 1, pickle_paths
-    pickle_fp = zf.open(pickle_paths[0], 'r')
-    unpickler = LazyUnpickler(pickle_fp,
-                              data_base_path=pickle_paths[0][:-4],
-                              zip_file=zf)
-    model = unpickler.load()
-    if 'model' in model: model = model['model']
-    as_dict = dict(model.items())
-    return ModelPlus(model=as_dict, paths=[path], format='torch', vocab=None)
-
-
 def lazy_load_safetensors_file(fp: IO[bytes], path: Path) -> ModelPlus:
    header_size, = struct.unpack('<Q', fp.read(8))
    header: dict[str, dict[str, Any]] = json.loads(fp.read(header_size))
@@ -1082,14 +988,11 @@ def lazy_load_file(path: Path) -> ModelPlus:
    fp = open(path, 'rb')
    first8 = fp.read(8)
    fp.seek(0)
-    if first8[:2] == b'PK':
-        # A zip file, i.e. PyTorch format
-        return lazy_load_torch_file(fp, path)
-    elif struct.unpack('<Q', first8)[0] < 16 * 1024 * 1024:
-        # Probably safetensors
+    if struct.unpack('<Q', first8)[0] < 16 * 1024 * 1024:
+        # Safetensors format
        return lazy_load_safetensors_file(fp, path)
    else:
-        raise ValueError(f"unknown format: {path}")
+        raise ValueError(f"unknown format: {path}. Only safetensors format is supported.")


 In = TypeVar('In')
@@ -1500,15 +1403,11 @@ def load_some_model(path: Path) -> ModelPlus:
    '''Load a model of any supported format.'''
    # Be extra-friendly and accept either a file or a directory:
    if path.is_dir():
-        # Check if it's a set of safetensors files first
-        globs = ["model-00001-of-*.safetensors", "model.safetensors", "consolidated.safetensors", "model-int2.pth"]
+        # Check if it's a set of safetensors files
+        globs = ["model-00001-of-*.safetensors", "model.safetensors", "consolidated.safetensors"]
        files = [file for glob in globs for file in path.glob(glob)]
        if not files:
-            # Try the PyTorch patterns too, with lower priority
-            globs = ["consolidated.00.pth", "pytorch_model-00001-of-*.bin", "*.pt", "pytorch_model.bin"]
-            files = [file for glob in globs for file in path.glob(glob)]
-        if not files:
-            raise FileNotFoundError(f"Can't find model in directory {path}")
+            raise FileNotFoundError(f"Can't find safetensors model in directory {path}")
        if len(files) > 1:
            raise ValueError(f"Found multiple models in {path}, not sure which to pick: {files}")
        path = files[0]
@@ -1600,7 +1499,7 @@ def do_dump_model(model_plus: ModelPlus) -> None:

 def main(args_in: list[str] | None = None) -> None:
    output_choices = ["f32", "f16", "i2"]
-    if np.uint32(1) == np.uint32(1).newbyteorder("<"):
+    if sys.byteorder == "little":
        # We currently only support Q8_0 output on little endian systems.
        output_choices.append("q8_0")
    parser = argparse.ArgumentParser(description="Convert a LLaMA model to a GGML compatible file")
@@ -0,0 +1,473 @@
+#!/usr/bin/env python3
+"""
+Embedding Quantization Script
+This script converts ggml-model-f32.gguf to multiple quantized versions
+with different token embedding types.
+"""
+
+import subprocess
+import os
+import argparse
+import re
+import csv
+from pathlib import Path
+from datetime import datetime
+
+
+class EmbeddingQuantizer:
+    def __init__(self, input_model, output_dir, quantize_bin="../build/bin/llama-quantize", 
+                 bench_bin="../build/bin/llama-bench", stats_dir="../stats", csv_output=None):
+        self.input_model = Path(input_model)
+        self.output_dir = Path(output_dir)
+        self.quantize_bin = Path(quantize_bin)
+        self.bench_bin = Path(bench_bin)
+        self.stats_dir = Path(stats_dir)
+        self.csv_output = Path(csv_output) if csv_output else None
+        
+        # Verify input file exists
+        if not self.input_model.exists():
+            raise FileNotFoundError(f"Input model not found: {self.input_model}")
+        
+        # Verify quantize tool exists
+        if not self.quantize_bin.exists():
+            raise FileNotFoundError(f"Quantize binary not found: {self.quantize_bin}")
+        
+        # Verify bench tool exists
+        if not self.bench_bin.exists():
+            raise FileNotFoundError(f"Benchmark binary not found: {self.bench_bin}")
+        
+        # Create output directories
+        self.output_dir.mkdir(parents=True, exist_ok=True)
+        self.stats_dir.mkdir(parents=True, exist_ok=True)
+        
+        self.results = []
+        self.newly_created_files = set()  # Track newly created files
+        
+    def quantize(self, embedding_type, output_suffix):
+        """
+        Perform single quantization
+        
+        Args:
+            embedding_type: Token embedding type (uppercase format, e.g., Q6_K)
+            output_suffix: Output file suffix (lowercase format, e.g., q6_k)
+        
+        Returns:
+            bool: Whether successful
+        """
+        output_file = self.output_dir / f"ggml-model-i2_s-embed-{output_suffix}.gguf"
+        
+        # Check if file already exists
+        file_already_existed = output_file.exists()
+        
+        if file_already_existed:
+            print(f"ℹ️  File already exists: {output_file}")
+            print(f"   Skipping quantization, will use existing file for benchmark")
+            return True
+        
+        cmd = [
+            str(self.quantize_bin),
+            "--token-embedding-type", embedding_type,
+            str(self.input_model),
+            str(output_file),
+            "I2_S",
+            "1",
+            "1"
+        ]
+        
+        print(f"\n{'='*80}")
+        print(f"🔄 Quantizing with embedding type: {embedding_type}")
+        print(f"📥 Input:  {self.input_model}")
+        print(f"📤 Output: {output_file}")
+        print(f"💻 Command: {' '.join(cmd)}")
+        print(f"{'='*80}\n")
+        
+        start_time = datetime.now()
+        
+        try:
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                cwd=os.getcwd(),
+                timeout=600  # 10 minute timeout
+            )
+            
+            end_time = datetime.now()
+            duration = (end_time - start_time).total_seconds()
+            
+            if result.returncode == 0:
+                # Get output file size
+                file_size_mb = output_file.stat().st_size / (1024 * 1024)
+                
+                print(f"✅ Success! Duration: {duration:.2f}s, Size: {file_size_mb:.2f} MB")
+                
+                # Record newly created file
+                if not file_already_existed:
+                    self.newly_created_files.add(output_file)
+                
+                # Print part of output
+                if result.stdout:
+                    print("\n📊 Quantization output:")
+                    print(result.stdout[-500:] if len(result.stdout) > 500 else result.stdout)
+                
+                return True
+            else:
+                print(f"❌ Failed with return code {result.returncode}")
+                print(f"Error: {result.stderr}")
+                return False
+                
+        except subprocess.TimeoutExpired:
+            print(f"❌ Timeout (exceeded 10 minutes)")
+            return False
+            
+        except Exception as e:
+            print(f"❌ Exception: {e}")
+            return False
+    
+    def benchmark_model(self, output_suffix):
+        """
+        Benchmark model
+        
+        Args:
+            output_suffix: Output file suffix (lowercase format, e.g., q6_k)
+        
+        Returns:
+            dict: Dictionary with benchmark results, or None if failed
+        """
+        model_file = self.output_dir / f"ggml-model-i2_s-embed-{output_suffix}.gguf"
+        
+        if not model_file.exists():
+            print(f"❌ Model file not found for benchmarking: {model_file}")
+            return None
+        
+        cmd = [
+            str(self.bench_bin),
+            "-m", str(model_file),
+            "-p", "128",
+            "-n", "0",
+            "-t", "1,2,4,8",
+            "-ngl", "0"
+        ]
+        
+        print(f"\n{'='*80}")
+        print(f"🏃 Running benchmark for: {output_suffix}")
+        print(f"💻 Command: {' '.join(cmd)}")
+        print(f"{'='*80}\n")
+        
+        try:
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                cwd=os.getcwd(),
+                timeout=300  # 5 minute timeout
+            )
+            
+            if result.returncode == 0:
+                print("✅ Benchmark completed successfully")
+                print("\n📊 Benchmark output:")
+                print(result.stdout)
+                
+                # 解析输出
+                bench_results = self.parse_benchmark_output(result.stdout, output_suffix)
+                return bench_results
+            else:
+                print(f"❌ Benchmark failed with return code {result.returncode}")
+                print(f"Error: {result.stderr}")
+                return None
+                
+        except subprocess.TimeoutExpired:
+            print(f"❌ Benchmark timeout (exceeded 5 minutes)")
+            return None
+            
+        except Exception as e:
+            print(f"❌ Benchmark exception: {e}")
+            return None
+    
+    def parse_benchmark_output(self, output, output_suffix):
+        """
+        Parse benchmark output to extract t/s data (mean±std)
+        
+        Args:
+            output: Benchmark command output
+            output_suffix: Output file suffix
+        
+        Returns:
+            dict: Dictionary with parsed results
+        """
+        results = {
+            'embedding_type': output_suffix,
+            'threads_1': None,
+            'threads_2': None,
+            'threads_4': None,
+            'threads_8': None,
+        }
+        
+        # Parse table data
+        # Find lines containing pp128 and t/s
+        lines = output.strip().split('\n')
+        
+        for line in lines:
+            # Skip header and separator lines
+            if '|' not in line or 'model' in line or '---' in line:
+                continue
+            
+            # Try to extract data
+            # Format similar to: | bitnet-25 2B I2_S - 2 bpw ternary | 1012.28 MiB |     2.74 B | CPU        |      12 |         pp128 |        405.73 ± 3.69 |
+            parts = [p.strip() for p in line.split('|')]
+            
+            if len(parts) >= 8 and 'pp128' in parts[6]:
+                threads_str = parts[5].strip()
+                throughput_str = parts[7].strip()
+                
+                # Extract thread count
+                try:
+                    threads = int(threads_str)
+                except:
+                    continue
+                
+                # Extract t/s data (format: "405.73 ± 3.69" or "405.73")
+                # Try to match "mean ± std" format
+                match_with_std = re.search(r'([\d.]+)\s*±\s*([\d.]+)', throughput_str)
+                if match_with_std:
+                    mean = float(match_with_std.group(1))
+                    std = float(match_with_std.group(2))
+                    throughput = f"{mean:.2f}±{std:.2f}"
+                else:
+                    # Only mean, no std
+                    match = re.search(r'([\d.]+)', throughput_str)
+                    if match:
+                        throughput = f"{float(match.group(1)):.2f}"
+                    else:
+                        continue
+                
+                # Store result based on thread count
+                if threads == 1:
+                    results['threads_1'] = throughput
+                elif threads == 2:
+                    results['threads_2'] = throughput
+                elif threads == 4:
+                    results['threads_4'] = throughput
+                elif threads == 8:
+                    results['threads_8'] = throughput
+        
+        return results
+    
+    def cleanup_model(self, output_suffix):
+        """
+        Cleanup model files (only delete newly created files)
+        
+        Args:
+            output_suffix: Output file suffix
+        """
+        model_file = self.output_dir / f"ggml-model-i2_s-embed-{output_suffix}.gguf"
+        
+        if model_file in self.newly_created_files:
+            try:
+                model_file.unlink()
+                print(f"🗑️  Deleted newly created file: {model_file}")
+                self.newly_created_files.remove(model_file)
+            except Exception as e:
+                print(f"⚠️  Failed to delete {model_file}: {e}")
+        else:
+            print(f"ℹ️  Keeping existing file: {model_file}")
+    
+    def run_all_quantizations(self, types_to_quantize):
+        """
+        Run all quantizations
+        
+        Args:
+            types_to_quantize: List of quantization types, tuples of (embedding_type, output_suffix)
+        """
+        print(f"\n{'='*80}")
+        print(f"🚀 Starting Embedding Quantization and Benchmarking")
+        print(f"{'='*80}")
+        print(f"📥 Input model: {self.input_model}")
+        print(f"📤 Output directory: {self.output_dir}")
+        print(f"📊 Stats directory: {self.stats_dir}")
+        print(f"🔢 Total quantizations: {len(types_to_quantize)}")
+        print(f"{'='*80}\n")
+        
+        total_start = datetime.now()
+        
+        for i, (embedding_type, output_suffix) in enumerate(types_to_quantize, 1):
+            print(f"\n{'#'*80}")
+            print(f"[{i}/{len(types_to_quantize)}] Processing {output_suffix} ({embedding_type})")
+            print(f"{'#'*80}\n")
+            
+            # Quantize model
+            success = self.quantize(embedding_type, output_suffix)
+            
+            if not success:
+                print(f"⚠️  Skipping benchmark for {output_suffix} due to quantization failure")
+                continue
+            
+            # Run benchmark
+            bench_results = self.benchmark_model(output_suffix)
+            
+            if bench_results:
+                self.results.append(bench_results)
+            else:
+                print(f"⚠️  Benchmark failed for {output_suffix}")
+            
+            # Cleanup model files (only delete newly created files)
+            self.cleanup_model(output_suffix)
+            
+            print(f"\n{'#'*80}")
+            print(f"✅ Completed {output_suffix}")
+            print(f"{'#'*80}\n")
+        
+        total_end = datetime.now()
+        total_duration = (total_end - total_start).total_seconds()
+        
+        # 保存结果到CSV
+        self.save_results_to_csv()
+        
+        # 打印总结
+        self.print_summary(total_duration)
+    
+    def save_results_to_csv(self):
+        """将benchmark结果保存到CSV文件"""
+        if not self.results:
+            print("⚠️  No results to save")
+            return
+        
+        # Use user-specified CSV path, otherwise use default path
+        if self.csv_output:
+            csv_file = self.csv_output
+            # Ensure parent directory exists
+            csv_file.parent.mkdir(parents=True, exist_ok=True)
+        else:
+            csv_file = self.stats_dir / f"embedding_benchmark.csv"
+        
+        print(f"\n💾 Saving results to: {csv_file}")
+        
+        try:
+            with open(csv_file, 'w', newline='') as f:
+                fieldnames = ['embedding_type', 'threads_1', 'threads_2', 'threads_4', 'threads_8']
+                writer = csv.DictWriter(f, fieldnames=fieldnames)
+                
+                writer.writeheader()
+                for result in self.results:
+                    writer.writerow(result)
+            
+            print(f"✅ Results saved successfully")
+            
+            # Also print table
+            print(f"\n📊 Benchmark Results:")
+            print(f"{'Type':<15} {'1 thread':<18} {'2 threads':<18} {'4 threads':<18} {'8 threads':<18}")
+            print("-" * 87)
+            for result in self.results:
+                t1 = result['threads_1'] if result['threads_1'] else "N/A"
+                t2 = result['threads_2'] if result['threads_2'] else "N/A"
+                t4 = result['threads_4'] if result['threads_4'] else "N/A"
+                t8 = result['threads_8'] if result['threads_8'] else "N/A"
+                print(f"{result['embedding_type']:<15} {t1:<18} {t2:<18} {t4:<18} {t8:<18}")
+                
+        except Exception as e:
+            print(f"❌ Failed to save results: {e}")
+        
+    def print_summary(self, total_duration):
+        """Print quantization summary"""
+        print(f"\n\n{'='*80}")
+        print(f"📊 QUANTIZATION AND BENCHMARK SUMMARY")
+        print(f"{'='*80}\n")
+        
+        successful = len(self.results)
+        total = len(self.results)
+        
+        print(f"✅ Completed: {successful} benchmarks")
+        print(f"⏱️  Total duration: {total_duration/60:.2f} minutes\n")
+        
+        if self.results:
+            if self.csv_output and self.csv_output.exists():
+                print(f"📁 Results saved to: {self.csv_output}")
+            else:
+                csv_files = list(self.stats_dir.glob("embedding_benchmark*.csv"))
+                if csv_files:
+                    latest_csv = max(csv_files, key=lambda p: p.stat().st_mtime)
+                    print(f"📁 Results saved to: {latest_csv}")
+        
+        print(f"\n{'='*80}\n")
+
+
+def main():
+    parser = argparse.ArgumentParser(description='Quantize model embeddings to multiple formats')
+    parser.add_argument('--input', '-i',
+                        default='../models/BitNet-b1.58-2B-4T/ggml-model-f32.gguf',
+                        help='Input model path (default: ../models/BitNet-b1.58-2B-4T/ggml-model-f32.gguf)')
+    parser.add_argument('--output-dir', '-o',
+                        default='../models/BitNet-b1.58-2B-4T',
+                        help='Output directory (default: ../models/BitNet-b1.58-2B-4T)')
+    parser.add_argument('--quantize-bin', '-q',
+                        default='../build/bin/llama-quantize',
+                        help='Path to llama-quantize binary (default: ../build/bin/llama-quantize)')
+    parser.add_argument('--bench-bin', '-b',
+                        default='../build/bin/llama-bench',
+                        help='Path to llama-bench binary (default: ../build/bin/llama-bench)')
+    parser.add_argument('--stats-dir',
+                        default='../stats',
+                        help='Directory to save benchmark results (default: ../stats)')
+    parser.add_argument('--csv-output', '-c',
+                        help='Custom path for CSV output file (e.g., stats/my_results.csv)')
+    parser.add_argument('--types', '-t',
+                        nargs='+',
+                        help='Specific types to quantize (e.g., f32 q6_k q4_0)')
+    parser.add_argument('--skip-existing', '-s',
+                        action='store_true',
+                        help='Skip quantization if output file already exists (will still benchmark existing files)')
+    
+    args = parser.parse_args()
+    
+    # Define all supported quantization types
+    # Format: (embedding_type for command line, output_suffix for filename)
+    all_types = [
+        ('F32', 'f32'),
+        ('F16', 'f16'),
+        ('Q8_0', 'q8_0'),
+        ('Q6_K', 'q6_k'),
+        ('Q5_0', 'q5_0'),
+        ('Q4_0', 'q4_0'),
+        ('Q3_K', 'q3_k'),
+        ('TQ2_0', 'tq2_0'),
+    ]
+    
+    # If specific types are specified, filter the list
+    if args.types:
+        types_lower = [t.lower() for t in args.types]
+        types_to_quantize = [(et, os) for et, os in all_types if os.lower() in types_lower]
+        if not types_to_quantize:
+            print(f"❌ No valid types specified. Available types: {', '.join([os for _, os in all_types])}")
+            return
+    else:
+        types_to_quantize = all_types
+    
+    # If skip existing files is enabled, no need to filter
+    # Because new logic will automatically detect and skip during quantization, but will still benchmark
+    
+    # 创建量化器并运行
+    try:
+        quantizer = EmbeddingQuantizer(
+            args.input, 
+            args.output_dir, 
+            args.quantize_bin,
+            args.bench_bin,
+            args.stats_dir,
+            args.csv_output
+        )
+        quantizer.run_all_quantizations(types_to_quantize)
+    except FileNotFoundError as e:
+        print(f"❌ Error: {e}")
+        return 1
+    except KeyboardInterrupt:
+        print("\n\n⚠️  Quantization interrupted by user")
+        return 1
+    except Exception as e:
+        print(f"\n❌ Unexpected error: {e}")
+        import traceback
+        traceback.print_exc()
+        return 1
+
+
+if __name__ == "__main__":
+    exit(main() or 0)
@@ -0,0 +1,573 @@
+#!/bin/bash
+# Unified GEMM kernel benchmark script
+# Builds, tests, and benchmarks the GEMM kernel with configurable output
+
+set -e
+
+# Default values
+BUILD_DIR="../build"
+ITERATIONS=1000
+OUTPUT_CSV=""
+SKIP_BUILD=false
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Print usage
+print_usage() {
+    cat << EOF
+Usage: $0 [options]
+
+Options:
+  -o, --output <path>     Output CSV file path (default: ../stats/gemm_kernel_test_noparal.csv)
+  -i, --iterations <num>  Number of iterations per test (default: 1000)
+  -s, --skip-build        Skip building the benchmark binary
+  -h, --help              Show this help message
+
+Examples:
+  # Run with default settings
+  $0
+
+  # Specify custom output file
+  $0 -o /path/to/my_results.csv
+
+  # Quick test with fewer iterations
+  $0 -i 100 -o quick_test.csv
+
+  # Skip build if already compiled
+  $0 -s -o results.csv
+EOF
+}
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        -o|--output)
+            OUTPUT_CSV="$2"
+            shift 2
+            ;;
+        -i|--iterations)
+            ITERATIONS="$2"
+            shift 2
+            ;;
+        -s|--skip-build)
+            SKIP_BUILD=true
+            shift
+            ;;
+        -h|--help)
+            print_usage
+            exit 0
+            ;;
+        *)
+            echo "Unknown option: $1"
+            print_usage
+            exit 1
+            ;;
+    esac
+done
+
+# Set default output CSV if not specified
+if [ -z "$OUTPUT_CSV" ]; then
+    OUTPUT_CSV="${SCRIPT_DIR}/../stats/gemm_kernel_test_noparal.csv"
+fi
+
+# Create output directory first
+mkdir -p "$(dirname "$OUTPUT_CSV")"
+
+# Convert to absolute path
+if [[ "$OUTPUT_CSV" = /* ]]; then
+    # Already absolute path
+    OUTPUT_CSV="$OUTPUT_CSV"
+else
+    # Convert relative path to absolute
+    OUTPUT_CSV="$(cd "$(dirname "$OUTPUT_CSV")" && pwd)/$(basename "$OUTPUT_CSV")"
+fi
+
+echo "=========================================="
+echo "GEMM Kernel Benchmark Suite"
+echo "=========================================="
+echo "Configuration:"
+echo "  Iterations: $ITERATIONS"
+echo "  Output CSV: $OUTPUT_CSV"
+echo "  Skip build: $SKIP_BUILD"
+echo "=========================================="
+echo ""
+
+# Build the benchmark binary
+if [ "$SKIP_BUILD" = false ]; then
+    echo "Step 1: Building GEMM kernel benchmark..."
+    echo "------------------------------------------"
+    
+    CXX=${CXX:-g++}
+    
+    # Create build directory if it doesn't exist
+    mkdir -p "${SCRIPT_DIR}/${BUILD_DIR}"
+    
+    # Create temporary C++ source file
+    TEMP_CPP="${SCRIPT_DIR}/${BUILD_DIR}/test_gemm_kernel_temp.cpp"
+    
+    cat > "${TEMP_CPP}" << 'EOF'
+/**
+ * Standalone benchmark for ggml_gemm_i2_i8_s kernel
+ * 
+ * This program tests the performance of the ggml_gemm_i2_i8_s kernel
+ * with configurable matrix sizes and iteration counts.
+ * 
+ * Usage: ./test_gemm_kernel [options]
+ *   -n <size>   : embedding dimension (must be divisible by 4, default: 2048)
+ *   -r <rows>   : number of rows in matrix Y (default: 32)
+ *   -c <cols>   : number of columns in matrix X (default: 128)
+ *   -i <iters>  : number of iterations (default: 1000)
+ *   -w <warmup> : number of warmup iterations (default: 10)
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <stdint.h>
+#include <math.h>
+#include <assert.h>
+
+// Include necessary headers
+#include "../include/gemm-config.h"
+
+// Function declarations (from ggml-quants.h)
+extern "C" void ggml_vec_dot_i2_i8_s(int n, float * s, size_t bs, const void * vx, size_t bx, const void * vy, size_t by, int nrc);
+
+// GEMM kernel definition
+void ggml_gemm_i2_i8_s(int n, float * s, size_t bs, const void * vx, const void * vy, int nr, int nc) {
+#if defined(ACT_PARALLEL)
+    const int64_t row_block = ROW_BLOCK_SIZE;
+    const int64_t col_block = COL_BLOCK_SIZE;
+
+    for (int64_t c0 = 0; c0 < nc; c0 += col_block) {
+        int64_t cur_c = (c0 + col_block <= nc) ? col_block : (nc - c0);
+        for (int64_t r0 = 0; r0 < nr; r0 += row_block) {
+            int64_t cur_r = (r0 + row_block <= nr) ? row_block : (nr - r0);
+            const void * vy_r = (const uint8_t *)vy + r0 * n;
+            for (int64_t c = 0; c < cur_c; ++c) {
+                const int64_t col = c0 + c;
+                float * s_col = s + col;
+                const void * vx_col = (const uint8_t *)vx + col * n / 4;
+                ggml_vec_dot_i2_i8_s(n, s_col + r0 * bs, bs, vx_col, n, vy_r, n, cur_r);
+            }
+        }
+    }
+#else
+    const int64_t row_block = ROW_BLOCK_SIZE;
+    const int64_t col_block = COL_BLOCK_SIZE;
+
+    for (int64_t r0 = 0; r0 < nr; r0 += row_block) {
+        int64_t cur_r = (r0 + row_block <= nr) ? row_block : (nr - r0);
+        for (int64_t c0 = 0; c0 < nc; c0 += col_block) {
+            int64_t cur_c = (c0 + col_block <= nc) ? col_block : (nc - c0);
+            const void * vx_c = (const uint8_t *)vx + c0 * n / 4;
+            for (int64_t r = 0; r < cur_r; ++r) {
+                const int64_t row = r0 + r;
+                float * s_row = s + row * bs;
+                const void * vy_row = (const uint8_t *)vy + row * n;
+                ggml_vec_dot_i2_i8_s(n, s_row + c0, bs, vx_c, n, vy_row, n, cur_c);
+            }
+        }
+    }
+#endif
+}
+
+// Helper function to get current time in nanoseconds
+double get_time_ns() {
+    struct timespec ts;
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    return ts.tv_sec * 1e9 + ts.tv_nsec;
+}
+
+// Initialize matrix with random i2 values (2-bit quantized)
+void init_matrix_i2(uint8_t* data, int n, int cols) {
+    // i2 format: 4 values per byte (2 bits each)
+    int total_bytes = n * cols / 4;
+    for (int i = 0; i < total_bytes; i++) {
+        data[i] = rand() & 0xFF;
+    }
+}
+
+// Initialize matrix with random i8 values
+void init_matrix_i8(int8_t* data, int n, int rows) {
+    int total_elements = n * rows;
+    for (int i = 0; i < total_elements; i++) {
+        data[i] = (int8_t)((rand() % 256) - 128);
+    }
+}
+
+// Benchmark configuration
+struct BenchmarkConfig {
+    int n;           // embedding dimension (must be divisible by 4)
+    int nr;          // number of rows in Y matrix
+    int nc;          // number of columns in X matrix
+    int iterations;  // number of benchmark iterations
+    int warmup;      // number of warmup iterations
+};
+
+void print_config(const BenchmarkConfig& config) {
+    printf("=" "=%.78s\n", "===============================================================================");
+    printf("Benchmark Configuration:\n");
+    printf("=" "=%.78s\n", "===============================================================================");
+    printf("  Embedding dimension (n)    : %d\n", config.n);
+    printf("  Matrix Y rows (nr)         : %d\n", config.nr);
+    printf("  Matrix X columns (nc)      : %d\n", config.nc);
+    printf("  Iterations                 : %d\n", config.iterations);
+    printf("  Warmup iterations          : %d\n", config.warmup);
+    printf("\nMatrix sizes:\n");
+    printf("  X (i2): %d x %d (%.2f KB)\n", config.nc, config.n, 
+           (config.nc * config.n / 4) / 1024.0);
+    printf("  Y (i8): %d x %d (%.2f KB)\n", config.nr, config.n,
+           (config.nr * config.n) / 1024.0);
+    printf("  S (f32): %d x %d (%.2f KB)\n", config.nr, config.nc,
+           (config.nr * config.nc * sizeof(float)) / 1024.0);
+    printf("\nGEMM Config:\n");
+#if defined(ACT_PARALLEL)
+    printf("  ACT_PARALLEL              : ON\n");
+#else
+    printf("  ACT_PARALLEL              : OFF\n");
+#endif
+    printf("  ROW_BLOCK_SIZE            : %d\n", ROW_BLOCK_SIZE);
+    printf("  COL_BLOCK_SIZE            : %d\n", COL_BLOCK_SIZE);
+    printf("  PARALLEL_SIZE             : %d\n", PARALLEL_SIZE);
+    printf("=" "=%.78s\n\n", "===============================================================================");
+}
+
+void run_benchmark(const BenchmarkConfig& config) {
+    // Allocate matrices
+    printf("Allocating matrices...\n");
+    
+    // X matrix (i2 format): nc x n, but stored as nc x (n/4) bytes
+    // Align to 64 bytes for AVX-512, which is backward compatible with AVX2 (32 bytes)
+    size_t x_size = config.nc * config.n / 4;
+    size_t x_size_aligned = ((x_size + 63) / 64) * 64;
+    uint8_t* X = (uint8_t*)aligned_alloc(64, x_size_aligned);
+    
+    // Y matrix (i8 format): nr x n
+    size_t y_size = config.nr * config.n;
+    size_t y_size_aligned = ((y_size + 63) / 64) * 64;
+    int8_t* Y = (int8_t*)aligned_alloc(64, y_size_aligned);
+    
+    // Result matrix (float32): nr x nc
+    size_t s_size = config.nr * config.nc * sizeof(float);
+    size_t s_size_aligned = ((s_size + 63) / 64) * 64;
+    float* S = (float*)aligned_alloc(64, s_size_aligned);
+    
+    if (!X || !Y || !S) {
+        fprintf(stderr, "Failed to allocate memory\n");
+        exit(1);
+    }
+    
+    // Initialize matrices with random data
+    printf("Initializing matrices with random data...\n");
+    srand(time(NULL));
+    init_matrix_i2(X, config.n, config.nc);
+    init_matrix_i8(Y, config.n, config.nr);
+    memset(S, 0, config.nr * config.nc * sizeof(float));
+    
+    // Warmup
+    printf("Running %d warmup iterations...\n", config.warmup);
+    for (int i = 0; i < config.warmup; i++) {
+        ggml_gemm_i2_i8_s(config.n, S, config.nc, X, Y, config.nr, config.nc);
+    }
+    
+    // Benchmark
+    printf("Running %d benchmark iterations...\n", config.iterations);
+    double total_time = 0.0;
+    double min_time = 1e20;
+    double max_time = 0.0;
+    
+    for (int i = 0; i < config.iterations; i++) {
+        double start = get_time_ns();
+        ggml_gemm_i2_i8_s(config.n, S, config.nc, X, Y, config.nr, config.nc);
+        double end = get_time_ns();
+        
+        double elapsed = end - start;
+        total_time += elapsed;
+        if (elapsed < min_time) min_time = elapsed;
+        if (elapsed > max_time) max_time = elapsed;
+        
+        if ((i + 1) % 100 == 0) {
+            printf("  Progress: %d/%d iterations\n", i + 1, config.iterations);
+        }
+    }
+    
+    // Calculate statistics
+    double avg_time_ns = total_time / config.iterations;
+    double avg_time_ms = avg_time_ns / 1e6;
+    double min_time_ms = min_time / 1e6;
+    double max_time_ms = max_time / 1e6;
+    
+    // Calculate GFLOPS
+    // For GEMM: nr x nc x n multiply-adds = 2 * nr * nc * n FLOPs
+    double flops = 2.0 * config.nr * config.nc * config.n;
+    double gflops = (flops / avg_time_ns);
+    
+    // Calculate throughput (tokens/s assuming each column is a token)
+    double throughput = (config.nc * 1e9) / avg_time_ns;
+    
+    // Print results
+    printf("\n");
+    printf("=" "=%.78s\n", "===============================================================================");
+    printf("Benchmark Results:\n");
+    printf("=" "=%.78s\n", "===============================================================================");
+    printf("  Average time  : %.3f ms\n", avg_time_ms);
+    printf("  Min time      : %.3f ms\n", min_time_ms);
+    printf("  Max time      : %.3f ms\n", max_time_ms);
+    printf("  Std dev       : %.3f ms\n", sqrt((max_time_ms - min_time_ms) * (max_time_ms - min_time_ms) / 12));
+    printf("\nPerformance:\n");
+    printf("  GFLOPS        : %.2f\n", gflops);
+    printf("  Throughput    : %.2f tokens/s\n", throughput);
+    printf("  Latency/token : %.3f us\n", (avg_time_ms * 1000) / config.nc);
+    printf("=" "=%.78s\n", "===============================================================================");
+    
+    // Cleanup
+    free(X);
+    free(Y);
+    free(S);
+}
+
+void print_usage(const char* program) {
+    printf("Usage: %s [options]\n", program);
+    printf("Options:\n");
+    printf("  -n <size>    Embedding dimension (must be divisible by 4, default: 2048)\n");
+    printf("  -r <rows>    Number of rows in matrix Y (default: 32)\n");
+    printf("  -c <cols>    Number of columns in matrix X (default: 128)\n");
+    printf("  -i <iters>   Number of iterations (default: 1000)\n");
+    printf("  -w <warmup>  Number of warmup iterations (default: 10)\n");
+    printf("  -h           Show this help message\n");
+}
+
+int main(int argc, char** argv) {
+    BenchmarkConfig config = {
+        .n = 2048,
+        .nr = 32,
+        .nc = 128,
+        .iterations = 1000,
+        .warmup = 10
+    };
+    
+    // Parse command line arguments
+    for (int i = 1; i < argc; i++) {
+        if (strcmp(argv[i], "-n") == 0 && i + 1 < argc) {
+            config.n = atoi(argv[++i]);
+        } else if (strcmp(argv[i], "-r") == 0 && i + 1 < argc) {
+            config.nr = atoi(argv[++i]);
+        } else if (strcmp(argv[i], "-c") == 0 && i + 1 < argc) {
+            config.nc = atoi(argv[++i]);
+        } else if (strcmp(argv[i], "-i") == 0 && i + 1 < argc) {
+            config.iterations = atoi(argv[++i]);
+        } else if (strcmp(argv[i], "-w") == 0 && i + 1 < argc) {
+            config.warmup = atoi(argv[++i]);
+        } else if (strcmp(argv[i], "-h") == 0) {
+            print_usage(argv[0]);
+            return 0;
+        } else {
+            fprintf(stderr, "Unknown option: %s\n", argv[i]);
+            print_usage(argv[0]);
+            return 1;
+        }
+    }
+    
+    // Validate configuration
+    if (config.n % 4 != 0) {
+        fprintf(stderr, "Error: Embedding dimension (-n) must be divisible by 4\n");
+        return 1;
+    }
+    
+    if (config.n <= 0 || config.nr <= 0 || config.nc <= 0 || config.iterations <= 0) {
+        fprintf(stderr, "Error: All size parameters must be positive\n");
+        return 1;
+    }
+    
+    // Run benchmark
+    print_config(config);
+    run_benchmark(config);
+    
+    return 0;
+}
+EOF
+    
+    # Compiler flags
+    CXXFLAGS="-O3 -march=native -mtune=native -std=c++17 -fopenmp"
+    CXXFLAGS+=" -I${SCRIPT_DIR}/.. -I${SCRIPT_DIR}/../include"
+    CXXFLAGS+=" -I${SCRIPT_DIR}/../3rdparty/llama.cpp/ggml/include"
+    CXXFLAGS+=" -I${SCRIPT_DIR}/../3rdparty/llama.cpp/ggml/src"
+    CXXFLAGS+=" -I${SCRIPT_DIR}/../3rdparty/llama.cpp/include"
+    CXXFLAGS+=" -DNDEBUG -ffast-math"
+    
+    # Link flags
+    LDFLAGS="-lm -lpthread"
+    
+    # Link with pre-built libraries
+    GGML_LIB_DIR="${SCRIPT_DIR}/../build/3rdparty/llama.cpp/ggml/src"
+    GGML_SO="${GGML_LIB_DIR}/libggml.so"
+    
+    if [ ! -f "${GGML_SO}" ]; then
+        echo "❌ Error: Cannot find libggml.so at ${GGML_SO}"
+        echo "Please build the project first with: cmake --build build"
+        rm -f "${TEMP_CPP}"
+        exit 1
+    fi
+    
+    LDFLAGS+=" -L${GGML_LIB_DIR} -lggml -Wl,-rpath,${GGML_LIB_DIR}"
+    
+    # Output binary
+    BENCHMARK_BIN="${SCRIPT_DIR}/${BUILD_DIR}/test_gemm_kernel"
+    
+    echo "Compiler: ${CXX}"
+    echo "Building from embedded source..."
+    echo ""
+    
+    # Build
+    ${CXX} ${CXXFLAGS} "${TEMP_CPP}" -o ${BENCHMARK_BIN} ${LDFLAGS}
+    
+    if [ $? -eq 0 ]; then
+        echo "✅ Build successful!"
+        rm -f "${TEMP_CPP}"
+        echo ""
+    else
+        echo "❌ Build failed!"
+        rm -f "${TEMP_CPP}"
+        exit 1
+    fi
+else
+    echo "Step 1: Skipping build (using existing binary)"
+    echo "------------------------------------------"
+    BENCHMARK_BIN="${SCRIPT_DIR}/${BUILD_DIR}/test_gemm_kernel"
+    
+    if [ ! -f "${BENCHMARK_BIN}" ]; then
+        echo "❌ Error: Benchmark binary not found at ${BENCHMARK_BIN}"
+        echo "Please run without -s to build it first."
+        exit 1
+    fi
+    echo "✅ Found existing binary"
+    echo ""
+fi
+
+# Set LD_LIBRARY_PATH to include the GGML library directory
+GGML_LIB_DIR="${SCRIPT_DIR}/../build/3rdparty/llama.cpp/ggml/src"
+export LD_LIBRARY_PATH="${GGML_LIB_DIR}:${LD_LIBRARY_PATH}"
+
+echo "Step 2: Running benchmark tests"
+echo "------------------------------------------"
+echo "Library path: ${GGML_LIB_DIR}"
+echo ""
+
+# Write CSV header
+echo "test_name,n,nr,nc,time_ms,gflops,throughput_tokens_per_sec" > "$OUTPUT_CSV"
+echo "Results will be saved to: $OUTPUT_CSV"
+echo ""
+
+# Function to extract metrics and append to CSV
+extract_and_save() {
+    local test_name="$1"
+    local output="$2"
+    
+    # Extract values using grep and awk
+    local n=$(echo "$output" | grep "Embedding dimension" | awk '{print $5}')
+    local nr=$(echo "$output" | grep "Matrix Y rows" | awk '{print $6}')
+    local nc=$(echo "$output" | grep "Matrix X columns" | awk '{print $6}')
+    local avg_time=$(echo "$output" | grep "Average time" | awk '{print $4}')
+    local min_time=$(echo "$output" | grep "Min time" | awk '{print $4}')
+    local max_time=$(echo "$output" | grep "Max time" | awk '{print $4}')
+    local gflops=$(echo "$output" | grep "GFLOPS" | awk '{print $3}')
+    local throughput=$(echo "$output" | grep "Throughput" | awk '{print $3}')
+    
+    # Check if values were extracted successfully
+    if [ -z "$avg_time" ] || [ -z "$min_time" ] || [ -z "$max_time" ]; then
+        echo "Warning: Failed to extract timing data for ${test_name}"
+        echo "${test_name},${n},${nr},${nc},N/A,N/A,N/A" >> "$OUTPUT_CSV"
+        return
+    fi
+    
+    # Calculate standard deviation estimate from range
+    # Using awk with proper variable passing
+    local std_time=$(awk -v min="$min_time" -v max="$max_time" 'BEGIN {printf "%.4f", (max - min) / 4}')
+    
+    # Format as mean±std
+    local time_formatted="${avg_time}±${std_time}"
+    
+    # Append to CSV
+    echo "${test_name},${n},${nr},${nc},${time_formatted},${gflops},${throughput}" >> "$OUTPUT_CSV"
+}
+
+# Run benchmark tests
+echo "=========================================="
+echo "BitNet-2B Typical Shapes Performance Test"
+echo "=========================================="
+echo ""
+
+echo "Test 1: Single Token Generation (Attention QKV projection)"
+echo "  Scenario: Generating 1 token at a time"
+echo "  Shape: n=2048, r=1, c=2048"
+OUTPUT=$($BENCHMARK_BIN -n 2048 -r 1 -c 2048 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "single_token_gen" "$OUTPUT"
+echo ""
+
+echo "Test 2: Small Batch Prompt Processing (Attention QKV projection)"
+echo "  Scenario: Processing prompt with 128 tokens, batch size 1"
+echo "  Shape: n=2048, r=128, c=2048"
+OUTPUT=$($BENCHMARK_BIN -n 2048 -r 128 -c 2048 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "small_batch_prompt" "$OUTPUT"
+echo ""
+
+echo "Test 3: Medium Batch Prompt Processing (Attention QKV projection)"
+echo "  Scenario: Processing prompt with 256 tokens or batch of 256"
+echo "  Shape: n=2048, r=256, c=2048"
+OUTPUT=$($BENCHMARK_BIN -n 2048 -r 256 -c 2048 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "medium_batch_prompt" "$OUTPUT"
+echo ""
+
+echo "Test 4: Large Batch Processing (Attention QKV projection)"
+echo "  Scenario: Processing 512 tokens or batch of 512"
+echo "  Shape: n=2048, r=512, c=2048"
+OUTPUT=$($BENCHMARK_BIN -n 2048 -r 512 -c 2048 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "large_batch_prompt" "$OUTPUT"
+echo ""
+
+echo "Test 5: FFN Up-projection (Small batch)"
+echo "  Scenario: Feed-forward network expansion, 128 tokens"
+echo "  Shape: n=2048, r=128, c=8192"
+OUTPUT=$($BENCHMARK_BIN -n 2048 -r 128 -c 8192 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "ffn_up_projection" "$OUTPUT"
+echo ""
+
+echo "Test 6: FFN Down-projection (Small batch)"
+echo "  Scenario: Feed-forward network reduction, 128 tokens"
+echo "  Shape: n=8192, r=128, c=2048"
+OUTPUT=$($BENCHMARK_BIN -n 8192 -r 128 -c 2048 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "ffn_down_projection" "$OUTPUT"
+echo ""
+
+echo "Test 7: Long Context Processing"
+echo "  Scenario: Processing very long context (2048 tokens)"
+echo "  Shape: n=2048, r=2048, c=2048"
+OUTPUT=$($BENCHMARK_BIN -n 2048 -r 2048 -c 2048 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "long_context" "$OUTPUT"
+echo ""
+
+echo "Test 8: Batched Token Generation"
+echo "  Scenario: Generating tokens for 32 sequences simultaneously"
+echo "  Shape: n=2048, r=32, c=2048"
+OUTPUT=$($BENCHMARK_BIN -n 2048 -r 32 -c 2048 -i $ITERATIONS 2>&1)
+echo "$OUTPUT"
+extract_and_save "batched_token_gen" "$OUTPUT"
+echo ""
+
+echo "=========================================="
+echo "All tests completed successfully!"
+echo "=========================================="
+echo "Results saved to: $OUTPUT_CSV"
+echo ""
+echo "Summary:"
+wc -l "$OUTPUT_CSV" | awk '{print "  Total records:", $1 - 1}'
+echo "  Output file: $OUTPUT_CSV"
+echo "=========================================="
@@ -0,0 +1,608 @@
+#!/usr/bin/env python3
+"""
+Perplexity Test Script
+Tests GGUF model perplexity on multiple datasets using llama-perplexity.
+"""
+
+import os
+import subprocess
+import time
+import csv
+import re
+from datetime import datetime
+from pathlib import Path
+import argparse
+import tempfile
+import shutil
+import statistics
+
+
+class PerplexityTester:
+    def __init__(self, model_path, llama_perplexity_bin="../build/bin/llama-perplexity", 
+                 data_dir="../data", output_dir="perplexity_results", quick_mode=False,
+                 quantize_bin="../build/bin/llama-quantize", test_embeddings=False, csv_output=None):
+        self.model_path = Path(model_path)
+        self.llama_perplexity_bin = Path(llama_perplexity_bin)
+        self.quantize_bin = Path(quantize_bin)
+        self.data_dir = Path(data_dir)
+        self.output_dir = Path(output_dir)
+        self.quick_mode = quick_mode
+        self.test_embeddings = test_embeddings
+        self.csv_output = Path(csv_output) if csv_output else None
+        self.results = []
+        self.created_models = set()  # Track newly created model files
+        self.temp_files = []  # Track temporary files for cleanup
+        
+        # Embedding types to test
+        self.embedding_types = [
+            ('F32', 'f32'),
+            ('F16', 'f16'),
+            ('Q8_0', 'q8_0'),
+            ('Q6_K', 'q6_k'),
+            ('Q5_0', 'q5_0'),
+            ('Q4_0', 'q4_0'),
+            ('Q3_K', 'q3_k'),
+            ('TQ2_0', 'tq2_0'),
+        ]
+        
+        # Create output directory
+        self.output_dir.mkdir(parents=True, exist_ok=True)
+        
+        # Verify llama-perplexity binary exists
+        if not self.llama_perplexity_bin.exists():
+            raise FileNotFoundError(f"llama-perplexity binary not found: {self.llama_perplexity_bin}")
+        
+        # Verify quantize binary exists if testing embeddings
+        if self.test_embeddings and not self.quantize_bin.exists():
+            raise FileNotFoundError(f"llama-quantize binary not found: {self.quantize_bin}")
+        
+        # Verify model file exists
+        if not self.model_path.exists():
+            raise FileNotFoundError(f"Model file not found: {self.model_path}")
+    
+    def find_datasets(self):
+        """Find all test.txt files in dataset directories."""
+        datasets = []
+        
+        if not self.data_dir.exists():
+            print(f"❌ Data directory not found: {self.data_dir}")
+            return datasets
+        
+        print(f"\n🔍 Searching for datasets in {self.data_dir}...")
+        
+        # Look for test.txt files in subdirectories
+        for dataset_dir in sorted(self.data_dir.iterdir()):
+            if dataset_dir.is_dir():
+                test_file = dataset_dir / "test.txt"
+                if test_file.exists():
+                    size_mb = test_file.stat().st_size / (1024 * 1024)
+                    datasets.append({
+                        'name': dataset_dir.name,
+                        'path': test_file,
+                        'size': test_file.stat().st_size,
+                        'size_mb': size_mb
+                    })
+                    print(f"   ✅ {dataset_dir.name:<20} ({size_mb:.2f} MB)")
+                else:
+                    print(f"   ⚠️  {dataset_dir.name:<20} (no test.txt found)")
+        
+        return datasets
+    
+    def create_quick_dataset(self, dataset_path, num_chars=4096):
+        """Create a temporary dataset with only the first N characters for quick testing."""
+        temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt', encoding='utf-8')
+        self.temp_files.append(temp_file.name)
+        
+        try:
+            with open(dataset_path, 'r', encoding='utf-8', errors='ignore') as f:
+                content = f.read(num_chars)
+                temp_file.write(content)
+            temp_file.close()
+            return Path(temp_file.name)
+        except Exception as e:
+            print(f"⚠️  Failed to create quick dataset: {e}")
+            temp_file.close()
+            return dataset_path
+    
+    def cleanup_temp_files(self):
+        """Clean up temporary files."""
+        for temp_file in self.temp_files:
+            try:
+                os.unlink(temp_file)
+            except:
+                pass
+        self.temp_files = []
+    
+    def run_perplexity_test(self, dataset_name, dataset_path, threads=16, ctx_size=512, model_override=None):
+        """Run perplexity test on a single dataset."""
+        test_model = model_override if model_override else self.model_path
+        
+        print(f"\n{'='*80}")
+        print(f"📊 Testing on dataset: {dataset_name}")
+        print(f"   File: {dataset_path}")
+        print(f"   Model: {test_model.name}")
+        print(f"{'='*80}")
+        
+        cmd = [
+            str(self.llama_perplexity_bin),
+            "-m", str(test_model),
+            "-f", str(dataset_path),
+            "-t", str(threads),
+            "-c", str(ctx_size),
+            "-ngl", "0"  # CPU only
+        ]
+        
+        print(f"💻 Command: {' '.join(cmd)}")
+        print(f"⏱️  Starting test...\n")
+        
+        start_time = time.time()
+        
+        try:
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                timeout=3600,  # 1 hour timeout
+                cwd=os.getcwd()
+            )
+            
+            elapsed_time = time.time() - start_time
+            
+            if result.returncode == 0:
+                # Parse perplexity from output (check both stdout and stderr)
+                combined_output = result.stdout + "\n" + result.stderr
+                ppl = self.parse_perplexity(combined_output)
+                
+                if ppl is not None:
+                    print(f"\n✅ Perplexity: {ppl}")
+                    print(f"⏱️  Time: {elapsed_time:.2f}s ({elapsed_time/60:.2f} min)")
+                    status = "success"
+                else:
+                    print(f"\n⚠️  Test completed but could not parse perplexity")
+                    print(f"Last 500 chars of stdout:")
+                    print(result.stdout[-500:])
+                    print(f"Last 500 chars of stderr:")
+                    print(result.stderr[-500:])
+                    status = "parse_error"
+                    ppl = None
+            else:
+                print(f"\n❌ Test failed with return code {result.returncode}")
+                print(f"Error: {result.stderr[:500]}")
+                status = "failed"
+                ppl = None
+                elapsed_time = time.time() - start_time
+            
+            return {
+                'dataset': dataset_name,
+                'perplexity': ppl,
+                'time': elapsed_time,
+                'status': status,
+                'stdout': result.stdout,
+                'stderr': result.stderr
+            }
+            
+        except subprocess.TimeoutExpired:
+            elapsed_time = time.time() - start_time
+            print(f"\n❌ Timeout after {elapsed_time:.2f}s")
+            return {
+                'dataset': dataset_name,
+                'perplexity': None,
+                'time': elapsed_time,
+                'status': 'timeout',
+                'stdout': '',
+                'stderr': 'Test exceeded 1 hour timeout'
+            }
+        except Exception as e:
+            elapsed_time = time.time() - start_time
+            print(f"\n❌ Error: {e}")
+            return {
+                'dataset': dataset_name,
+                'perplexity': None,
+                'time': elapsed_time,
+                'status': 'error',
+                'stdout': '',
+                'stderr': str(e)
+            }
+    
+    def parse_perplexity(self, output):
+        """Parse perplexity value (mean±std format) from llama-perplexity output."""
+        # First try to match "PPL = mean +/- std" format
+        pattern_with_std = r'PPL\s*=\s*(\d+\.?\d*)\s*\+/-\s*(\d+\.?\d*)'
+        match = re.search(pattern_with_std, output, re.IGNORECASE | re.MULTILINE)
+        if match:
+            try:
+                mean = float(match.group(1))
+                std = float(match.group(2))
+                return f"{mean:.4f}±{std:.4f}"
+            except ValueError:
+                pass
+        
+        # Fallback to patterns without std
+        patterns = [
+            r'Final estimate:\s*PPL\s*=\s*(\d+\.?\d*)',
+            r'Final perplexity:\s*(\d+\.?\d*)',
+            r'PPL\s*=\s*(\d+\.?\d*)',
+            r'PPL:\s*(\d+\.?\d*)',
+            r'perplexity:\s*(\d+\.?\d*)',
+            r'ppl\s*=\s*(\d+\.?\d*)',
+            r'Perplexity:\s*(\d+\.?\d*)',
+        ]
+        
+        for pattern in patterns:
+            match = re.search(pattern, output, re.IGNORECASE | re.MULTILINE)
+            if match:
+                try:
+                    return f"{float(match.group(1)):.4f}"
+                except ValueError:
+                    continue
+        
+        return None
+    
+    def quantize_embedding(self, embedding_type, output_suffix):
+        """
+        Quantize model with specific embedding type.
+        
+        Args:
+            embedding_type: Token embedding type (uppercase, e.g., 'Q6_K')
+            output_suffix: Output file suffix (lowercase, e.g., 'q6_k')
+        
+        Returns:
+            Path to quantized model or None if failed
+        """
+        # Construct output path
+        model_dir = self.model_path.parent
+        output_path = model_dir / f"ggml-model-i2_s-embed-{output_suffix}.gguf"
+        
+        # Check if file already exists
+        file_existed = output_path.exists()
+        
+        if file_existed:
+            print(f"ℹ️  Model already exists: {output_path.name}")
+            return output_path
+        
+        cmd = [
+            str(self.quantize_bin),
+            "--token-embedding-type", embedding_type,
+            str(self.model_path),
+            str(output_path),
+            "I2_S",
+            "1",
+            "1"
+        ]
+        
+        print(f"\n{'='*80}")
+        print(f"🔄 Quantizing with embedding type: {embedding_type}")
+        print(f"📥 Input:  {self.model_path.name}")
+        print(f"📤 Output: {output_path.name}")
+        print(f"💻 Command: {' '.join(cmd)}")
+        print(f"{'='*80}\n")
+        
+        start_time = time.time()
+        
+        try:
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                cwd=os.getcwd(),
+                timeout=600  # 10 minutes timeout
+            )
+            
+            duration = time.time() - start_time
+            
+            if result.returncode == 0:
+                file_size_mb = output_path.stat().st_size / (1024 * 1024)
+                print(f"✅ Quantization successful!")
+                print(f"   Duration: {duration:.2f}s")
+                print(f"   Size: {file_size_mb:.2f} MB")
+                
+                # Mark as newly created
+                self.created_models.add(output_path)
+                return output_path
+            else:
+                print(f"❌ Quantization failed with return code {result.returncode}")
+                print(f"Error: {result.stderr[:500]}")
+                return None
+                
+        except subprocess.TimeoutExpired:
+            print(f"❌ Quantization timeout (exceeded 10 minutes)")
+            return None
+        except Exception as e:
+            print(f"❌ Quantization error: {e}")
+            return None
+    
+    def cleanup_model(self, model_path):
+        """Delete model file if it was created during this session."""
+        if model_path in self.created_models:
+            try:
+                model_path.unlink()
+                print(f"🗑️  Deleted: {model_path.name}")
+                self.created_models.remove(model_path)
+            except Exception as e:
+                print(f"⚠️  Failed to delete {model_path.name}: {e}")
+        else:
+            print(f"ℹ️  Keeping existing file: {model_path.name}")
+    
+    def run_all_tests(self, threads=16, ctx_size=512):
+        """Run perplexity tests on all datasets."""
+        datasets = self.find_datasets()
+        
+        if not datasets:
+            print(f"\n❌ No datasets found in {self.data_dir}")
+            print(f"   Make sure each dataset directory has a test.txt file")
+            return
+        
+        # Quick mode: test all datasets but only first 4096 chars with smaller context
+        if self.quick_mode:
+            ctx_size = min(ctx_size, 128)  # Use smaller context in quick mode
+            print(f"\n⚡ QUICK TEST MODE ENABLED")
+            print(f"   - Testing all datasets with first 4096 characters only")
+            print(f"   - Using reduced context size: {ctx_size}")
+        
+        # Determine models to test
+        if self.test_embeddings:
+            print(f"\n{'='*80}")
+            print(f"🧪 EMBEDDING QUANTIZATION TEST MODE")
+            print(f"{'='*80}")
+            print(f"📦 Base model: {self.model_path.name}")
+            print(f"🔢 Embedding types to test: {len(self.embedding_types)}")
+            print(f"📊 Datasets: {len(datasets)}")
+            print(f"🧵 Threads: {threads}")
+            print(f"📏 Context size: {ctx_size}")
+            print(f"{'='*80}")
+            
+            total_start = time.time()
+            
+            # Test each embedding type
+            for i, (embedding_type, output_suffix) in enumerate(self.embedding_types, 1):
+                print(f"\n\n{'#'*80}")
+                print(f"[{i}/{len(self.embedding_types)}] Testing embedding type: {output_suffix} ({embedding_type})")
+                print(f"{'#'*80}")
+                
+                # Quantize model
+                quantized_model = self.quantize_embedding(embedding_type, output_suffix)
+                
+                if quantized_model is None:
+                    print(f"⚠️  Skipping tests for {output_suffix} due to quantization failure")
+                    continue
+                
+                # Test on all datasets
+                for j, dataset in enumerate(datasets, 1):
+                    print(f"\n[{j}/{len(datasets)}] Testing {dataset['name']} with {output_suffix}...")
+                    
+                    # Use quick dataset if in quick mode
+                    test_path = dataset['path']
+                    if self.quick_mode:
+                        test_path = self.create_quick_dataset(dataset['path'])
+                    
+                    result = self.run_perplexity_test(
+                        f"{dataset['name']}_embed-{output_suffix}",
+                        test_path,
+                        threads,
+                        ctx_size,
+                        model_override=quantized_model
+                    )
+                    self.results.append(result)
+                
+                # Cleanup model after testing
+                print(f"\n🧹 Cleaning up {output_suffix} model...")
+                self.cleanup_model(quantized_model)
+                
+                print(f"\n{'#'*80}")
+                print(f"✅ Completed {output_suffix}")
+                print(f"{'#'*80}")
+            
+            total_time = time.time() - total_start
+            
+        else:
+            # Regular single model test
+            print(f"\n{'='*80}")
+            print(f"🚀 PERPLEXITY TEST SESSION{' (QUICK MODE)' if self.quick_mode else ''}")
+            print(f"{'='*80}")
+            print(f"📦 Model: {self.model_path.name}")
+            print(f"📁 Model path: {self.model_path}")
+            print(f"📊 Datasets {'to test' if self.quick_mode else 'found'}: {len(datasets)}")
+            print(f"🧵 Threads: {threads}")
+            print(f"📏 Context size: {ctx_size}")
+            print(f"{'='*80}")
+            
+            total_start = time.time()
+            
+            # Run tests
+            for i, dataset in enumerate(datasets, 1):
+                print(f"\n\n[{i}/{len(datasets)}] Processing {dataset['name']}...")
+                
+                # Use quick dataset if in quick mode
+                test_path = dataset['path']
+                if self.quick_mode:
+                    test_path = self.create_quick_dataset(dataset['path'])
+                
+                result = self.run_perplexity_test(
+                    dataset['name'],
+                    test_path,
+                    threads,
+                    ctx_size
+                )
+                self.results.append(result)
+            
+            total_time = time.time() - total_start
+        
+        # Clean up temporary files
+        if self.quick_mode:
+            print(f"\n🧹 Cleaning up temporary files...")
+            self.cleanup_temp_files()
+        
+        # Save results
+        self.save_results()
+        
+        # Print summary
+        self.print_summary(total_time)
+    
+    def save_results(self):
+        """Save results to CSV file."""
+        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+        model_name = self.model_path.stem
+        
+        # Use custom CSV path if provided
+        if self.csv_output:
+            csv_file = self.csv_output
+            # Create parent directory if needed
+            csv_file.parent.mkdir(parents=True, exist_ok=True)
+        else:
+            csv_file = self.output_dir / f"ppl_{model_name}_{timestamp}.csv"
+        
+        print(f"\n💾 Saving results...")
+        
+        with open(csv_file, 'w', newline='') as f:
+            writer = csv.DictWriter(f, fieldnames=['dataset', 'perplexity', 'time_seconds', 'status'])
+            writer.writeheader()
+            for result in self.results:
+                writer.writerow({
+                    'dataset': result['dataset'],
+                    'perplexity': result['perplexity'] if result['perplexity'] is not None else 'N/A',
+                    'time_seconds': f"{result['time']:.2f}",
+                    'status': result['status']
+                })
+        
+        print(f"   ✅ CSV saved: {csv_file}")
+        
+        # Save detailed log
+        log_file = self.output_dir / f"ppl_{model_name}_{timestamp}.log"
+        with open(log_file, 'w') as f:
+            f.write(f"Perplexity Test Results\n")
+            f.write(f"{'='*80}\n")
+            f.write(f"Model: {self.model_path}\n")
+            f.write(f"Timestamp: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
+            f.write(f"{'='*80}\n\n")
+            
+            for result in self.results:
+                f.write(f"\n{'='*80}\n")
+                f.write(f"Dataset: {result['dataset']}\n")
+                f.write(f"Perplexity: {result['perplexity']}\n")
+                f.write(f"Time: {result['time']:.2f}s\n")
+                f.write(f"Status: {result['status']}\n")
+                f.write(f"\nOutput:\n{result['stdout']}\n")
+                if result['stderr']:
+                    f.write(f"\nErrors:\n{result['stderr']}\n")
+        
+        print(f"   ✅ Log saved: {log_file}")
+    
+    def print_summary(self, total_time):
+        """Print summary of all tests."""
+        print(f"\n\n{'='*80}")
+        print(f"📊 TEST SUMMARY")
+        print(f"{'='*80}\n")
+        
+        # Sort results by perplexity (lower is better)
+        successful = [r for r in self.results if r['perplexity'] is not None]
+        failed = [r for r in self.results if r['perplexity'] is None]
+        
+        if successful:
+            # Extract numeric value from "mean±std" format for sorting
+            def get_ppl_value(result):
+                ppl = result['perplexity']
+                if isinstance(ppl, str) and '±' in ppl:
+                    return float(ppl.split('±')[0])
+                elif isinstance(ppl, str):
+                    try:
+                        return float(ppl)
+                    except ValueError:
+                        return float('inf')
+                return ppl
+            
+            successful_sorted = sorted(successful, key=get_ppl_value)
+            
+            print(f"{'Dataset':<20} {'Perplexity':>20} {'Time (s)':>12} {'Status':<15}")
+            print(f"{'-'*80}")
+            
+            for result in successful_sorted:
+                ppl_str = str(result['perplexity']) if result['perplexity'] is not None else 'N/A'
+                print(f"{result['dataset']:<20} {ppl_str:>20} "
+                      f"{result['time']:>12.2f} {result['status']:<15}")
+            
+            best_ppl = str(successful_sorted[0]['perplexity'])
+            print(f"\n🏆 Best result: {successful_sorted[0]['dataset']} "
+                  f"(PPL: {best_ppl})")
+        
+        if failed:
+            print(f"\n❌ Failed tests ({len(failed)}):")
+            for result in failed:
+                print(f"   - {result['dataset']}: {result['status']}")
+        
+        print(f"\n{'='*80}")
+        print(f"✅ Completed: {len(successful)}/{len(self.results)}")
+        print(f"⏱️  Total time: {total_time:.2f}s ({total_time/60:.2f} min)")
+        print(f"📁 Results saved in: {self.output_dir}")
+        print(f"{'='*80}\n")
+
+
+def main():
+    parser = argparse.ArgumentParser(description='Test model perplexity on multiple datasets')
+    parser.add_argument('--model', '-m',
+                        required=True,
+                        help='Path to GGUF model file')
+    parser.add_argument('--data-dir', '-d',
+                        default='data',
+                        help='Directory containing dataset folders (default: data)')
+    parser.add_argument('--threads', '-t',
+                        type=int,
+                        default=16,
+                        help='Number of threads (default: 16)')
+    parser.add_argument('--ctx-size', '-c',
+                        type=int,
+                        default=512,
+                        help='Context size (default: 512)')
+    parser.add_argument('--output-dir', '-o',
+                        default='perplexity_results',
+                        help='Output directory for results (default: perplexity_results)')
+    parser.add_argument('--llama-perplexity',
+                        default='./build/bin/llama-perplexity',
+                        help='Path to llama-perplexity binary (default: ./build/bin/llama-perplexity)')
+    parser.add_argument('--quick', '-q',
+                        action='store_true',
+                        help='Quick test mode: test all datasets with first 4096 characters and reduced context size (128)')
+    parser.add_argument('--test-embeddings', '-e',
+                        action='store_true',
+                        help='Test different embedding quantization types (f32, f16, q8_0, q6_k, q5_0, q4_0, q3_k, tq2_0)')
+    parser.add_argument('--csv-output',
+                        help='Custom path for CSV output file (e.g., results/my_ppl_results.csv)')
+    parser.add_argument('--quantize-bin',
+                        default='./build/bin/llama-quantize',
+                        help='Path to llama-quantize binary (default: ./build/bin/llama-quantize)')
+    
+    args = parser.parse_args()
+    
+    try:
+        tester = PerplexityTester(
+            model_path=args.model,
+            llama_perplexity_bin=args.llama_perplexity,
+            data_dir=args.data_dir,
+            output_dir=args.output_dir,
+            quick_mode=args.quick,
+            quantize_bin=args.quantize_bin,
+            test_embeddings=args.test_embeddings,
+            csv_output=args.csv_output
+        )
+        
+        tester.run_all_tests(
+            threads=args.threads,
+            ctx_size=args.ctx_size
+        )
+        
+    except FileNotFoundError as e:
+        print(f"❌ Error: {e}")
+        return 1
+    except KeyboardInterrupt:
+        print("\n\n⚠️  Test interrupted by user")
+        return 1
+    except Exception as e:
+        print(f"\n❌ Unexpected error: {e}")
+        import traceback
+        traceback.print_exc()
+        return 1
+    
+    return 0
+
+
+if __name__ == "__main__":
+    exit(main())
@@ -0,0 +1,151 @@
+#!/bin/bash
+# Monitor power consumption for llama-bench with different thread configurations
+# Usage: ./monitor_power.sh <model_path> <output_csv> <pp_threads> <tg_threads>
+# Example: ./monitor_power.sh models/model.gguf results.csv "1,2,4,8" "1,2,4,8"
+
+set -e
+
+# Parse arguments
+if [ $# -ne 4 ]; then
+    echo "Usage: $0 <model_path> <output_csv> <pp_threads> <tg_threads>"
+    echo "Example: $0 models/model.gguf results.csv \"1,2,4,8\" \"1,2,4,8\""
+    exit 1
+fi
+
+MODEL_PATH="$1"
+OUTPUT_CSV="$2"
+PP_THREADS="$3"
+TG_THREADS="$4"
+
+TEMP_LOG="/tmp/power_monitor_$$.log"
+PID_FILE="/tmp/monitor_$$.pid"
+BENCH_OUTPUT="/tmp/bench_output_$$.txt"
+
+# Validate model exists
+if [ ! -f "$MODEL_PATH" ]; then
+    echo "Error: Model file not found: $MODEL_PATH"
+    exit 1
+fi
+
+# Create output directory if needed
+mkdir -p "$(dirname "$OUTPUT_CSV")"
+
+# Function to monitor CPU stats
+monitor_cpu() {
+    local log_file="$1"
+    echo "Timestamp,CPU_Usage(%),Avg_Freq(MHz)" > "$log_file"
+    while [ -f "$PID_FILE" ]; do
+        cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print 100-$8}')
+        avg_freq=$(grep "cpu MHz" /proc/cpuinfo | awk '{sum+=$4; count++} END {printf "%.0f", sum/count}')
+        timestamp=$(date +%s.%N)
+        echo "$timestamp,$cpu_usage,$avg_freq" >> "$log_file"
+        sleep 0.5
+    done
+}
+
+# Function to calculate average power
+calculate_power() {
+    local log_file="$1"
+    awk -F',' 'NR>1 {sum_cpu+=$2; count++} END {
+        if (count > 0) {
+            avg_cpu = sum_cpu/count
+            est_power = avg_cpu * 200 / 100
+            printf "%.2f", est_power
+        } else {
+            print "0"
+        }
+    }' "$log_file"
+}
+
+# Function to extract throughput from llama-bench output
+extract_throughput() {
+    local bench_output="$1"
+    local workload="$2"
+    grep "$workload" "$bench_output" | awk '{
+        # Extract mean from "mean ± std" format
+        for (i=1; i<=NF; i++) {
+            if ($(i+1) == "±") {
+                printf "%.2f", $i
+                exit
+            }
+        }
+    }'
+}
+
+# Function to run single benchmark
+run_benchmark() {
+    local workload="$1"  # "pp" or "tg"
+    local threads="$2"
+    local n_flag=""
+    
+    if [ "$workload" = "pp" ]; then
+        n_flag="-n 0"
+        workload_name="pp128"
+    else
+        n_flag="-n 128"
+        workload_name="tg128"
+    fi
+    
+    # Output progress to stderr (won't be captured in CSV)
+    echo "Testing $workload_name with $threads threads..." >&2
+    
+    # Start monitoring
+    touch "$PID_FILE"
+    monitor_cpu "$TEMP_LOG" &
+    local monitor_pid=$!
+    
+    # Run benchmark
+    ./build/bin/llama-bench -m "$MODEL_PATH" -p 128 $n_flag -t "$threads" -ngl 0 > "$BENCH_OUTPUT" 2>&1
+    
+    # Stop monitoring
+    rm -f "$PID_FILE"
+    wait $monitor_pid 2>/dev/null || true
+    
+    # Extract results
+    local throughput=$(extract_throughput "$BENCH_OUTPUT" "$workload_name")
+    local power=$(calculate_power "$TEMP_LOG")
+    
+    if [ -z "$throughput" ] || [ "$throughput" = "0" ]; then
+        echo "Warning: Failed to extract throughput for $workload_name, threads=$threads" >&2
+        throughput="0"
+    fi
+    
+    # Calculate J/t (Joules per token)
+    local j_per_token=$(awk -v p="$power" -v t="$throughput" 'BEGIN {
+        if (t > 0) printf "%.4f", p/t; else print "0"
+    }')
+    
+    # Output progress to stderr
+    echo "  Throughput: $throughput t/s, Power: $power W, Energy: $j_per_token J/t" >&2
+    
+    # Only output CSV line to stdout (this will be captured)
+    echo "$workload_name,$threads,$throughput,$power,$j_per_token"
+}
+
+# Initialize CSV
+echo "Workload,Threads,Throughput(t/s),Power(W),Energy(J/t)" > "$OUTPUT_CSV"
+
+# Test PP workloads
+IFS=',' read -ra PP_ARRAY <<< "$PP_THREADS"
+for threads in "${PP_ARRAY[@]}"; do
+    threads=$(echo "$threads" | xargs)  # trim whitespace
+    result=$(run_benchmark "pp" "$threads")
+    echo "$result" >> "$OUTPUT_CSV"
+done
+
+# Test TG workloads
+IFS=',' read -ra TG_ARRAY <<< "$TG_THREADS"
+for threads in "${TG_ARRAY[@]}"; do
+    threads=$(echo "$threads" | xargs)  # trim whitespace
+    result=$(run_benchmark "tg" "$threads")
+    echo "$result" >> "$OUTPUT_CSV"
+done
+
+# Cleanup
+rm -f "$TEMP_LOG" "$BENCH_OUTPUT" "$PID_FILE"
+
+echo ""
+echo "=== Benchmark Complete ==="
+echo "Results saved to: $OUTPUT_CSV"
+echo ""
+cat "$OUTPUT_CSV"
@@ -0,0 +1,362 @@
+#!/usr/bin/env python3
+"""
+GEMM Configuration Tuning Script
+This script automatically tunes ROW_BLOCK_SIZE, COL_BLOCK_SIZE, and PARALLEL_SIZE
+to find the optimal configuration for maximum throughput (t/s).
+"""
+
+import subprocess
+import os
+import re
+import csv
+import shutil
+from datetime import datetime
+from pathlib import Path
+import argparse
+
+
+class GemmTuner:
+    def __init__(self, config_path, model_path, threads=16):
+        self.config_path = Path(config_path)
+        self.model_path = model_path
+        self.threads = threads
+        self.backup_path = self.config_path.parent / f"gemm-config.h.backup_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
+        self.build_dir = Path("../build")
+        self.results = []
+        
+    def backup_config(self):
+        """Backup current configuration file"""
+        print(f"📦 Backing up current config to {self.backup_path}")
+        shutil.copy2(self.config_path, self.backup_path)
+        
+    def restore_config(self):
+        """Restore original configuration file"""
+        print(f"♻️  Restoring original config from {self.backup_path}")
+        shutil.copy2(self.backup_path, self.config_path)
+        
+    def generate_config(self, act_parallel, row_block_size, col_block_size, parallel_size):
+        """Generate new configuration file with simplified format"""
+        content = ""
+        
+        # Simplified configuration format
+        if act_parallel:
+            content += "#define ACT_PARALLEL\n"
+        
+        content += f"#define ROW_BLOCK_SIZE {row_block_size}\n"
+        content += f"#define COL_BLOCK_SIZE {col_block_size}\n"
+        content += f"#define PARALLEL_SIZE {parallel_size}\n"
+        
+        with open(self.config_path, 'w') as f:
+            f.write(content)
+    
+    def rebuild_project(self):
+        """Rebuild project"""
+        print("🔨 Rebuilding project...")
+        result = subprocess.run(
+            ["cmake", "--build", str(self.build_dir), "--target", "llama-bench"],
+            capture_output=True,
+            text=True,
+            cwd=os.getcwd()
+        )
+        if result.returncode != 0:
+            print(f"⚠️  Build warning/error: {result.stderr}")
+            return False
+        return True
+        
+    def run_benchmark(self):
+        """Run benchmark test"""
+        cmd = [
+            f"{self.build_dir}/bin/llama-bench",
+            "-m", self.model_path,
+            "-p", "128",
+            "-n", "0",
+            "-t", str(self.threads),
+            "-ngl", "0"
+        ]
+        
+        print(f"⚡ Running benchmark: {' '.join(cmd)}")
+        
+        result = subprocess.run(
+            cmd,
+            capture_output=True,
+            text=True,
+            cwd=os.getcwd(),
+            timeout=300  # 5分钟超时
+        )
+        
+        if result.returncode != 0:
+            print(f"❌ Benchmark failed: {result.stderr}")
+            return None
+            
+        return result.stdout
+        
+    def parse_throughput(self, output):
+        """Parse pp128 throughput from output"""
+        # 匹配 pp128: |         pp128 |       501.06 ± 11.37 |
+        pp_pattern = r'\|\s+pp128\s+\|\s+([\d.]+)\s+±\s+([\d.]+)\s+\|'
+        pp_match = re.search(pp_pattern, output)
+        
+        if pp_match:
+            pp_throughput = float(pp_match.group(1))
+            pp_std_dev = float(pp_match.group(2))
+            
+            return {
+                'pp_throughput': pp_throughput,
+                'pp_std_dev': pp_std_dev
+            }
+        
+        return None
+        
+    def test_configuration(self, act_parallel, row_block_size, col_block_size, parallel_size):
+        """Test single configuration"""
+        config_name = f"ACT_{'ON' if act_parallel else 'OFF'}_R{row_block_size}_C{col_block_size}_P{parallel_size}"
+        print(f"\n{'='*80}")
+        print(f"🧪 Testing configuration: {config_name}")
+        print(f"   ACT_PARALLEL: {act_parallel}")
+        print(f"   ROW_BLOCK_SIZE: {row_block_size}")
+        print(f"   COL_BLOCK_SIZE: {col_block_size}")
+        print(f"   PARALLEL_SIZE: {parallel_size}")
+        print(f"{'='*80}")
+        
+        # Generate configuration
+        self.generate_config(act_parallel, row_block_size, col_block_size, parallel_size)
+        
+        # Rebuild project
+        if not self.rebuild_project():
+            print("⚠️  Build failed, skipping this configuration")
+            return None
+        
+        # Run benchmark test
+        output = self.run_benchmark()
+        if output is None:
+            return None
+            
+        # Parse results
+        metrics = self.parse_throughput(output)
+        
+        if metrics is not None:
+            result = {
+                'act_parallel': act_parallel,
+                'row_block_size': row_block_size,
+                'col_block_size': col_block_size,
+                'parallel_size': parallel_size,
+                'config_name': config_name,
+                **metrics
+            }
+            self.results.append(result)
+            print(f"✅ PP128: {metrics['pp_throughput']:.2f} ± {metrics['pp_std_dev']:.2f} t/s")
+            return result
+        else:
+            print("❌ Failed to parse throughput")
+            return None
+            
+    def save_results(self, csv_path):
+        """Save results to CSV file"""
+        print(f"\n💾 Saving results to {csv_path}")
+        
+        with open(csv_path, 'w', newline='') as f:
+            writer = csv.DictWriter(f, fieldnames=[
+                'config_name', 'act_parallel', 'row_block_size', 
+                'col_block_size', 'parallel_size', 
+                'pp_throughput', 'pp_std_dev'
+            ])
+            writer.writeheader()
+            writer.writerows(self.results)
+            
+    def find_best_config(self):
+        """Find the best configuration with highest throughput"""
+        if not self.results:
+            print("❌ No valid results found")
+            return None
+            
+        best = max(self.results, key=lambda x: x['pp_throughput'])
+        return best
+        
+    def run_tuning(self, configurations, output_csv=None):
+        """Run test for all configurations"""
+        print(f"\n🚀 Starting tuning process with {len(configurations)} configurations")
+        print(f"📊 Model: {self.model_path}")
+        print(f"🧵 Threads: {self.threads}\n")
+        
+        # Backup configuration
+        self.backup_config()
+        
+        try:
+            # Test all configurations
+            for i, config in enumerate(configurations, 1):
+                print(f"\n[{i}/{len(configurations)}]")
+                self.test_configuration(**config)
+                
+            # Save results
+            if output_csv is None:
+                timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
+                csv_path = f"../stats/tuning_results_{timestamp}.csv"
+            else:
+                csv_path = output_csv
+            
+            # Ensure stats directory exists
+            os.makedirs(os.path.dirname(csv_path), exist_ok=True)
+            self.save_results(csv_path)
+            
+            # Find best configuration
+            best = self.find_best_config()
+            if best:
+                print(f"\n{'='*80}")
+                print(f"🏆 BEST CONFIGURATION FOUND!")
+                print(f"{'='*80}")
+                print(f"Configuration: {best['config_name']}")
+                print(f"ACT_PARALLEL: {best['act_parallel']}")
+                print(f"ROW_BLOCK_SIZE: {best['row_block_size']}")
+                print(f"COL_BLOCK_SIZE: {best['col_block_size']}")
+                print(f"PARALLEL_SIZE: {best['parallel_size']}")
+                print(f"PP128 Throughput: {best['pp_throughput']:.2f} ± {best['pp_std_dev']:.2f} t/s")
+                print(f"{'='*80}\n")
+                
+                # Show the configuration that will be written
+                print("Configuration to be written to gemm-config.h:")
+                print("-" * 80)
+                if best['act_parallel']:
+                    print("#define ACT_PARALLEL")
+                print(f"#define ROW_BLOCK_SIZE {best['row_block_size']}")
+                print(f"#define COL_BLOCK_SIZE {best['col_block_size']}")
+                print(f"#define PARALLEL_SIZE {best['parallel_size']}")
+                print("-" * 80)
+                
+                # Apply best configuration
+                apply = input("\nDo you want to apply this configuration to gemm-config.h? (y/n): ").strip().lower()
+                if apply == 'y':
+                    self.generate_config(
+                        best['act_parallel'],
+                        best['row_block_size'],
+                        best['col_block_size'],
+                        best['parallel_size']
+                    )
+                    self.rebuild_project()
+                    print("✅ Best configuration applied and project rebuilt!")
+                else:
+                    self.restore_config()
+                    print("✅ Original configuration restored")
+                
+                # Clean up backup file
+                if self.backup_path.exists():
+                    self.backup_path.unlink()
+                    print(f"🗑️  Removed backup file: {self.backup_path}")
+            
+        except KeyboardInterrupt:
+            print("\n⚠️  Tuning interrupted by user")
+            self.restore_config()
+            # Clean up backup file
+            if self.backup_path.exists():
+                self.backup_path.unlink()
+                print(f"🗑️  Removed backup file: {self.backup_path}")
+        except Exception as e:
+            print(f"\n❌ Error during tuning: {e}")
+            self.restore_config()
+            # Clean up backup file
+            if self.backup_path.exists():
+                self.backup_path.unlink()
+                print(f"🗑️  Removed backup file: {self.backup_path}")
+            raise
+
+
+def generate_configurations():
+    """Generate list of configurations to test"""
+    configurations = []
+    
+    act_parallel_options = [True]
+    
+    row_sizes = [2, 4, 8]#[2, 4, 8, 16, 32]
+    col_sizes = [32, 64]#[32, 64, 128, 256, 512, 1024]
+    parallelism_degree = [4]
+    
+    for act_parallel in act_parallel_options:
+        for row in row_sizes:
+            for col in col_sizes:
+                for parallel in parallelism_degree:
+                    # Add filtering conditions
+                    if act_parallel:
+                        # When ACT_PARALLEL=True, only calculate combinations with parallel < row
+                        if parallel > row:
+                            continue
+                    else:
+                        # When ACT_PARALLEL=False, only calculate combinations with parallel < col
+                        if parallel > col:
+                            continue
+                    
+                    configurations.append({
+                        'act_parallel': act_parallel,
+                        'row_block_size': row,
+                        'col_block_size': col,
+                        'parallel_size': parallel
+                    })
+    
+    return configurations
+
+
+def main():
+    parser = argparse.ArgumentParser(description='Tune GEMM configuration for optimal performance')
+    parser.add_argument('--config', default='../include/gemm-config.h', 
+                        help='Path to gemm-config.h file')
+    parser.add_argument('--model', default='../models/BitNet-b1.58-2B-4T/ggml-model-i2_s-embed-q6_k.gguf',
+                        help='Path to model file')
+    parser.add_argument('--threads', type=int, default=8,
+                        help='Number of threads to use')
+    parser.add_argument('--quick', action='store_true',
+                        help='Quick test with fewer configurations')
+    parser.add_argument('--custom', action='store_true',
+                        help='Manually specify configurations to test')
+    parser.add_argument('--output', type=str, default=None,
+                        help='Output CSV file path (default: stats/tuning_results_<timestamp>.csv)')
+    
+    args = parser.parse_args()
+    
+    tuner = GemmTuner(args.config, args.model, args.threads)
+    
+    if args.custom:
+        # Custom configuration mode
+        print("Custom configuration mode")
+        configurations = []
+        while True:
+            print("\nEnter configuration (or 'done' to finish):")
+            act = input("ACT_PARALLEL (y/n): ").strip().lower() == 'y'
+            if input == 'done':
+                break
+            row = int(input("ROW_BLOCK_SIZE: "))
+            col = int(input("COL_BLOCK_SIZE: "))
+            par = int(input("PARALLEL_SIZE: "))
+            configurations.append({
+                'act_parallel': act,
+                'row_block_size': row,
+                'col_block_size': col,
+                'parallel_size': par
+            })
+    elif args.quick:
+        # Quick test mode - test only a few key configurations
+        configurations = [
+            {'act_parallel': True, 'row_block_size': 4, 'col_block_size': 128, 'parallel_size': 4},
+            {'act_parallel': True, 'row_block_size': 8, 'col_block_size': 128, 'parallel_size': 4},
+            {'act_parallel': True, 'row_block_size': 4, 'col_block_size': 64, 'parallel_size': 4},
+            {'act_parallel': False, 'row_block_size': 32, 'col_block_size': 4, 'parallel_size': 4},
+            {'act_parallel': False, 'row_block_size': 16, 'col_block_size': 4, 'parallel_size': 4},
+        ]
+    else:
+        # Full test mode
+        configurations = generate_configurations()
+    
+    print(f"\n{'='*80}")
+    print(f"GEMM Configuration Tuner")
+    print(f"{'='*80}")
+    print(f"Total configurations to test: {len(configurations)}")
+    print(f"Estimated time: ~{len(configurations) * 0.5:.1f} minutes (assuming 30s per test)")
+    print(f"{'='*80}\n")
+    
+    proceed = input("Proceed with tuning? (y/n): ").strip().lower()
+    if proceed != 'y':
+        print("Tuning cancelled")
+        return
+    
+    tuner.run_tuning(configurations, output_csv=args.output)
+
+
+if __name__ == "__main__":
+    main()
Author	SHA1	Message	Date
XsquirrelC	0b341e582a	Add VibeASR.cpp release to News (#596 )	2026-07-27 13:52:05 +08:00
Xin Huang	16da220ae2	Revamp README with structured layout, News section, and Model Releases (#590 )	2026-07-21 18:39:52 +08:00
Xin Huang	186365a717	Update README for BitNet Embedding 0.6B/270M release (#589 ) * Update README for BitNet Embedding 0.6B/270M release - Add NEW section with performance highlights and quick start guide - Add embedding prefill performance comparison chart - Add header banner for embedding release - Update Official Models table with embedding models (x86 only) - Add What's New entry for HuggingFace release * Update README for BitNet Embedding 0.6B/270M release - Add NEW section with performance highlights and quick start guide - Add embedding prefill performance comparison chart - Add header banner for embedding release - Update Official Models table with embedding models (x86 only) - Add What's New entry for HuggingFace release	2026-07-20 16:49:27 +08:00
tsong-ms	6f22876973	Merge pull request #587 from isHuangXin/release-bitnet-embedding-0.6b-270m Add BitNet Embeddings 0.6B/270M I2_S conversion guide and update README	2026-07-17 11:43:50 +08:00
isHuangXin	b6a21be4e0	Add BitNet Embeddings 0.6B/270M I2_S conversion guide and update README - Add docs/bitnet-embeddings-i2s-guide.md with model overview, I2_S GGUF conversion details, accuracy verification, inference performance benchmarks, and quick start examples for both 0.6B (Qwen3) and 270M (Gemma3) models - Add embedding quantization chart (fig1_quant_per_task.png) - Remove old docs/bitnet-embeddings-gguf-conversion.md (replaced by new guide) - Add bitnet-embedding-0.6b and bitnet-embedding-270m to supported HF models in setup_env.py - Update README.md What's New section with link to the new guide	2026-07-16 17:29:40 +02:00
tsong-ms	e50e95e0ec	Merge pull request #584 from isHuangXin/release-bitnet-embedding-0.6b-270m Add F16 and I2_S GGUF conversion for bitnet-embeddings 0.6B and 270M models	2026-07-16 16:01:37 +08:00
isHuangXin	10dca87a5d	Enable LLAMA_BUILD_COMMON and LLAMA_BUILD_SERVER to build llama-cli	2026-07-16 09:14:04 +02:00
isHuangXin	85a507c67d	Fix vocab encoding by disabling add_special_tokens in tokenizer.encode	2026-07-16 05:32:29 +02:00
isHuangXin	403efc7b00	Fix inference pipeline: remove hardcoded batch size, enable tool builds, update Python requirement	2026-07-15 11:59:58 +02:00
isHuangXin	425fbbcb75	Update submodule to release-bitnet-embedding-0.6b-270m branch	2026-07-15 10:30:33 +02:00
isHuangXin	1f6a25aeeb	Update llama.cpp submodule to include SafetensorRemote utility classes	2026-07-15 08:47:54 +02:00
isHuangXin	ce530b0fd0	Add bitnet-embeddings-270m model adaptation with F16 and I2_S GGUF conversion - Add LLM_ARCH_GEMMA3 in llama.cpp for gemma3_text model type (embedding scaling, GELU, post-attn/post-FFN norms, GQA) - Add GGUF conversion support for Gemma3-based 270m models (SPM tokenizer, RMSNorm w+1 offset, arch-specific tensor mapping) - Add tokenizer hash for multilingual-e5-0.6b-260311 - Add conversion documentation	2026-07-15 08:29:16 +02:00
isHuangXin	3b04140a54	feat: add I2_S GGUF conversion for bitnet-b1.58-2B-4T and refactor T-MAC LUT path - Add quantize_to_i2_s() for direct ternary-to-I2_S packing in conversion script - Support offline-quantized models (uint8 packed weights + weight_scale) - Fix weight_quant double-quantization bug for offline-quantized models - Fix I2_S scale computation to use first nonzero absolute value - Add I2_S ftype mapping and BitNetForCausalLM registration - Refactor ggml-bitnet-lut T-MAC wrapper with proper mul_mat implementation - Update llama.cpp submodule with I2_S ftype and 2B model type support	2026-07-13 06:26:16 +02:00
isHuangXin	69bf64d373	Add bitnet-embeddings-0.6b model adaptation with F16 and I2_S GGUF conversion - Add GGUF conversion tool for bitnet-embeddings-0.6b (safetensors -> F16/I2_S GGUF) - Add Qwen3 architecture support in llama.cpp submodule with per-projection RMSNorm - Add I2_S ternary quantization (2-bit packed -1/0/+1) for lossless precision - Add f16 norm weight support for correct embedding inference - Guard bitnet-lut-kernels.h include with TL1/TL2 preprocessor checks - Update llama.cpp submodule to dev-bitnet-embedding-0.6b branch - Document F16 (from multilingual-e5-0.6b) and I2_S (from bitnet-embeddings-0.6b) conversion process	2026-07-12 04:10:30 +02:00
Yan Xia	01eb415772	Update demo link in README.md	2026-03-10 15:49:46 +08:00
tsong-ms	0fdaa16ae3	Merge pull request #421 from microsoft/fix/unsafe-deserialization-gpu-pipeline fix: add weights_only=True to torch.load in GPU inference pipeline	2026-03-09 20:20:21 +08:00
Ubuntu	eb60fc39cb	fix: add weights_only=True to torch.load in GPU inference pipeline Mitigate unsafe deserialization vulnerability (CWE-502) in the GPU inference pipeline. torch.load without weights_only=True allows arbitrary code execution via malicious pickle payloads in checkpoint files. Affected locations: - gpu/convert_checkpoint.py:37 (checkpoint conversion utility) - gpu/generate.py:67,69 (fp16 and int2 checkpoint loading) The utils/ scripts already applied this parameter correctly; this commit brings the GPU pipeline to the same safety standard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-09 12:09:19 +00:00
tsong-ms	8fd3412fbc	Merge pull request #406 from XsquirrelC/main [fix] convert pt to gguf	2026-02-03 13:30:42 +08:00
XSquirrelC	3987a503fd	[fix] convert pt to gguf	2026-02-03 05:24:24 +00:00
tsong-ms	ade47a535c	Merge pull request #380 from XsquirrelC/main [fix] change README link	2026-01-27 13:48:51 +08:00
XSquirrelC	77e136fb08	[fix] change README link	2026-01-27 03:39:11 +00:00
tsong-ms	cc3c9e4c80	Merge pull request #379 from XsquirrelC/main BitNet CPU Inference Optimization	2026-01-27 11:24:02 +08:00
XSquirrelC	1876a3e889	[merge] submodule llama.cpp	2026-01-27 03:09:32 +00:00
XSquirrelC	e8c8107dcf	[modify] some test picture and add power test script	2026-01-25 06:51:33 +00:00
XSquirrelC	7b2c52b9d5	[modify] some utils test script	2026-01-24 08:40:36 +00:00
XSquirrelC	2fed9af730	[fix] setup_env.py bug COMPILER_EXTRA_ARGS	2026-01-22 11:11:14 +00:00
deva100	7e6f0e14f1	[modify] update README; [feat] some test script in utils	2026-01-22 06:33:03 +00:00
deva100	7ea1f2601f	[modify] fine_tuning_result.png	2026-01-20 07:40:37 +00:00
deva100	b68802ff17	[fix] embed-quant q6_k; [modify] README update	2026-01-20 04:56:50 +00:00
deva100	35b1c28585	[fix] correct README	2026-01-15 03:44:50 +00:00
deva100	53ffe5e92b	[chore] update README	2026-01-15 03:37:16 +00:00
deva100	43da5e5f76	[fix] make demo_benchmark.sh more fast	2025-12-23 07:23:14 +00:00
deva100	41cc304868	[chore] add some automation bash script for BitNet Tech Report	2025-12-23 06:48:33 +00:00
deva100	112f853414	[feat] I2S kernels for weight & activation parallel on Intel & ARM machine; [feat] I2S GEMV & GEMM(llama.cpp); [feat] quantize activation & dequantize embedding(llama.cpp); [fix] compile bug: cannot define __ARM_FEATURE_DOTPROD(llama.cpp)	2025-11-19 07:35:05 +00:00
Junhui He	404980eeca	Merge pull request #290 from microsoft/gpu-readme-dev Update readme for gpu kernels	2025-06-03 14:14:20 +08:00
Junhui He	088e607b25	Merge pull request #280 from microsoft/fix-convert-dev Enable conversion from .safetensors checkpoints to gguf files	2025-06-03 13:59:47 +08:00
ZeonfaiHo	c1e9a9a237	Update readme for gpu kernels	2025-05-31 21:41:41 +08:00