Update benchmarks README with K8s CI documentation

Author: Shekharrajak

benchmarks/README.md

@@ -17,10 +17,102 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# Running Comet Benchmarks in Microk8s
+# Comet Benchmarks
 
-This guide explains how to run benchmarks derived from TPC-H and TPC-DS in Apache DataFusion Comet deployed in a
-local Microk8s cluster.
+This guide explains how to run benchmarks derived from TPC-H and TPC-DS in Apache DataFusion Comet.
+
+## Table of Contents
+
+- [GitHub CI Benchmarks (Kind)](#github-ci-benchmarks-kind)
+- [Local Development (Kind)](#local-development-kind)
+- [Microk8s Deployment](#running-comet-benchmarks-in-microk8s)
+
+---
+
+## GitHub CI Benchmarks (Kind)
+
+The project includes automated benchmark CI that runs on every PR affecting Rust (`native/**/*.rs`) or Scala/Java (`spark/**/*.scala`, `spark/**/*.java`) code.
+
+### What the CI Does
+
+1. Creates a Kind Kubernetes cluster (1 control-plane + 2 workers)
+2. Installs Spark Operator via Helm
+3. Builds Comet from source
+4. Generates TPC-H SF=1 data (~1GB)
+5. Runs TPC-H Q1 with Spark baseline
+6. Runs TPC-H Q1 with Comet enabled
+7. **Validates that Comet achieves ≥1.1x speedup (10% improvement)**
+
+### Manual Trigger
+
+You can manually trigger the benchmark CI from GitHub Actions with custom parameters:
+
+- **scale_factor**: TPC-H scale factor (default: 1)
+- **query**: TPC-H query (q1, q6, q14, simple)
+- **min_speedup**: Minimum required speedup (default: 1.1)
+
+---
+
+## Local Development (Kind)
+
+Run benchmarks locally using Kind (Kubernetes in Docker).
+
+### Prerequisites
+
+```bash
+# Install Kind, kubectl, and Helm
+brew install kind kubectl helm  # macOS
+# Or see: https://kind.sigs.k8s.io/docs/user/quick-start/
+```
+
+### Quick Start
+
+```bash
+# 1. Setup Kind cluster with Spark Operator
+./hack/k8s-benchmark-setup.sh
+
+# 2. Build Comet
+make release PROFILES="-Pspark-3.5 -Pscala-2.12"
+
+# 3. Build benchmark Docker image
+docker build -t comet-bench:local -f benchmarks/Dockerfile.k8s .
+kind load docker-image comet-bench:local --name comet-bench
+
+# 4. Generate TPC-H data
+./benchmarks/scripts/generate-tpch-data.sh 1 /tmp/comet-bench-data/tpch
+
+# 5. Run Spark baseline
+./benchmarks/scripts/run-k8s-benchmark.sh spark q1
+
+# 6. Run Comet benchmark
+./benchmarks/scripts/run-k8s-benchmark.sh comet q1
+
+# 7. Compare results
+python3 benchmarks/scripts/compare-results.py \
+    --spark /tmp/comet-bench-results/spark_q1_result.json \
+    --comet /tmp/comet-bench-results/comet_q1_result.json \
+    --min-speedup 1.1
+
+# 8. Cleanup
+./hack/k8s-benchmark-setup.sh --delete
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `COMET_BENCH_CLUSTER` | `comet-bench` | Kind cluster name |
+| `COMET_BENCH_NAMESPACE` | `comet-bench` | Kubernetes namespace |
+| `COMET_DOCKER_IMAGE` | `comet-bench:local` | Docker image for benchmarks |
+| `DRIVER_MEMORY` | `2g` | Spark driver memory |
+| `EXECUTOR_MEMORY` | `2g` | Spark executor memory |
+| `EXECUTOR_INSTANCES` | `2` | Number of Spark executors |
+
+---
+
+## Running Comet Benchmarks in Microk8s
+
+This section explains how to run benchmarks in a local Microk8s cluster
 
 ## Use Microk8s locally