How to Build a Spec Search Tool With Cohere Command on Azure

Ask a PM where the retry-policy decision lives and you'll get three different Confluence links, a Slack thread from February, and someone saying "I think it's in the PRD, the old one, not the new one." Cohere Command A on Azure fixes the retrieval half of that problem and the reasoning half at the same time: you can deploy Command A, Rerank 3.5 and Embed 4 straight from the Azure AI Foundry model catalog, wire them into a small pipeline, and give your team one endpoint that answers spec questions in plain English with a citation to the actual document.

This walks through building that pipeline end to end: deploying the three models to Azure AI Foundry Managed Compute, indexing your specs, and exposing the result as a search endpoint your team can hit from a browser or a Slack bot. By the end you'll have something a PM can type "what's our SLA for webhook retries" into and get back the right paragraph, not a link to go search themselves.

Deploy Command A and Rerank 3.5 on Azure AI Foundry Managed Compute

Command A, Rerank 3.5 and Embed 4 are all available as Managed Compute deployments in the Azure AI Foundry model catalog, which means they run on GPU capacity in your own Azure subscription instead of a shared pay-per-token endpoint.

1. In the Azure AI Foundry portal, create or pick a hub and project in a region that supports Cohere Managed Compute, such as East US or Sweden Central.
2. Open the model catalog and search for "Cohere". You need three model cards: Cohere Command A, Cohere Rerank v3.5, and Cohere Embed v4.
3. On each card, click Deploy, choose Managed Compute, accept the Marketplace terms, pick a GPU SKU your quota supports, and deploy.
4. Once each deployment is running, open its Consume tab and copy the endpoint URL and key. You get three separate endpoints, one per model.

export AZURE_ENDPOINT_EMBED="https://embed-v4-0-xxxx.eastus.models.ai.azure.com/"
export AZURE_API_KEY_EMBED="<your-embed-key>"
export AZURE_ENDPOINT_RERANK="https://cohere-rerank-v3-5-xxxx.eastus.models.ai.azure.com/"
export AZURE_API_KEY_RERANK="<your-rerank-key>"
export AZURE_ENDPOINT_CHAT="https://cohere-command-a-xxxx.eastus.models.ai.azure.com/"
export AZURE_API_KEY_CHAT="<your-command-a-key>"

Keep these three pairs somewhere your app can read them, like Azure Key Vault or a .env file for a first pass. Each deployment speaks the Cohere API, so the same cohere Python SDK works against all three, you just point it at a different base_url.

Embed your specs with Cohere Embed

Pull your specs and PRDs out of wherever they live (exporting from Confluence or Google Docs to markdown works fine), split them into chunks small enough to reason about individually, and embed each chunk with Embed 4.

import cohere

co_embed = cohere.ClientV2(
    api_key="AZURE_API_KEY_EMBED",
    base_url="AZURE_ENDPOINT_EMBED",
)

def chunk(text, size=1500, overlap=200):
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks

docs = []          # (doc_title, chunk_text) pairs
for path in spec_paths:
    text = open(path, encoding="utf-8").read()
    for c in chunk(text):
        docs.append((path, c))

doc_embs = co_embed.embed(
    model="model",
    texts=[c for _, c in docs],
    input_type="search_document",
    embedding_types=["float"],
).embeddings.float

Index and retrieve the closest chunks

Store the vectors in a lightweight index so you can pull the top candidates for any query. hnswlib is enough for a few thousand spec chunks; swap in Azure AI Search or a real vector store once the corpus grows.

import hnswlib

index = hnswlib.Index(space="ip", dim=len(doc_embs[0]))
index.init_index(max_elements=len(doc_embs), ef_construction=512, M=64)
index.add_items(doc_embs, list(range(len(doc_embs))))

def retrieve(query, k=20):
    query_emb = co_embed.embed(
        model="model",
        texts=[query],
        input_type="search_query",
        embedding_types=["float"],
    ).embeddings.float
    ids = index.knn_query(query_emb, k=k)[0][0]
    return [docs[i] for i in ids]

Pull more candidates than you actually need here. Embeddings alone are a blunt instrument, and reranking is what actually sorts out which chunk answers the question.

Rerank results for better spec search

Cohere Rerank 3.5 takes the query and the 20 candidate chunks and scores each one by how well it actually answers the question, catching cases the vector search ranked wrong.

co_rerank = cohere.ClientV2(
    api_key="AZURE_API_KEY_RERANK",
    base_url="AZURE_ENDPOINT_RERANK",
)

def rerank(query, candidates, top_n=5):
    results = co_rerank.rerank(
        model="model",
        query=query,
        documents=[c for _, c in candidates],
        top_n=top_n,
    )
    return [candidates[r.index] for r in results.results]

Rerank 3.5 has a 4,096-token context window per document, so keep chunks under that. If a spec section runs long, split it rather than counting on the reranker to read past the limit.

Generate grounded answers with Command A

Feed the top five reranked chunks to Command A with a system prompt that keeps it honest: answer from the context or say it doesn't know, and always name the source document.

co_chat = cohere.ClientV2(
    api_key="AZURE_API_KEY_CHAT",
    base_url="AZURE_ENDPOINT_CHAT",
)

def answer(query, top_chunks):
    context = "\n\n".join(f"[{title}]\n{text}" for title, text in top_chunks)
    messages = [
        {"role": "system", "content": (
            "Answer only using the context below. If the answer isn't there, "
            "say you couldn't find it. Cite the document name for every claim."
        )},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
    ]
    response = co_chat.chat(model="model", messages=messages)
    return response.message.content[0].text

Wrap it in a search endpoint

Put the three functions behind one small FastAPI route so anyone on the team can call it from a browser, a Slack slash command, or a CLI.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/search")
def search(q: Query):
    candidates = retrieve(q.question)
    top = rerank(q.question, candidates)
    return {"answer": answer(q.question, top)}

Run it with uvicorn app:app --host 0.0.0.0 --port 8000, put it behind your usual internal auth, and point a Slack bot or an internal tool at /search.

Gotchas

• Three endpoints, three keys. Command A, Rerank 3.5 and Embed 4 each deploy separately on Managed Compute, so you're managing three URLs and three keys, not one.
• model="model" isn't a typo. The Cohere SDK still wants a model string in every call, but on Azure Managed Compute the endpoint itself determines which model runs, so a placeholder string is correct.
• Managed Compute bills by the hour, not the token. Unlike Cohere's serverless pay-as-you-go offering on Azure, a Managed Compute deployment charges for GPU time whether or not anyone's searching. Scale down or delete deployments you're only testing.
• Rerank has a token limit per document. 4,096 tokens per chunk for Rerank 3.5. Long spec sections need splitting before they hit the reranker.
• Region matters. Managed Compute for these models is only available in a handful of Azure regions, so pick your hub's region before you build anything else.

Get the pipeline working with a handful of specs first, then point it at the whole doc set. Once a PM can ask a plain-English question and get the right paragraph back with a citation, the "where's that spec" Slack thread stops happening, which is exactly what matters when you're the one responsible for an AI system actually working in production.