Stream a long-running deep review

Outcome: a deep review that shows progress in the UI instead of a spinner that times out.

The problem

deep_review runs an agentic pass over many papers and takes minutes. A synchronous HTTP request is the wrong shape for it.

Three correct patterns

Pattern	Use when	How
SSE streaming	a human is watching	`stream: true` on chat messages
Webhook	a pipeline consumes it	subscribe to `search.completed`
Poll + progress	simplest to build	`GET /v1/jobs/{id}`, render `progress`

Walkthrough

Webhooks — the default for servers

cURL

curl https://api.scispace.com/v1/webhook_endpoints \
  -H "Authorization: Bearer $SCISPACE_API_KEY" \
  -H "SciSpace-Version: 2026-08-01" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/hooks/scispace",
    "enabled_events": ["search.succeeded", "search.failed"]
  }'

Verify the signature before trusting the body, then look the search up by id — never take results straight from the payload. See webhook-endpoints.

Streaming — when a human is watching

Deep reviews take minutes. Streaming lets you show progress instead of a spinner.

Python

with client.searches.stream(
    query="How does climate change affect biodiversity?",
    depth="deep_review",
) as stream:
    for event in stream:
        if event.type == "search.progress":
            print(f"{event.stage}: {event.papers_processed}/{event.papers_total}")
        elif event.type == "answer.delta":
            print(event.delta, end="", flush=True)
    search = stream.get_final_search()

Polling — the fallback

Fetch the resource; there is no separate status endpoint. Retrieves are free, but each one counts against your rate limit, so back off.

Elapsed	Interval
0–30 s	2 s
30 s–2 min	5 s
2–5 min	15 s
5 min+	30 s
10 min	give up, mark stalled

That schedule costs about 40 requests over a full deep_review, against a p95 of 240 s.

Python

import time

def wait_for(search_id, deadline_s=600):
    started = time.monotonic()
    while True:
        elapsed = time.monotonic() - started
        if elapsed > deadline_s:
            raise TimeoutError(search_id)

        search = client.searches.retrieve(search_id)
        if search.status in ("succeeded", "failed", "canceled"):
            return search

        if search.progress:
            print(f"{search.progress.stage} "
                  f"{search.progress.papers_processed}/{search.progress.papers_total}")

        time.sleep(2 if elapsed < 30 else 5 if elapsed < 120 else 15 if elapsed < 300 else 30)

While status is running, the object carries a progress block you can render directly:

json

{
  "object": "search",
  "id": "srch_9dm2pq4x1a",
  "status": "running",
  "depth": "deep_review",
  "progress": {
    "stage": "reading",
    "stage_index": 2,
    "stage_count": 4,
    "papers_processed": 84,
    "papers_total": 200
  }
}

stage runs retrieving → screening → reading → synthesizing. Drive a progress bar from papers_processed / papers_total, and the label from stage — the stages are not equal in duration, so weighting by stage_index alone reads as stalled during reading.

Handling the hard parts

Proxy and load-balancer idle timeouts kill long SSE connections — send heartbeats and reconnect
Persist the resource ID before you start waiting, so a crash does not lose a paid job
On reconnect, fetch the resource rather than restarting the search

depth · jobs · chats · webhook

Last updated 2026-08-10