# Estimated query progress

`API.get_data()` and `AsyncAPI.get_data()` display one estimated progress bar
from request execution through response decoding and final result assembly.
Authentication, metadata retrieval and planning happen before the bar starts.
Notebook calls retain the Jupyter widget, while scripts use a terminal bar.

The percentage is a prediction of work completed. It is not server-side SQL
progress, a byte-transfer counter or a completion-time guarantee. The SDK does
not make extra network requests to update it.

## Estimating a request's duration

For each planned request, let:

- $D$ be its nonnegative time span in days;
- $L$ be the number of requested locations;
- $S$ be the number of scalar metric rows in its planned request group;
- $C$ be the number of cycle-count metric rows;
- $M$ be the number of MPE metric rows.

Cycle counts have full names starting with `cc_`, case-insensitively. The MPE
category includes remaining metrics whose names start with `mpe` or whose
`data_group` is `mpe` or `mdl`. All other selected metric rows count toward $S$.
The counts describe rows in the planned request group, including metrics with
sparse data. Explicit selectors normally retain the matching overview rows.
The `all` shortcut uses a placeholder per location instead; these placeholders
are not expanded by the timing model and can lead to an underestimate.

The feature vector and expected duration are:

```{math}
\mathbf{x} = \left(1,\; \max(L-1,0),\;
S D\log\left(1+\frac{D}{30}\right),\; C D,\; M D\right)
```

```{math}
T = \boldsymbol{\beta}^{\mathsf{T}}\mathbf{x}.
```

The SDK supplies fixed calibrated coefficients $\boldsymbol{\beta}$, with a
positive intercept and nonnegative workload weights. The intercept accounts
for request overhead; the remaining terms represent location overhead and
metric-family work. The scalar term grows faster than linearly with long time
spans. This estimates uncached Parquet retrieval and decoding, rather than
compressed download size.

Splitting changes the set of requests, so the model estimates each request in
the actual plan separately. Cache state, response format, server load, data
availability and metric families can make actual durations differ from the
prediction. The estimate does not reserve execution capacity on the server.

## Progress during a running request

A request's clock starts after it acquires an execution slot. Queued requests
contribute no completed work. For elapsed time $e$ and expected duration $T$,
let $r=\max(0,e)/T$. The estimated fraction is:

```{math}
f(r) =
\begin{cases}
0.9r, & 0 \le r \le 1, \\
0.9 + 0.09\left(1-\exp(1-r)\right), & r > 1.
\end{cases}
```

At the expected duration, a running request has reached 90% of its own work.
An overrun continues to approach 99% without reporting completion. Successful
retrieval and decoding set that request's fraction to 1 immediately. A cache
hit can therefore finish well before its predicted duration; it is never
artificially delayed to fit the estimate.

A background thread refreshes the display at approximately 0.2-second
intervals while the request waits. Request completions also trigger refreshes.

## Combining split requests and final assembly

Each request $i$ is weighted by its expected duration $T_i$. Let $q_i$ be 0
while queued, $f(r_i)$ while running, and 1 after successful retrieval and
decoding. Before final assembly finishes, the displayed percentage is:

```{math}
P(t) = \max\left(P(t^-),\;
99\frac{\sum_i T_i q_i(t)}{\sum_i T_i}\right).
```

Here $P(t^-)$ is the previously displayed percentage. Taking the maximum keeps
the display monotonic. Weighting by duration prevents a small request and a
large request from contributing the same amount merely because both count as
one request.

The factor 99 reserves the final percentage point for assembly and output
conversion. When all requests have returned, the label changes to
`Assembling result` and the bar stays at 99%. Only a successful exit from the
query's progress context sets it to 100%. Failure, cancellation and keyboard
interrupts close the bar without promoting it to 100%.

## Approximate time remaining

The remaining-time estimate accounts for bounded concurrency. For an active
request, its predicted remaining duration is $\max(0,T_i-e_i)$; for a queued
request it is $T_i$. Completed requests are omitted.

To schedule these durations across $K$ slots, initialize each slot's accumulated
work $a_k$ to zero. For each remaining duration $d_i$, choose the earliest
available slot and update it:

```{math}
k^* = \operatorname*{arg\,min}_{1\le k\le K} a_k,
\qquad a_{k^*} \leftarrow a_{k^*}+d_i.
```

The predicted remaining wall time is $\max_k a_k$, rounded up to seconds for
display. It is not simply the sum of request durations, because requests may
run concurrently. The synchronous client's thread limit can reduce $K$ below
the request plan's concurrency limit.

When any active request exceeds its estimate, `longer than estimate` replaces
the remaining-time label. The request continues under its normal timeout;
the bar does not cancel it or invent a new completion deadline.

## Interpreting the display

The completed-request count reports observed completions, while the percentage
also includes estimated progress on running requests. A high percentage can
still be followed by a wait for a slow response or output conversion. Local
cache hits still require decoding and assembly.

For request-size, format and cache controls, see the
[retrieval guide](../how-to/core-api.md). For the surrounding request pipeline,
see [SDK architecture](architecture.md).