Estimated query progress#

API.get_data() and AsyncAPI.get_data() display one estimated progress bar from request execution through response decoding and final result assembly. Authentication, metadata retrieval and planning happen before the bar starts. Notebook calls retain the Jupyter widget, while scripts use a terminal bar.

The percentage is a prediction of work completed. It is not server-side SQL progress, a byte-transfer counter or a completion-time guarantee. The SDK does not make extra network requests to update it.

Estimating a request’s duration#

For each planned request, let:

  • \(D\) be its nonnegative time span in days;

  • \(L\) be the number of requested locations;

  • \(S\) be the number of scalar metric rows in its planned request group;

  • \(C\) be the number of cycle-count metric rows;

  • \(M\) be the number of MPE metric rows.

Cycle counts have full names starting with cc_, case-insensitively. The MPE category includes remaining metrics whose names start with mpe or whose data_group is mpe or mdl. All other selected metric rows count toward \(S\). The counts describe rows in the planned request group, including metrics with sparse data. Explicit selectors normally retain the matching overview rows. The all shortcut uses a placeholder per location instead; these placeholders are not expanded by the timing model and can lead to an underestimate.

The feature vector and expected duration are:

\[\mathbf{x} = \left(1,\; \max(L-1,0),\; S D\log\left(1+\frac{D}{30}\right),\; C D,\; M D\right)\]
\[T = \boldsymbol{\beta}^{\mathsf{T}}\mathbf{x}.\]

The SDK supplies fixed calibrated coefficients \(\boldsymbol{\beta}\), with a positive intercept and nonnegative workload weights. The intercept accounts for request overhead; the remaining terms represent location overhead and metric-family work. The scalar term grows faster than linearly with long time spans. This estimates uncached Parquet retrieval and decoding, rather than compressed download size.

Splitting changes the set of requests, so the model estimates each request in the actual plan separately. Cache state, response format, server load, data availability and metric families can make actual durations differ from the prediction. The estimate does not reserve execution capacity on the server.

Progress during a running request#

A request’s clock starts after it acquires an execution slot. Queued requests contribute no completed work. For elapsed time \(e\) and expected duration \(T\), let \(r=\max(0,e)/T\). The estimated fraction is:

\[\begin{split}f(r) = \begin{cases} 0.9r, & 0 \le r \le 1, \\ 0.9 + 0.09\left(1-\exp(1-r)\right), & r > 1. \end{cases}\end{split}\]

At the expected duration, a running request has reached 90% of its own work. An overrun continues to approach 99% without reporting completion. Successful retrieval and decoding set that request’s fraction to 1 immediately. A cache hit can therefore finish well before its predicted duration; it is never artificially delayed to fit the estimate.

A background thread refreshes the display at approximately 0.2-second intervals while the request waits. Request completions also trigger refreshes.

Combining split requests and final assembly#

Each request \(i\) is weighted by its expected duration \(T_i\). Let \(q_i\) be 0 while queued, \(f(r_i)\) while running, and 1 after successful retrieval and decoding. Before final assembly finishes, the displayed percentage is:

\[P(t) = \max\left(P(t^-),\; 99\frac{\sum_i T_i q_i(t)}{\sum_i T_i}\right).\]

Here \(P(t^-)\) is the previously displayed percentage. Taking the maximum keeps the display monotonic. Weighting by duration prevents a small request and a large request from contributing the same amount merely because both count as one request.

The factor 99 reserves the final percentage point for assembly and output conversion. When all requests have returned, the label changes to Assembling result and the bar stays at 99%. Only a successful exit from the query’s progress context sets it to 100%. Failure, cancellation and keyboard interrupts close the bar without promoting it to 100%.

Approximate time remaining#

The remaining-time estimate accounts for bounded concurrency. For an active request, its predicted remaining duration is \(\max(0,T_i-e_i)\); for a queued request it is \(T_i\). Completed requests are omitted.

To schedule these durations across \(K\) slots, initialize each slot’s accumulated work \(a_k\) to zero. For each remaining duration \(d_i\), choose the earliest available slot and update it:

\[k^* = \operatorname*{arg\,min}_{1\le k\le K} a_k, \qquad a_{k^*} \leftarrow a_{k^*}+d_i.\]

The predicted remaining wall time is \(\max_k a_k\), rounded up to seconds for display. It is not simply the sum of request durations, because requests may run concurrently. The synchronous client’s thread limit can reduce \(K\) below the request plan’s concurrency limit.

When any active request exceeds its estimate, longer than estimate replaces the remaining-time label. The request continues under its normal timeout; the bar does not cancel it or invent a new completion deadline.

Interpreting the display#

The completed-request count reports observed completions, while the percentage also includes estimated progress on running requests. A high percentage can still be followed by a wait for a slow response or output conversion. Local cache hits still require decoding and assembly.

For request-size, format and cache controls, see the retrieval guide. For the surrounding request pipeline, see SDK architecture.