SDK architecture

The SDK exposes synchronous and asynchronous DataSignals clients over shared validation, request planning, response decoding and output conversion. This page maps the current implementation for developers maintaining or integrating the package.

Mermaid diagram source

Public clients and shared state

Both API and AsyncAPI are defined in api_24sea.datasignals.core and inherit from DataSignalsClient. They are separate facades: AsyncAPI does not inherit from API. The pandas DataSignals accessor delegates to API and remains synchronous.

DataSignalsClient, in datasignals/base_client.py, provides common authentication and metrics-overview behavior. Each client keeps its own state. Its parent, AuthABC in abc.py, supplies credential handling and lazy authentication. The common base has no data-retrieval endpoint methods.

authenticate() is synchronous on both facades. For asynchronous metadata retrieval, use await api.get_metrics_overview(); accessing the inherited metrics_overview property follows the synchronous authentication path.

Data retrieval

A get_data() call follows these steps:

1. Authenticate as needed and obtain the account’s metrics overview.

2. Validate the arguments with schemas.GetData and select authorized overview rows using the requested sites, locations and metric selectors.

3. Build a frozen DataRequestPlan. The planner packs locations by site and divides oversized requests into contiguous, half-open time windows.

4. Start QueryProgress and execute the plan with bounded concurrency. The synchronous executor uses HTTPX and a thread pool; the asynchronous executor uses an AsyncClient and available execution slots. Each reuses a client within the call.

5. Check the optional local response cache, fetch misses, and decode each response. Parquet is preferred; JSON is supported, including fallback when the server rejects Parquet negotiation with HTTP 406 or 415. HTTPX handles HTTP content decompression.

6. Assemble request results and apply the selected DataFrame, records or star schema output conversion. Successful completion of this stage finishes the progress bar.

The numeric-data budget used by planning is an estimate based on metric counts, metric families and the requested time span. It is not a limit on compressed response size or process memory. Execution timeouts apply to individual requests after they acquire a slot; a split query can take longer overall.

Progress is an estimate driven by request metadata and elapsed time. It does not poll a server task or inspect database execution. Its equations and completion rules are described in Estimated query progress.

Module responsibilities

All paths in the table are relative to api_24sea/.

Module	Responsibility
datasignals/__init__.py	Register the synchronous pandas accessor and delegate calls to API.
datasignals/core.py	Define both public clients; coordinate endpoint calls and output conversions.
datasignals/base_client.py	Share authentication, URL normalization and metrics-overview state.
abc.py	Provide credential handling and lazy-authentication behavior.
datasignals/schemas.py	Validate retrieval inputs and select or group overview rows.
datasignals/planning.py	Define DataRequest and DataRequestPlan; plan location and time splits.
datasignals/execution.py	Execute blocking data requests and validated writes.
datasignals/async_execution.py	Execute asynchronous data requests and validated writes.
datasignals/responses.py	Negotiate formats, decode Parquet/JSON and assemble response tables.
datasignals/progress.py	Estimate request work, schedule remaining work and manage one progress bar.
datasignals/writes.py	Validate write payloads and create immutable write-request descriptions.
cache.py	Store optional bounded, expiring response bytes on disk.
utils.py	Supply shared HTTP, cache integration, sizing and DataFrame utilities.
datasignals/fatigue.py	Provide optional cycle-count conversion through the fatigue accessor.

The diagram describes data retrieval. Other endpoint methods remain coordinated by their facade and use the shared helpers relevant to their contracts.

Caching and writes

The API server’s data cache is automatic and shared between authorized users. It checks permissions before looking up cached values. Entries describe coverage by project, location, canonical metric and time interval rather than one user’s exact request. The server projects cached coverage onto the current request, reads the uncovered portion from SQL and combines both sources. Selections using data groups and statistics first resolve to canonical metric names, so equivalent selections and response formats can share values. Application writes invalidate the affected project/time partitions.

QueryCache is optional and disabled by default. It stores successful response bytes, scoped by credentials and request identity. A hit still requires response decoding and output assembly. Authentication and metadata retrieval remain separate from this cache. This local cache defaults to a 24-hour snapshot lifetime and is separate from the server’s shared coverage cache. Start with server caching; cache=True opts into local reuse when needed. See retrieval controls for details.

put_data() and patch_data() use the shared write validator and their facade’s executor. They do not use the retrieval cache. Their row identity, merge behavior and permissions are documented in Write data.