Core DataSignals API#

The core DataSignals API module is designed to provide a more direct interaction with the 24SEA API and is the base for the DataSignals module.

Installation#

The package supports Python 3.8 and above. To install it, run the following command in your terminal:

pip install api_24sea

Usage#

The following example shows the classical usage of the core DataSignals API module, which can be integrated within other standalone classes or functions.

  • The first step is to import the package and the necessary libraries.

  • Then, the environment variables are loaded from a .env file.

  • After that, the package is initialized and the user is authenticated with the API.

  • Finally, the user can get data from the API.

The first two steps are the same as in the DataSignals module and will not be repeated here.

Note

The core DataSignals API module contains two main classes with the same functionality, but different logic: API and AsyncAPI. The first one is a synchronous class and the second one is an asynchronous class. The user can choose which one to use depending on the needs. The API class is the default class and is used in the following examples. The AsyncAPI class is used in the Asynchronous API Usage section.

Response compression#

Data retrieval prefers Parquet by default. PyArrow is installed with the SDK, and get_data() decodes Parquet into the same pandas table or dictionary outputs. It removes schema-only null columns and metadata-only rows, matching the JSON endpoint. Nested metric values remain Python lists and dictionaries. Servers returning JSON also work; a server rejecting Parquet negotiation with HTTP 406 or 415 is retried with JSON. Other errors remain errors.

Use response_format="json" to select JSON explicitly. A custom Accept header takes precedence over response_format.

API and AsyncAPI automatically advertise gzip support through HTTPX. The API can return Content-Encoding: gzip, and HTTPX decompresses the response before the SDK parses its JSON and builds the requested table. Your get_data() calls and returned values stay the same, with no extra packages or archive extraction required. Servers returning plain JSON also work.

Compression reduces bytes transferred over the network; it does not shorten the server’s database queries. A large decoded response can therefore be much larger than the compressed download.

Request splitting and concurrency#

The default estimated numeric-data budget is 128 MiB per HTTP request. Locations within the same site are packed into this budget. A location that exceeds it is split into contiguous, half-open time windows. Estimates account for the selected metrics at each location, including the all expansion and additional weights for vector-valued data. They assume ten-minute sampling; the budget does not represent compressed download size or process memory.

Set max_request_mb=100 for smaller requests; raise the budget only after checking the server’s capacity. Both core clients use the same plan and bounded concurrency; the synchronous threads argument can further reduce the worker count. Connections are reused within a query. Asynchronous requests start as slots become available, and results retain query order.

get_data() defaults to a 3,600-second timeout in both clients. Each scheduled request has this timeout after acquiring its concurrency slot; queued requests do not consume it. A query containing multiple requests can take longer overall.

Estimated query progress#

Both API.get_data() and AsyncAPI.get_data() show one progress bar for the whole call, including split requests and final table assembly. Jupyter keeps its notebook widget; Python scripts use the terminal bar.

The bar advances during running requests using a timing estimate based on metric family, date range, location count and the request plan. Completed requests update it using their expected work. The percentage and approximate remaining time are estimates, not server-side SQL progress. A request that takes longer than predicted displays longer than estimate; 100% means the result has finished assembling. Failed or canceled calls do not show 100%. Cache hits return as soon as they are ready, with no artificial delay.

See Estimated query progress for the timing model, progress equations and limits of the estimate.

Server-side caching#

Server-side caching is automatic for data retrieval. Each request checks your current permissions first, then reuses any valid cached data you can access, including data previously requested by another authorized user.

The server can reuse part of a previous request. If you first fetch one month and then extend the time window, add locations or add metrics, it reads the overlapping data from cache and queries SQL only for the missing coverage. The result combines both sources. Equivalent explicit-metric and data-group/statistic selections can share this cache, as can JSON and Parquet responses.

Start with this automatic cache; most callers can leave local caching disabled. Pass force_cache_miss=True to bypass cached reads and refresh the result:

data = await api.get_data(
    sites, locations, metrics, start_timestamp, end_timestamp,
    force_cache_miss=True,
)

Optional local query caching#

Local caching is disabled by default. Add cache=True to reuse successful responses between calls and Python sessions without downloading them again. Both API.get_data() and AsyncAPI.get_data() accept this option:

data = await api.get_data(
    sites, locations, metrics, start_timestamp, end_timestamp,
    cache=True,
)

The default directory is $XDG_CACHE_HOME/api-24sea/queries, or ~/.cache/api-24sea/queries when XDG is unset. Entries expire after 24 hours, and retained entries are bounded to 1 GiB, evicting the oldest first. Files contain query results; their filenames and metadata contain no credentials. Queries are isolated by credentials, endpoint, HTTP method, parameters, and headers. Output options such as as_dict are applied after reading the response, so they can reuse the same cached download. Sync and async clients share entries.

For a custom directory, lifetime, or size:

from pathlib import Path
from api_24sea import QueryCache

query_cache = QueryCache(
    Path.home() / ".cache" / "my-api24sea-queries",
    ttl=3600,
    max_bytes=2 * 1024**3,
)
data = await api.get_data(
    sites, locations, metrics, start_timestamp, end_timestamp,
    cache=query_cache,
)

Cached results are snapshots. force_cache_miss=True bypasses local and server cache reads and replaces the local entry after a successful response. Use query_cache.clear() to remove only local entries, retaining unrelated files. Corrupt or expired entries are misses; unsuccessful queries are never stored. An unwritable cache does not prevent returning a successful remote result.

Authentication and the metrics overview still run when a new client needs them. Local caching covers data retrieval; it does not cache writes or provide an offline authentication mode.

Authenticating with the API (optional)#

The authentication step is performed automatically if the environment variables 24SEA_API_USERNAME and 24SEA_API_PASSWORD are loaded. The user can also authenticate manually by calling the authenticate method from the API.

This step is optional, since the API is able to authenticate lazily if the environment variables are loaded, or the user can authenticate manually.

# %%
# Package Imports
from api_24sea.datasignals.core import API
# %%
# **Authentication**
api = API()
api.authenticate("some_other_username", "some_other_password")

Checking the available metrics#

overview = api.metrics_overview
assert overview is not None
overview[["metric", "statistic", "short_hand", "site", "location"]]

The overview lists the metrics and time windows your account may access. For the pandas accessor, use df.datasignals.metrics_overview; with AsyncAPI, use overview = await api.get_metrics_overview().

Select metrics by name or regular expression#

metrics filters the full names in the overview’s metric column. It does not search short_hand, description, or data_group separately. The SDK selects matching overview rows locally, then sends their full metric names to the server. sites and locations further restrict those rows; they match names or IDs case-insensitively, rather than regular expressions or partial location names.

A typical full metric name has this structure:

<statistic>_<site_id>_<location_id>_<short_hand>
mean_WF_A01_windspeed

Here statistic is mean, site_id is WF, location_id is A01, and short_hand is windspeed. Short names can themselves contain underscores, such as TP_INC_LAT015_DEG240_X_nr1. The IDs can differ from the human-readable site and location labels. Use the overview’s metric value as the authoritative full name instead of constructing names from display labels.

The matching rules are shared by API, AsyncAPI, and the pandas accessor:

  • Matching is case-insensitive and searches anywhere in the full name unless you anchor the expression.

  • Every space and hyphen (-) in a selector becomes .*, meaning zero or more characters. Thus mean windspeed becomes mean.*windspeed.

  • An underscore (_) stays a literal underscore. It is neither escaped nor expanded, and needs no backslash. mean_windspeed therefore does not match mean_WF_A01_windspeed: the site and location IDs lie between them.

  • A list combines its expressions with | (OR). For example, ["mean windspeed", "mean power"] selects either family.

Selector

Meaning

"windspeed"

Any full name containing windspeed, including different statistics.

"mean windspeed" or "mean-windspeed"

Names containing mean, followed later by windspeed.

r"^mean_.*_windspeed$"

Names starting with mean_ and ending with _windspeed.

r"^mean_WF_A01_windspeed$"

One complete name, matched case-insensitively.

r"^CC_.*" / r"^mpe.*"

Full names starting with CC_ / mpe; availability depends on your account.

In these expressions, ^ anchors the start, $ anchors the end, . matches any character, and * repeats the preceding expression zero or more times. (?:a|b) groups alternatives without capturing them. For example, r"^mean_.*_(?:windspeed|power)$" selects mean windspeed or power metrics. Use raw Python strings (r"...") when an expression needs backslashes, such as r"\d" for a digit. Other regex metacharacters remain active; for example, a literal dot needs r"\.". The space/hyphen replacement happens even inside regex syntax, so use r"\d" or [0123456789] instead of a hyphenated range such as [0-9].

Preview the selection before starting a large query. This example uses the Sample site’s full name; substitute names listed in your overview:

sites = ["windfarm"]
locations = ["a01", "a02"]
metrics = ["mean windspeed", "mean power"]

site_names = [site.lower() for site in sites]
location_names = [location.lower() for location in locations]
site_mask = (
    overview["site"].str.lower().isin(site_names)
    | overview["site_id"].str.lower().isin(site_names)
)
location_mask = (
    overview["location"].str.lower().isin(location_names)
    | overview["location_id"].str.lower().isin(location_names)
)
pattern = "|".join(
    selector.replace(" ", ".*").replace("-", ".*")
    for selector in metrics
)
selected = overview.loc[
    site_mask & location_mask
    & overview["metric"].str.contains(pattern, case=False, regex=True),
    ["metric", "statistic", "short_hand", "site", "location"],
]
selected

Pass metrics to get_data() to use these selectors. To select one known metric instead, copy its full metric value and add ^ and $, as in the table. For all metrics at explicitly selected sites and locations, pass metrics="all"; preview those rows using only site_mask & location_mask. The SDK raises DataSignalsError if no overview rows match. These selection rules also apply to metric-based statistics and availability queries.

The get-data notebook walks through the same selection before retrieving data.

Getting sample data from the API#

After loading the environment variables and authenticating with the API, the user can get data from 24SEA API endpoints.

The retrieved data can be retrieved in multiple formats, depending on the combination of the following three options:

  • as_dict: If True, any dataframe output is returned as a dictionary according to the following formula: dataframe.reset_index().to_dict('records').

  • as_star_schema: If True, the data is returned in a star schema format, i.e. a dictionary with the following keys: DimCalendar, DimWindFarm, DimDataGroup, DimMetric, and FactData.

  • outer_join_on_timestamp: if True, the response from each metric is joined on the timestamp which is then set as the index of DataFrame. This option will necessarily drop the location and site columns from the DataFrame, but they can still be retrieved from the metrics overview. If False, the response from each metric is stored in a separate DataFrame and the site and location columns are kept. This means that the DataFrame will be “diagonal” with repeated timestamps (as many times as the number of queried (locations, sites) pairs).

Therefore, the following combinations are possible:

  • Single pandas.DataFrame: outer_join_on_timestamp=True, as_star_schema=False, and as_dict=False. This is the default behavior.

  • dict[str, Union[float, dict[str, Any]]]: outer_join_on_timestamp=False, as_star_schema=False, and as_dict=True.

  • Star schema as dict[str, pandas.Dataframe]: outer_join_on_timestamp is irrelevant, as_star_schema=True, and as_dict=False.

  • Star schema as dict[str, dict[str, Any]]: outer_join_on_timestamp is irrelevant, as_star_schema=True, and as_dict=True.

The optional method parameter controls the HTTP verb used for data retrieval. Supported values are "GET" and "POST". Use put_data() and patch_data() for writes; write payloads have a different contract from retrieval queries.

# %%
# **Data Retrieval**
sites = ["wf"]

locations = ["a01", "a02"]

metrics = ["mean WinDSpEed", "mean pitch", "mean-Yaw", "mean power"]

start_timestamp = "2020-03-01T00:00:00Z"
end_timestamp = "2020-06-01T00:00:00Z"

data_with_get = api.get_data(sites, locations, metrics,
                             start_timestamp, end_timestamp,
                             method="GET")

data_with_post = api.get_data(sites, locations, metrics,
                              start_timestamp, end_timestamp,
                              method="POST")

Data as Star Schema#

Overview#

The data as star schema feature is designed to provide a more user-friendly experience when getting data for BI purposes. It is implemented only for the core DataSignals API module.

from api_24sea.datasignals.core import API

# %%
# **Star schema**

# %%
# **Data Retrieval**
sites = ["wf"]

locations = ["a01", "a02"]

metrics = ["mean WinDSpEed", "mean pitch", "mean-Yaw", "mean_power"]

start_timestamp = "2020-03-01T00:00:00Z"
end_timestamp = "2020-06-01T00:00:00Z"

star_schema = api.get_data(sites, locations, metrics,
                           start_timestamp, end_timestamp, as_star_schema=True)

This command is equivalent to the following:

from api_24sea.datasignals.core import to_star_schema, API

# %%
# **Data Retrieval**
api = API()

sites = ["wf"]

locations = ["a01", "a02"]

metrics = ["mean WinDSpEed", "mean pitch", "mean-Yaw", "mean_power"]

start_timestamp = "2020-03-01T00:00:00Z"
end_timestamp = "2020-06-01T00:00:00Z"

data = api.get_data(sites, locations, metrics,
                    start_timestamp, end_timestamp)

star_schema = to_star_schema(data, api.selected_metrics(data).reset_index())

The star_schema variable will contain a dictionary with the following keys:

  • DimCalendar: A DataFrame with the timestamps and their corresponding calendar information.

  • DimWindFarm: A DataFrame with the wind farm information (site and locations).

  • DimDataGroup: A DataFrame with the metric group information (e.g. TP, SCADA).

  • DimMetric: A DataFrame with the metric information (e.g. mean pitch, mean power).

  • FactData: A DataFrame with the data itself.

  • FactPivotData: A DataFrame with the data pivoted by metric.

Data as Category-Value#

The data as category-value feature is designed to reshape the data so that all the information for a metric is stored in a single row.

# %%
# **Data Retrieval**
from api_24sea.datasignals.core import to_category_value
sites = ["wf"]

locations = ["a01", "a02"]

metrics = ["mean WinDSpEed", "mean pitch", "mean-Yaw", "mean_power"]

start_timestamp = "2020-03-01T00:00:00Z"
end_timestamp = "2020-06-01T00:00:00Z"

data = api.get_data(sites, locations, metrics,
                    start_timestamp, end_timestamp)

category_value = to_category_value(data, api.selected_metrics(data).reset_index())

Example table#

The data is transformed from the following (example) shape:

timestamp

mean_WF_A01_TP_SG_LAT005_DEG000

mean_WF_A01_TP_SG_LAT005_DEG045

mean_WF_A01_TP_SG_LAT005_DEG090

mean_WF_A02_TP_SG_LAT015_DEG000

mean_WF_A02_TP_SG_LAT015_DEG045

mean_WF_A02_TP_SG_LAT015_DEG090

2020-03-01 00:00:00

1.0

2.0

3.0

4.0

5.0

6.0

2020-03-01 00:10:00

1.1

2.1

3.1

4.1

5.1

6.1

2020-03-01 00:20:00

1.2

2.2

3.2

4.2

5.2

6.2

2020-03-01 00:30:00

1.3

2.3

3.3

4.3

5.3

6.3

to the following (example) shape:

timestamp

full_metric_name

value

unit

statistic

short_hand

site_id

location_id

lat

heading

site

location

metric_group

2020-03-01 00:00:00

mean_WF_A01_TP_SG_LAT005_DEG000

1.0

unit

mean

TP_SG_LAT005_DEG000

WF

A01

5.0

0.0

WindFarm

WFA01

TP

2020-03-01 00:10:00

mean_WF_A01_TP_SG_LAT005_DEG000

1.1

unit

mean

TP_SG_LAT005_DEG000

WF

A01

5.0

0.0

WindFarm

WFA01

TP

2020-03-01 00:20:00

mean_WF_A01_TP_SG_LAT005_DEG000

1.2

unit

mean

TP_SG_LAT005_DEG000

WF

A01

5.0

0.0

WindFarm

WFA01

TP

2020-03-01 00:30:00

mean_WF_A01_TP_SG_LAT005_DEG000

1.3

unit

mean

TP_SG_LAT005_DEG000

WF

A01

5.0

0.0

WindFarm

WFA01

TP

2020-03-01 00:00:00

mean_WF_A01_TP_SG_LAT005_DEG045

2.0

unit

mean

TP_SG_LAT005_DEG045

WF

A01

5.0

45.0

WindFarm

WFA01

TP

2020-03-01 00:10:00

mean_WF_A01_TP_SG_LAT005_DEG045

2.1

unit

mean

TP_SG_LAT005_DEG045

WF

A01

5.0

45.0

WindFarm

WFA01

TP

2020-03-01 00:20:00

mean_WF_A01_TP_SG_LAT005_DEG045

2.2

unit

mean

TP_SG_LAT005_DEG045

WF

A01

5.0

45.0

WindFarm

WFA01

TP

2020-03-01 00:30:00

mean_WF_A01_TP_SG_LAT005_DEG045

2.3

unit

mean

TP_SG_LAT005_DEG045

WF

A01

5.0

45.0

WindFarm

WFA01

TP

2020-03-01 00:00:00

mean_WF_A01_TP_SG_LAT005_DEG090

3.0

unit

mean

TP_SG_LAT005_DEG090

WF

A01

5.0

90.0

WindFarm

WFA01

TP

2020-03-01 00:10:00

mean_WF_A01_TP_SG_LAT005_DEG090

3.1

unit

mean

TP_SG_LAT005_DEG090

WF

A01

5.0

90.0

WindFarm

WFA01

TP

2020-03-01 00:20:00

mean_WF_A01_TP_SG_LAT005_DEG090

3.2

unit

mean

TP_SG_LAT005_DEG090

WF

A01

5.0

90.0

WindFarm

WFA01

TP

2020-03-01 00:30:00

mean_WF_A01_TP_SG_LAT005_DEG090

3.3

unit

mean

TP_SG_LAT005_DEG090

WF

A01

5.0

90.0

WindFarm

WFA01

TP

2020-03-01 00:00:00

mean_WF_A02_TP_SG_LAT015_DEG000

4.0

unit

mean

TP_SG_LAT015_DEG000

WF

A02

15.0

0.0

WindFarm

WFA02

TP

2020-03-01 00:10:00

mean_WF_A02_TP_SG_LAT015_DEG000

4.1

unit

mean

TP_SG_LAT015_DEG000

WF

A02

15.0

0.0

WindFarm

WFA02

TP

2020-03-01 00:20:00

mean_WF_A02_TP_SG_LAT015_DEG000

4.2

unit

mean

TP_SG_LAT015_DEG000

WF

A02

15.0

0.0

WindFarm

WFA02

TP

2020-03-01 00:30:00

mean_WF_A02_TP_SG_LAT015_DEG000

4.3

unit

mean

TP_SG_LAT015_DEG000

WF

A02

15.0

0.0

WindFarm

WFA02

TP

2020-03-01 00:00:00

mean_WF_A02_TP_SG_LAT015_DEG045

5.0

unit

mean

TP_SG_LAT015_DEG045

WF

A02

15.0

45.0

WindFarm

WFA02

TP

2020-03-01 00:10:00

mean_WF_A02_TP_SG_LAT015_DEG045

5.1

unit

mean

TP_SG_LAT015_DEG045

WF

A02

15.0

45.0

WindFarm

WFA02

TP

2020-03-01 00:20:00

mean_WF_A02_TP_SG_LAT015_DEG045

5.2

unit

mean

TP_SG_LAT015_DEG045

WF

A02

15.0

45.0

WindFarm

WFA02

TP

2020-03-01 00:30:00

mean_WF_A02_TP_SG_LAT015_DEG045

5.3

unit

mean

TP_SG_LAT015_DEG045

WF

A02

15.0

45.0

WindFarm

WFA02

TP

2020-03-01 00:00:00

mean_WF_A02_TP_SG_LAT015_DEG090

6.0

unit

mean

TP_SG_LAT015_DEG090

WF

A02

15.0

90.0

WindFarm

WFA02

TP

2020-03-01 00:10:00

mean_WF_A02_TP_SG_LAT015_DEG090

6.1

unit

mean

TP_SG_LAT015_DEG090

WF

A02

15.0

90.0

WindFarm

WFA02

TP

2020-03-01 00:20:00

mean_WF_A02_TP_SG_LAT015_DEG090

6.2

unit

mean

TP_SG_LAT015_DEG090

WF

A02

15.0

90.0

WindFarm

WFA02

TP

2020-03-01 00:30:00

mean_WF_A02_TP_SG_LAT015_DEG090

6.3

unit

mean

TP_SG_LAT015_DEG090

WF

A02

15.0

90.0

WindFarm

WFA02

TP

Tip

Try it live Run these examples directly in the browser with the Interactive Example.