Handler base¶

This section is for adding a dataset to this library. The classes below are internal: this repository is their only subclasser, and the hooks are not a stable API for other packages. Documenting them does not change that. See the decision.

The guides never use these names. A caller passes a source and may set crop_to_source. The classes live in gigaspatial.handlers.base, which is not a public import path.

Search key and search geometry¶

A handler turns a source into files in two different ways.

The search key is what the dataset's index understands. It may be a country code, a filter, a geometry, or nothing, for a dataset published as one global file. get_search_key is the method a dataset overrides.

The search geometry is always a Shapely geometry in the handler's CRS. Cropping uses only that geometry. extract_search_geometry builds it. A search key that is a geometry is the search geometry.

The split, and the cases where a country source is not cropped, are recorded in Search key and search geometry.

Anything that changes which file is fetched belongs on the data unit, not in an option the downloader has to interpret.

The classes¶

BaseHandler is what a caller holds. load_data ensures the files are stored, then reads them. BaseHandlerConfig resolves a source into data units and stored paths. BaseHandlerDownloader fetches units. BaseHandlerReader loads stored paths and may crop.

A downloader or reader declares its config class as config_type, so it can also be built on its own from a config, a dict of config fields, or nothing.

Keyword arguments stop at those public methods. Each class declares an options type: ResolveOptions, DownloadOptions, or ReadOptions. An unknown keyword raises TypeError.

Bases: ABC

Main entry point and orchestrator for dataset handlers.

Encapsulates a configuration, downloader, and reader to provide a unified API for data acquisition and access. Handlers managed by this class are intended to be the primary interface for library users.

Subclasses must implement the factory methods to provide specific component implementations.

`load_data(source, crop_to_source=False, ensure_available=True, **kwargs)` ¶

Load data from a source, ensuring its availability first.

This is the primary high-level method for users to retrieve data from a handler.

Parameters:

Name	Type	Description	Default
`source`	`Union[str, List[Union[tuple, Point]], BaseGeometry, GeoDataFrame, Path, List[Union[str, Path]]]`	Specification of the data to load.	required
`crop_to_source`	`bool`	If True, crops the result to the source geometry.	`False`
`ensure_available`	`bool`	If True, downloads missing data before loading.	`True`
`**kwargs`		Resolve, download and read options.	`{}`

Returns:

Type	Description
`Any`	The loaded data object.

Raises:

Type	Description
`TypeError`	If a keyword is not an option of any component.
`RuntimeError`	If data availability cannot be ensured.

`download_and_load(source, crop_to_source=False, force_download=False, **kwargs)` ¶

Download (if missing) and load data in a single call.

Parameters:

Name	Type	Description	Default
`source`	`Union[str, List[Union[tuple, Point]], BaseGeometry, GeoDataFrame, Path, List[Union[str, Path]]]`	Specification of the data to acquire and load.	required
`crop_to_source`	`bool`	If True, crops the result to the source geometry.	`False`
`force_download`	`bool`	If True, re-downloads data even if present.	`False`
`**kwargs`		Resolve, download and read options.	`{}`

Returns:

Type	Description
`Any`	The loaded data object.

Raises:

Type	Description
`TypeError`	If a keyword is not an option of any component.

`ensure_data_available(source, force_download=False, **kwargs)` ¶

Guarantee that the required data for a source exists in the DataStore.

Resolves the source into data units and downloads those that are not stored yet (config.is_unit_available), or all of them if force_download is True. Path sources are only checked for existence.

Parameters:

Name	Type	Description	Default
`source`	`Union[str, List[Union[tuple, Point]], BaseGeometry, GeoDataFrame, Path, List[Union[str, Path]]]`	Specification of the required data.	required
`force_download`	`bool`	If True, downloads every unit regardless of local state.	`False`
`**kwargs`		Resolve and download options.	`{}`

Returns:

Type	Description
`bool`	True if all required data is available in the DataStore, False if
`bool`	some units (or explicit paths) are still missing after the
`bool`	download attempt.

Raises:

Type	Description
`TypeError`	If a keyword is neither a resolve nor a download option.
`Exception`	Errors raised while resolving the source into data units or paths propagate unchanged, as do programming errors (`TypeError`, `AttributeError`, `NameError`) raised while downloading. Other download errors are logged per unit, and the availability check that follows decides the return value.

`get_available_data_info(source, **kwargs)` ¶

Determine the availability status of data for a given source.

Parameters:

Name	Type	Description	Default
`source`	`Union[str, List[Union[tuple, Point]], BaseGeometry, GeoDataFrame]`	Specification of the data to check.	required
`**kwargs`		Resolve options.	`{}`

Returns:

Type	Description
`dict`	A dictionary with the number of available and missing data units,
`dict`	and the stored paths that exist and do not exist.

Raises:

Type	Description
`TypeError`	If a keyword is not a resolve option.

`create_config(data_store, logger, **kwargs)` `abstractmethod` ¶

Create and return a configuration object for this handler.

Parameters:

Name	Type	Description	Default
`data_store`	`DataStore`	The data store instance to use	required
`logger`	`Logger`	The logger instance to use	required
`**kwargs`		Additional configuration parameters	`{}`

Returns:

Type	Description
`BaseHandlerConfig`	Configured BaseHandlerConfig instance

`create_downloader(config, data_store, logger, **kwargs)` `abstractmethod` ¶

Create and return a downloader object for this handler.

Parameters:

Name	Type	Description	Default
`config`	`BaseHandlerConfig`	The configuration object	required
`data_store`	`DataStore`	The data store instance to use	required
`logger`	`Logger`	The logger instance to use	required
`**kwargs`		Additional downloader parameters	`{}`

Returns:

Type	Description
`BaseHandlerDownloader`	Configured BaseHandlerDownloader instance

`create_reader(config, data_store, logger, **kwargs)` `abstractmethod` ¶

Create and return a reader object for this handler.

Parameters:

Name	Type	Description	Default
`config`	`BaseHandlerConfig`	The configuration object	required
`data_store`	`DataStore`	The data store instance to use	required
`logger`	`Logger`	The logger instance to use	required
`**kwargs`		Additional reader parameters	`{}`

Returns:

Type	Description
`BaseHandlerReader`	Configured BaseHandlerReader instance

Bases: ABC

Abstract base class for handler configurations.

Standardizes common fields used across all handlers, such as file paths, worker counts, and data store abstractions.

A source is resolved in two steps. get_search_key turns it into the value the dataset's own lookup understands (by default the search geometry), and get_relevant_data_units_by_key turns that key into data units. The search geometry, the area used for cropping, is always built by extract_search_geometry in crs.

Attributes:

Name	Type	Description
`base_path`	`Path`	Root directory for the dataset's files.
`n_workers`	`int`	Number of threads downloading data units at once.
`data_store`	`DataStore`	Abstraction for file-system access (local or cloud).
`logger`	`Logger`	Logger instance for tracking handler operations.
`units_match_country`	`bool`	True when the dataset is published per country, so data loaded for a country source already covers exactly that country and is not cropped.
`options_type`	`Type[ResolveOptions]`	The `ResolveOptions` subclass this config's hooks take.

`get_search_key(source, options)` ¶

Turn a source into the value the dataset's unit lookup understands.

Override for datasets indexed by something other than area, such as a country code, or return None for a single global dataset. The default is the search geometry; a key that is a geometry must be the search geometry, since the cache reuses it for cropping.

Parameters:

Name	Type	Description	Default
`source`		Input source (country, geometry, points, DataFrame or GDF).	required
`options`	`ResolveOptions`	The resolve options.	required

Returns:

Type	Description
`Any`	The search key passed to `get_relevant_data_units_by_key`.

`get_relevant_data_units_by_key(search_key, options)` `abstractmethod` ¶

Identify data units associated with a search key.

Parameters:

Name	Type	Description	Default
`search_key`	`Any`	The value returned by `get_search_key`.	required
`options`		The resolve options.	required

Returns:

Type	Description
`Any`	A collection of relevant data unit identifiers.

`get_relevant_data_units(source, force_recompute=False, **kwargs)` ¶

Retrieve data unit identifiers relevant to a specific source.

Standardizes the resolution of a source (e.g., country code or geometry) into a list of data units (e.g., specific files or tiles needing to be loaded).

Parameters:

Name	Type	Description	Default
`source`	`Union[str, List[Union[Tuple[float, float], Point]], BaseGeometry, GeoDataFrame]`	Input source identifier (ISO country code, geometry, point list, or GDF).	required
`force_recompute`	`bool`	If True, ignores cached units and recomputes.	`False`
`**kwargs`		Resolve options (see `options_type`).	`{}`

Returns:

Type	Description
	A list of relevant data unit identifiers.

Raises:

Type	Description
`TypeError`	If a keyword is not a resolve option.

Note

Handlers customise resolution through get_search_key and get_relevant_data_units_by_key; this method and its cache are not meant to be overridden.

`extract_search_geometry(source, options=None)` ¶

Build the search geometry of a source, in self.crs.

The input coordinates are read in the GeoDataFrame's own CRS, else in the crs option, else in EPSG:4326, and reprojected to self.crs. Country sources use the country's admin boundary.

Parameters:

Name	Type	Description	Default
`source`		Country code or name, GeoDataFrame, shapely geometry, DataFrame with coordinate columns, or iterable of `(lat, lon)` pairs or `Point` objects.	required
`options`	`Optional[ResolveOptions]`	The resolve options; `crs` is the CRS of a source that carries none, and country sources read their boundary from `country_geom_path` / `country_data_store` when given.	`None`

Returns:

Type	Description
`BaseGeometry`	A single shapely geometry in `self.crs`.

Raises:

Type	Description
`ValueError`	If the source has no geometry (e.g. a path or a filter dict).

`get_search_geometry(source, **kwargs)` ¶

Return the search geometry of a source, computing it at most once per source and caching it.

Parameters:

Name	Type	Description	Default
`source`		Input source.	required
`**kwargs`		Resolve options (see `options_type`).	`{}`

`get_data_unit_path(unit, options=None)` ¶

Resolve a data unit to its single stored file.

Implement this for datasets that store one file per unit, or override get_stored_paths instead.

Parameters:

Name	Type	Description	Default
`unit`	`Any`	The data unit identifier (e.g., a filename or tile ID).	required
`options`	`Optional[ResolveOptions]`	The resolve options; None when called outside resolution.	`None`

Returns:

Type	Description
`Path`	The path of the unit's stored file in the data store.

`get_stored_paths(unit, options=None)` ¶

Return the files a data unit is stored as in the data store.

These are the files that end up in storage after download (e.g. the .tif extracted from a downloaded zip), not the remote resource. The default is the single path from get_data_unit_path.

Parameters:

Name	Type	Description	Default
`unit`	`Any`	The data unit identifier.	required
`options`	`Optional[ResolveOptions]`	The resolve options; None when called outside resolution.	`None`

Returns:

Type	Description
`List[Path]`	The unit's stored file paths.

`get_data_unit_paths(units, **kwargs)` ¶

Return the stored file paths of several data units, flattened.

Parameters:

Name	Type	Description	Default
`units`	`Union[Iterable[Any]]`	Iterable (or DataFrame) of data unit identifiers.	required
`**kwargs`		Resolve options (see `options_type`).	`{}`

Returns:

Type	Description
`list`	A flat list of all stored file paths.

Raises:

Type	Description
`TypeError`	If a keyword is not a resolve option.

`is_unit_available(unit, paths)` ¶

Return True if a data unit is already stored and needs no download.

Parameters:

Name	Type	Description	Default
`unit`	`Any`	The data unit identifier.	required
`paths`	`List[Path]`	The unit's stored paths, from `get_stored_paths`.	required

Returns:

Type	Description
`bool`	True if every stored path exists in the data store.

Bases: ABC

Abstract base class for data downloaders.

Handles the logic for acquiring dataset files from remote sources and saving them to a DataStore. Supports parallel downloads and progress tracking.

Attributes:

Name	Type	Description
`config`		Handler configuration object.
`data_store`		Abstraction for file-system access.
`logger`		Logger instance for tracking download progress.
`options_type`	`Type[DownloadOptions]`	The `DownloadOptions` subclass `download_data_unit` takes.
`config_type`	`Optional[Type[BaseHandlerConfig]]`	The config class built when the downloader is given a dict or no config.

`download(source, **kwargs)` ¶

Acquire all data units relevant to a specific source.

Resolves the source (e.g., country code) into units and triggers the download process.

Parameters:

Name	Type	Description	Default
`source`		Input source identifier.	required
`**kwargs`		Resolve and download options.	`{}`

Returns:

Type	Description
	A list of all downloaded items.

Raises:

Type	Description
`TypeError`	If a keyword is neither a resolve nor a download option.

`download_data_units(units, **kwargs)` ¶

Download multiple data units, in config.n_workers threads.

A unit that fails is logged and skipped; the other units still download.

Parameters:

Name	Type	Description	Default
`units`	`Iterable[Any]`	Iterable of data unit identifiers.	required
`**kwargs`		Download options (see `options_type`).	`{}`

Returns:

Type	Description
`List[Any]`	A flat list of the downloaded file paths.

Raises:

Type	Description
`TypeError`	If a keyword is not a download option, or a downloader raises a programming error (`TypeError`, `AttributeError` or `NameError`).

`download_data_unit(unit, options)` `abstractmethod` ¶

Download a single data unit.

Must be implemented by subclasses to define the specific download logic for the dataset. Units may be downloaded concurrently from threads, so implementations must not share mutable state unguarded.

Parameters:

Name	Type	Description	Default
`unit`	`Any`	Data unit identifier.	required
`options`		The download options.	required

Returns:

Type	Description
	The stored path, or a list of paths for a unit stored as several
	files.

Raises:

Type	Description
`Exception`	Any failure to download the unit. The base logs it and carries on with the other units.

Bases: ABC

Abstract base class for data readers.

Provides high-level methods for resolving data paths and loading various data types (raster, tabular) into memory. Standardizes the loading interface across disparate datasets.

Attributes:

Name	Type	Description
`config`		Handler configuration object.
`data_store`		Abstraction for file-system access.
`logger`		Logger instance for tracking data loading.
`options_type`	`Type[ReadOptions]`	The `ReadOptions` subclass `load_from_paths` takes.
`config_type`	`Optional[Type[BaseHandlerConfig]]`	The config class built when the reader is given a dict or no config.

`load(source, crop_to_source=False, **kwargs)` ¶

Orchestrate the resolution and loading of data from a source.

Resolves the source to stored paths, loads them with load_from_paths, then optionally crops to the source.

Parameters:

Name	Type	Description	Default
`source`	`Union[str, List[Union[Tuple[float, float], Point]], BaseGeometry, GeoDataFrame, Path, str, List[Union[str, Path]]]`	specification of the data to load.	required
`crop_to_source`	`bool`	If True, the data is cropped to the search geometry of the source. Path sources are never cropped, and neither are country sources when `config.units_match_country` is True.	`False`
`**kwargs`		Resolve and read options.	`{}`

Returns:

Type	Description
`Any`	The loaded data object.

Raises:

Type	Description
`TypeError`	If a keyword is neither a resolve nor a read option.
`ValueError`	If cropping is requested for a source with no geometry, such as a filter dict.

`load_from_paths(source_data_path, options)` `abstractmethod` ¶

Abstract method to load source data from paths.

Parameters:

Name	Type	Description	Default
`source_data_path`	`List[Union[str, Path]]`	List of source paths	required
`options`		The read options.	required

Returns:

Type	Description
`Any`	Loaded data (DataFrame, GeoDataFrame, etc.)

`resolve_source_paths(source, **kwargs)` ¶

Identify data file paths corresponding to a specific source input.

Parses the input source (e.g., country code, geometry, or explicit paths) and resolves them into a list of absolute paths within the DataStore.

Parameters:

Name	Type	Description	Default
`source`	`Union[str, List[Union[Tuple[float, float], Point]], BaseGeometry, GeoDataFrame, Path, str, List[Union[str, Path]]]`	Source specification (country, geometry, points, or paths).	required
`**kwargs`		Resolve options (see `config.options_type`).	`{}`

Returns:

Type	Description
`List[Union[str, Path]]`	A list of resolved file paths.

Raises:

Type	Description
`TypeError`	If a keyword is not a resolve option.

Options that change which data units and stored paths a source resolves to.

They are part of the unit cache key, so every value must be hashable.

Attributes:

Name	Type	Description
`crs`	`Any`	CRS of a source that carries none (a bare geometry, points or a DataFrame). Defaults to EPSG:4326.
`country_geom_path`	`Optional[Union[str, Path]]`	File holding the boundary of a country source, used instead of the default admin boundaries.
`country_data_store`	`Any`	Data store the country boundary is read from.

Options for fetching data units.

Attributes:

Name	Type	Description
`show_progress`	`bool`	If True, shows a progress bar over the units.

Options for loading stored files into memory.

Handler base¶

Search key and search geometry¶

The classes¶

load_data(source, crop_to_source=False, ensure_available=True, **kwargs) ¶

download_and_load(source, crop_to_source=False, force_download=False, **kwargs) ¶

ensure_data_available(source, force_download=False, **kwargs) ¶

get_available_data_info(source, **kwargs) ¶

create_config(data_store, logger, **kwargs) abstractmethod ¶

create_downloader(config, data_store, logger, **kwargs) abstractmethod ¶

create_reader(config, data_store, logger, **kwargs) abstractmethod ¶

get_search_key(source, options) ¶

get_relevant_data_units_by_key(search_key, options) abstractmethod ¶

get_relevant_data_units(source, force_recompute=False, **kwargs) ¶

extract_search_geometry(source, options=None) ¶

get_search_geometry(source, **kwargs) ¶

get_data_unit_path(unit, options=None) ¶

get_stored_paths(unit, options=None) ¶

get_data_unit_paths(units, **kwargs) ¶

is_unit_available(unit, paths) ¶

download(source, **kwargs) ¶

download_data_units(units, **kwargs) ¶

download_data_unit(unit, options) abstractmethod ¶

load(source, crop_to_source=False, **kwargs) ¶

load_from_paths(source_data_path, options) abstractmethod ¶

resolve_source_paths(source, **kwargs) ¶

`load_data(source, crop_to_source=False, ensure_available=True, **kwargs)` ¶

`download_and_load(source, crop_to_source=False, force_download=False, **kwargs)` ¶

`ensure_data_available(source, force_download=False, **kwargs)` ¶

`get_available_data_info(source, **kwargs)` ¶

`create_config(data_store, logger, **kwargs)` `abstractmethod` ¶

`create_downloader(config, data_store, logger, **kwargs)` `abstractmethod` ¶

`create_reader(config, data_store, logger, **kwargs)` `abstractmethod` ¶

`get_search_key(source, options)` ¶

`get_relevant_data_units_by_key(search_key, options)` `abstractmethod` ¶

`get_relevant_data_units(source, force_recompute=False, **kwargs)` ¶

`extract_search_geometry(source, options=None)` ¶

`get_search_geometry(source, **kwargs)` ¶

`get_data_unit_path(unit, options=None)` ¶

`get_stored_paths(unit, options=None)` ¶

`get_data_unit_paths(units, **kwargs)` ¶

`is_unit_available(unit, paths)` ¶

`download(source, **kwargs)` ¶

`download_data_units(units, **kwargs)` ¶

`download_data_unit(unit, options)` `abstractmethod` ¶

`load(source, crop_to_source=False, **kwargs)` ¶

`load_from_paths(source_data_path, options)` `abstractmethod` ¶

`resolve_source_paths(source, **kwargs)` ¶