Fetchers¶
A fetcher queries an API or a database and returns records. It does not store the dataset as files.
from gigaspatial import OSMLocationFetcher, OSMRoadsFetcher, GigaSchoolLocationFetcher
from gigaspatial.fetchers import MLabConfig, GEEConfig
MLabFetcher queries M-Lab speed tests in BigQuery. GEEProfiler is also a fetcher and has its own page.
OpenStreetMap¶
Fetcher for OpenStreetMap POIs and boundaries using the Overpass API.
Handles high-level queries for diverse OSM categories (amenities, buildings, shops, etc.) within specific administrative regions or countries.
Attributes:
| Name | Type | Description |
|---|---|---|
country | Optional[str] | ISO country code or name. |
admin_level | Optional[int] | OSM administrative level for the search area. |
admin_value | Optional[str] | Name of the administrative area. |
location_types | Union[List[str], Dict[str, List[str]]] | Mapping of category keys (e.g., 'amenity') to values. |
base_url | str | Overpass API endpoint URL. |
timeout | int | Overpass server-side query timeout, in seconds. The actual HTTP client timeout used for requests will be this value plus a small buffer, so we don't give up locally right as the server is about to respond. |
max_retries | int | Number of retry attempts for failed requests. |
retry_delay | int | Delay between retries in seconds. |
fetch_locations(since_date=None, handle_duplicates='separate', include_metadata=False) ¶
Fetches OSM locations matching the configured categories.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
since_date | Optional[Union[str, datetime]] | Optional date filter (addition or modification date). Supports ISO 8601 strings or datetime objects. | None |
handle_duplicates | Literal['separate', 'combine', 'primary'] | Strategy for elements matching multiple types. - 'separate': Duplicate records (default). - 'combine': Single record with list of categories. - 'primary': Keep only the first matching category. | 'separate' |
include_metadata | bool | If True, includes timestamp and user metadata. | False |
Returns:
| Type | Description |
|---|---|
DataFrame | A pandas DataFrame containing processed OSM locations. |
Raises:
| Type | Description |
|---|---|
ValueError | If an invalid duplicate handling strategy is provided. |
fetch_locations_changed_between(start_date, end_date, handle_duplicates='separate', include_metadata=True) ¶
Fetch OSM locations that changed within a specific date range.
Use this for historical analysis or tracking changes in a specific period.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
start_date | Union[str, datetime] | Start date/time in ISO 8601 format (str: "YYYY-MM-DDThh:mm:ssZ") or datetime object. Changes after this date will be included. | required |
end_date | Union[str, datetime] | End date/time in ISO 8601 format (str: "YYYY-MM-DDThh:mm:ssZ") or datetime object. Changes before this date will be included. | required |
handle_duplicates | Literal['separate', 'combine', 'primary'] | How to handle objects matching multiple categories: - 'separate': Create separate entries for each category (default) - 'combine': Use a single entry with a list of matching categories - 'primary': Keep only the first matching category | 'separate' |
include_metadata | bool | If True, include change tracking metadata (timestamp, version, changeset, user, uid) Defaults to True since change tracking is the main use case. | True |
Returns:
| Type | Description |
|---|---|
DataFrame | A pandas DataFrame of historical changes. |
fetch_by_osmid(osmid, element_type='node', include_metadata=False) ¶
Retrieve and process a single OSM element by its unique identifier.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
osmid | int | The OpenStreetMap ID. | required |
element_type | Literal['node', 'way', 'relation'] | The type of element ('node', 'way', or 'relation'). | 'node' |
include_metadata | bool | If True, includes object-level metadata. | False |
Returns:
| Type | Description |
|---|---|
Optional[Dict] | A processed dictionary of the element, or None if not found. |
OSMRoadsFetcher returns a road network rather than records: a SpatialNetwork. It needs pip install "giga-spatial[roads]", and the road networks guide says which area sizes suit Overpass.
Fetch the road network of a source from OpenStreetMap, through osmnx and the Overpass API.
The source is turned into an area to query: a country's boundary, a geometry as given, the union of a frame's polygons, or for points the convex hull (area="hull"), bounding box ("bbox"), or union of buffers ("buffers"), buffered by buffer meters. The result is a directed :class:SpatialNetwork, with list-valued OSM attributes normalised so it can be saved; the fetcher saves nothing itself.
Needs pip install 'giga-spatial[roads]'.
Attributes:
| Name | Type | Description |
|---|---|---|
network_type | str | osmnx network type, such as |
custom_filter | Optional[Union[str, List[str]]] | Overpass way filter replacing |
simplify | bool | Join edges between junctions, as osmnx does. |
retain_all | bool | Keep every disconnected part, not only the largest. |
truncate_by_edge | bool | Keep edges that leave the area at one end. |
buffer | float | Meters around point and line sources. |
area | Literal['hull', 'bbox', 'buffers'] | How points become an area: |
overpass_url | Optional[str] | Overpass endpoint, such as a mirror when the main server is down. The |
timeout | int | Seconds for each Overpass request. |
max_retries | int | Retries after a failed request, with growing pauses. |
retry_delay | float | Seconds before the first retry. |
fetch(source) ¶
The road network of a source.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source | Any | Country name or code, shapely geometry, GeoDataFrame, DataFrame with latitude and longitude columns, or list of | required |
Returns:
| Type | Description |
|---|---|
SpatialNetwork | A directed network in EPSG:4326 with lengths in meters. Its |
SpatialNetwork | metadata records the source, the area, and the fetch parameters. |
Raises:
| Type | Description |
|---|---|
ImportError | If osmnx is not installed. |
ValueError | If the source has no area, or OSM has no matching roads in it. |
RuntimeError | If Overpass keeps failing after the retries. |
query_area(source) ¶
The area queried for a source, in EPSG:4326.
Giga schools¶
Fetch and process school location data from the Giga School Geolocation API.
Wraps the Giga REST API with authentication, retry logic, and pagination handled transparently via gigaspatial's HTTP module.
Parameters¶
country : str Country name or ISO 3166 alpha-2/alpha-3 code. api_key : str Bearer token for the Giga API. Defaults to GIGA_SCHOOL_LOCATION_API_KEY from global config. page_size : int Number of records per API page. Defaults to 1000. max_retries : int Maximum number of retry attempts on transient errors. Defaults to 3. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.
Examples¶
fetcher = GigaSchoolLocationFetcher(country="Kenya") df = fetcher.fetch_locations() gdf = fetcher.fetch_locations(process_geospatial=True) gdf = fetcher.fetch_locations(process_geospatial=True, max_pages=5)
fetch_locations(process_geospatial=False, max_pages=None) ¶
Fetch school locations from the Giga API.
Parameters¶
process_geospatial : bool If True, returns a GeoDataFrame with a Point geometry column. Defaults to False. max_pages : int, optional Limit the number of pages fetched. Useful for testing.
Returns¶
pd.DataFrame or gpd.GeoDataFrame School location records. Empty DataFrame if no data is returned.
Fetch and process school profile data from the Giga School Profile API.
Includes connectivity information and other school details.
Parameters¶
country : str Country name or ISO 3166 alpha-2/alpha-3 code. api_key : str Bearer token for the Giga API. Defaults to GIGA_SCHOOL_PROFILE_API_KEY. page_size : int Number of records per API page. Defaults to 1000. max_retries : int Maximum retry attempts on transient errors. Defaults to 3. giga_id_school : str, optional Filter results to a specific school by Giga ID. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.
Examples¶
fetcher = GigaSchoolProfileFetcher(country="Kenya") df = fetcher.fetch_profiles() df = fetcher.fetch_profiles(max_pages=5) df = fetcher.fetch_profiles(giga_id_school="abc-123")
fetch_profiles(max_pages=None, giga_id_school=None) ¶
Fetch school profiles including connectivity information.
Parameters¶
max_pages : int, optional Limit the number of pages fetched. Useful for testing. giga_id_school : str, optional Override the instance-level school ID filter.
Returns¶
pd.DataFrame School profiles. Empty DataFrame if no data is returned.
Fetch and process school daily realtime connectivity measurements from the Giga API.
Includes download/upload speeds, latency, and connectivity performance data.
Parameters¶
country : str Country name or ISO 3166 alpha-2/alpha-3 code. start_date : str, date, or datetime Start of the measurement date range (inclusive). end_date : str, date, or datetime End of the measurement date range (inclusive). api_key : str Bearer token for the Giga API. Defaults to GIGA_SCHOOL_MEASUREMENTS_API_KEY. page_size : int Number of records per API page. Defaults to 1000. max_retries : int Maximum retry attempts on transient errors. Defaults to 3. giga_id_school : str, optional Filter results to a specific school by Giga ID. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.
Examples¶
fetcher = GigaSchoolMeasurementsFetcher( ... country="Kenya", ... start_date="2024-01-01", ... end_date="2024-01-31", ... ) df = fetcher.fetch_measurements() df = fetcher.fetch_measurements(max_pages=3)
fetch_measurements(max_pages=None, giga_id_school=None, start_date=None, end_date=None) ¶
Fetch school connectivity measurements.
Parameters¶
max_pages : int, optional Limit the number of pages fetched. Useful for testing. giga_id_school : str, optional Override the instance-level school ID filter. start_date : str, date, or datetime, optional Override the instance-level start date. end_date : str, date, or datetime, optional Override the instance-level end date.
Returns¶
pd.DataFrame Measurements with processed performance columns. Empty DataFrame if no data is returned.
Fetch and process country data from the Giga Country API.
Parameters¶
api_key : str Bearer token for the Giga API. Defaults to GIGA_COUNTRIES_API_KEY from global config. page_size : int Number of records per API page. Defaults to 100. max_retries : int Maximum number of retry attempts on transient errors. Defaults to 3. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.
Examples¶
fetcher = GigaCountryFetcher() df = fetcher.fetch_countries() df_mng = fetcher.fetch_countries(country_iso3_code="MNG")
fetch_countries(id=None, country_iso3_code=None) ¶
Healthsites¶
Fetch and process health facility data from the Healthsites.io API.
Supports both GeoJSON and JSON output formats, bounding box filtering, date range filtering, and per-facility lookups.
Parameters¶
country : str, optional Country name or ISO 3166 code. Converted to OSM English name internally. api_key : str Healthsites API key. Defaults to HEALTHSITES_API_KEY from global config. extent : tuple of float, optional Bounding box as (minLng, minLat, maxLng, maxLat). page_size : int Number of records per API page. Defaults to 100. flat_properties : bool Return properties in flat format. Defaults to True. tag_format : str Tag format, either 'osm' or 'hxl'. Defaults to 'osm'. output_format : str Response format, either 'geojson' or 'json'. Defaults to 'geojson'. max_retries : int Maximum retry attempts on transient errors. Defaults to 3. logger : logging.Logger, optional Logger instance. Defaults to global config logger.
Examples¶
fetcher = HealthSitesFetcher(country="Kenya") gdf = fetcher.fetch_facilities() gdf = fetcher.fetch_facilities(max_pages=3, output_format="geojson") df = fetcher.fetch_facilities(output_format="json")
fetch_facilities(max_pages=None, country=None, extent=None, output_format=None, flat_properties=None, from_date=None, to_date=None) ¶
Fetch health facility locations.
Parameters¶
max_pages : int, optional Limit the number of pages fetched. Useful for testing. country : str, optional Override the instance-level country filter. extent : tuple, optional Override the instance-level bounding box. output_format : str, optional Override the instance-level output format ('geojson' or 'json'). flat_properties : bool, optional Override the instance-level flat_properties setting. from_date : str, date, or datetime, optional Return facilities modified after this timestamp. to_date : str, date, or datetime, optional Return facilities modified before this timestamp.
Returns¶
gpd.GeoDataFrame When output_format is 'geojson' (default). pd.DataFrame When output_format is 'json'.
fetch_statistics(country=None, extent=None, from_date=None, to_date=None) ¶
Fetch aggregate statistics for health facilities.
Parameters¶
country : str, optional Override the instance-level country filter. extent : tuple, optional Override the instance-level bounding box. from_date : str, date, or datetime, optional Filter by modification timestamp start. to_date : str, date, or datetime, optional Filter by modification timestamp end.
Returns¶
dict Statistics returned by the API.
Overture¶
Fetch and process amenity locations from the Overture Places theme.
This handler queries the Overture Places GeoParquet on S3, filters by a country boundary, and returns a GeoDataFrame of point locations for the requested amenity categories.
Attributes:
| Name | Type | Description |
|---|---|---|
release | Optional[str] | Overture release version (e.g., "2024-11-13.1"). |
base_url | Optional[str] | S3 pattern for the Places theme. |
country | str | ISO 3166-1 alpha-2 or alpha-3 country code. |
amenity_types | List[str] | List of primary categories to retrieve. |
geom | Union[Polygon, MultiPolygon] | Optional spatial boundary for filtering. |
data_store | DataStore | Optional storage interface for boundary retrieval. |
country_geom_path | Optional[Union[str, Path]] | Path to local country boundary file. |
fetch_locations(match_pattern=False, **kwargs) ¶
Execute the query and return locations as a GeoDataFrame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
match_pattern | bool | If True, uses ILIKE for category matching. | False |
**kwargs | Additional execution parameters. | {} |
Returns:
| Type | Description |
|---|---|
GeoDataFrame | A GeoDataFrame of retrieved POIs. |
UNICEF¶
Fetch UNICEF child welfare indicators from the UNICEF SDMX Data Warehouse.
Wraps the unicefdata Python package, providing access to 700+ indicators across categories such as child mortality, education, nutrition, immunisation, WASH, and SDGs.
Parameters¶
indicator : str SDMX indicator code, e.g. "CME_MRY0T4" (Under-5 mortality rate). Use :func:search_indicators or :func:list_categories from unicefdata to discover available codes. countries : list of str, optional ISO 3166 alpha-3 country codes (e.g. ["KEN", "NGA", "BRA"]). If None, fetches data for all available countries. year : str or int, optional Single year (2020), range ("2015:2023"), or list. If None, fetches all available years. sex : str, optional Sex disaggregation: "_T" (total), "F", "M", or "ALL". Defaults to "_T". format : str, optional Output format: "long" (default), "wide", or "wide_indicators". latest : bool, optional If True, keep only the most recent value per country. mrv : int, optional Keep the N most recent values per country. circa : bool, optional If True, return the closest available year when an exact match is not found. logger : logging.Logger, optional Logger instance. Defaults to the global GigaSpatial logger.
Examples¶
fetcher = UnicefDataFetcher( ... indicator="ED_CR_L1", ... countries=["KEN", "NGA", "ETH"], ... year="2015:2023", ... ) df = fetcher.fetch() df = fetcher.fetch(latest=True)
Spatial enrichment with admin boundaries:
gdf = fetcher.fetch_as_geodataframe(boundaries_gdf=admin_gdf, boundaries_iso3_col="iso3")
fetch_as_geodataframe(boundaries_gdf, boundaries_iso3_col='iso3', indicator_iso3_col='REF_AREA') ¶
Fetch indicator data and spatially enrich it by joining to admin boundaries.
Parameters¶
boundaries_gdf : gpd.GeoDataFrame GeoDataFrame with administrative boundary geometries (e.g. country polygons). boundaries_iso3_col : str Column in boundaries_gdf containing ISO3 country codes. Defaults to "iso3". indicator_iso3_col : str Column in the fetched indicator DataFrame containing ISO3 codes. UNICEF SDMX uses "REF_AREA" by default.
Returns¶
gpd.GeoDataFrame Boundaries GeoDataFrame enriched with UNICEF indicator values, preserving the original CRS.
Raises¶
ValueError If boundaries_iso3_col is missing from boundaries_gdf or indicator_iso3_col is missing from the fetched data.
M-Lab¶
Fetcher for querying M-Lab NDT7 network measurement data from BigQuery.
Composes BigQueryClient for all query execution against the measurement-lab.ndt.ndt7 table. Returns results as DataFrames or GeoDataFrames using client.Geo.Latitude / client.Geo.Longitude available in the ndt7 table schema.
Parameters¶
config : MLabConfig, optional Dataset-level configuration. Defaults to MLabConfig(). client_config : BigQueryClientConfig, optional Auth/project configuration for the underlying BigQuery client. Defaults to BigQueryClientConfig() which reads from global_config.
Examples¶
fetcher = MLabFetcher() df = fetcher.query_ndt7("TR", start_date="2023-01-01") df = fetcher.query_ndt7("KE", measurement="upload", start_date="2023-06-01")
query_ndt7(country_code, start_date=None, end_date=None, measurement='download') ¶
Query the NDT7 table and return results as a DataFrame.
Parameters¶
country_code : str ISO 3166-1 alpha-2 country code (e.g. "TR", "KE"). start_date : str, optional ISO date lower bound (YYYY-MM-DD). Defaults to config.default_start_date. end_date : str, optional ISO date upper bound (YYYY-MM-DD). No upper bound if not provided. measurement : {"download", "upload", "both"} Which measurement direction to query. Defaults to "download".
Returns¶
pd.DataFrame NDT7 measurements with id, date, derived a.* fields, and raw measurement fields for the selected direction.
query_ndt7_gdf(country_code, start_date=None, end_date=None, measurement='download') ¶
Query the NDT7 table and return results as a GeoDataFrame.
Convenience wrapper around query_ndt7 that constructs point geometry from client.Geo.Latitude / client.Geo.Longitude. Rows where lat/lon are null are dropped before geometry construction.
Parameters¶
country_code : str ISO 3166-1 alpha-2 country code (e.g. "TR", "KE"). start_date : str, optional ISO date lower bound (YYYY-MM-DD). Defaults to config.default_start_date. end_date : str, optional ISO date upper bound (YYYY-MM-DD). measurement : {"download", "upload", "both"} Which measurement direction to query. Defaults to "download".
Returns¶
gpd.GeoDataFrame Point GeoDataFrame in EPSG:4326.