Skip to content

Fetchers

A fetcher queries an API or a database and returns records. It does not store the dataset as files.

from gigaspatial import OSMLocationFetcher, OSMRoadsFetcher, GigaSchoolLocationFetcher
from gigaspatial.fetchers import MLabConfig, GEEConfig

MLabFetcher queries M-Lab speed tests in BigQuery. GEEProfiler is also a fetcher and has its own page.

OpenStreetMap

Fetcher for OpenStreetMap POIs and boundaries using the Overpass API.

Handles high-level queries for diverse OSM categories (amenities, buildings, shops, etc.) within specific administrative regions or countries.

Attributes:

Name Type Description
country Optional[str]

ISO country code or name.

admin_level Optional[int]

OSM administrative level for the search area.

admin_value Optional[str]

Name of the administrative area.

location_types Union[List[str], Dict[str, List[str]]]

Mapping of category keys (e.g., 'amenity') to values.

base_url str

Overpass API endpoint URL.

timeout int

Overpass server-side query timeout, in seconds. The actual HTTP client timeout used for requests will be this value plus a small buffer, so we don't give up locally right as the server is about to respond.

max_retries int

Number of retry attempts for failed requests.

retry_delay int

Delay between retries in seconds.

fetch_locations(since_date=None, handle_duplicates='separate', include_metadata=False)

Fetches OSM locations matching the configured categories.

Parameters:

Name Type Description Default
since_date Optional[Union[str, datetime]]

Optional date filter (addition or modification date). Supports ISO 8601 strings or datetime objects.

None
handle_duplicates Literal['separate', 'combine', 'primary']

Strategy for elements matching multiple types. - 'separate': Duplicate records (default). - 'combine': Single record with list of categories. - 'primary': Keep only the first matching category.

'separate'
include_metadata bool

If True, includes timestamp and user metadata.

False

Returns:

Type Description
DataFrame

A pandas DataFrame containing processed OSM locations.

Raises:

Type Description
ValueError

If an invalid duplicate handling strategy is provided.

fetch_locations_changed_between(start_date, end_date, handle_duplicates='separate', include_metadata=True)

Fetch OSM locations that changed within a specific date range.

Use this for historical analysis or tracking changes in a specific period.

Parameters:

Name Type Description Default
start_date Union[str, datetime]

Start date/time in ISO 8601 format (str: "YYYY-MM-DDThh:mm:ssZ") or datetime object. Changes after this date will be included.

required
end_date Union[str, datetime]

End date/time in ISO 8601 format (str: "YYYY-MM-DDThh:mm:ssZ") or datetime object. Changes before this date will be included.

required
handle_duplicates Literal['separate', 'combine', 'primary']

How to handle objects matching multiple categories: - 'separate': Create separate entries for each category (default) - 'combine': Use a single entry with a list of matching categories - 'primary': Keep only the first matching category

'separate'
include_metadata bool

If True, include change tracking metadata (timestamp, version, changeset, user, uid) Defaults to True since change tracking is the main use case.

True

Returns:

Type Description
DataFrame

A pandas DataFrame of historical changes.

fetch_by_osmid(osmid, element_type='node', include_metadata=False)

Retrieve and process a single OSM element by its unique identifier.

Parameters:

Name Type Description Default
osmid int

The OpenStreetMap ID.

required
element_type Literal['node', 'way', 'relation']

The type of element ('node', 'way', or 'relation').

'node'
include_metadata bool

If True, includes object-level metadata.

False

Returns:

Type Description
Optional[Dict]

A processed dictionary of the element, or None if not found.

OSMRoadsFetcher returns a road network rather than records: a SpatialNetwork. It needs pip install "giga-spatial[roads]", and the road networks guide says which area sizes suit Overpass.

Fetch the road network of a source from OpenStreetMap, through osmnx and the Overpass API.

The source is turned into an area to query: a country's boundary, a geometry as given, the union of a frame's polygons, or for points the convex hull (area="hull"), bounding box ("bbox"), or union of buffers ("buffers"), buffered by buffer meters. The result is a directed :class:SpatialNetwork, with list-valued OSM attributes normalised so it can be saved; the fetcher saves nothing itself.

Needs pip install 'giga-spatial[roads]'.

Attributes:

Name Type Description
network_type str

osmnx network type, such as "drive_service", "drive", "walk", "bike", or "all".

custom_filter Optional[Union[str, List[str]]]

Overpass way filter replacing network_type, such as '["highway"~"primary|secondary"]'.

simplify bool

Join edges between junctions, as osmnx does.

retain_all bool

Keep every disconnected part, not only the largest.

truncate_by_edge bool

Keep edges that leave the area at one end.

buffer float

Meters around point and line sources.

area Literal['hull', 'bbox', 'buffers']

How points become an area: "hull", "bbox", or "buffers".

overpass_url Optional[str]

Overpass endpoint, such as a mirror when the main server is down. The OVERPASS_URL setting otherwise.

timeout int

Seconds for each Overpass request.

max_retries int

Retries after a failed request, with growing pauses.

retry_delay float

Seconds before the first retry.

fetch(source)

The road network of a source.

Parameters:

Name Type Description Default
source Any

Country name or code, shapely geometry, GeoDataFrame, DataFrame with latitude and longitude columns, or list of (lat, lon) tuples or Points.

required

Returns:

Type Description
SpatialNetwork

A directed network in EPSG:4326 with lengths in meters. Its

SpatialNetwork

metadata records the source, the area, and the fetch parameters.

Raises:

Type Description
ImportError

If osmnx is not installed.

ValueError

If the source has no area, or OSM has no matching roads in it.

RuntimeError

If Overpass keeps failing after the retries.

query_area(source)

The area queried for a source, in EPSG:4326.

Giga schools

Fetch and process school location data from the Giga School Geolocation API.

Wraps the Giga REST API with authentication, retry logic, and pagination handled transparently via gigaspatial's HTTP module.

Parameters

country : str Country name or ISO 3166 alpha-2/alpha-3 code. api_key : str Bearer token for the Giga API. Defaults to GIGA_SCHOOL_LOCATION_API_KEY from global config. page_size : int Number of records per API page. Defaults to 1000. max_retries : int Maximum number of retry attempts on transient errors. Defaults to 3. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.

Examples

fetcher = GigaSchoolLocationFetcher(country="Kenya") df = fetcher.fetch_locations() gdf = fetcher.fetch_locations(process_geospatial=True) gdf = fetcher.fetch_locations(process_geospatial=True, max_pages=5)

fetch_locations(process_geospatial=False, max_pages=None)

Fetch school locations from the Giga API.

Parameters

process_geospatial : bool If True, returns a GeoDataFrame with a Point geometry column. Defaults to False. max_pages : int, optional Limit the number of pages fetched. Useful for testing.

Returns

pd.DataFrame or gpd.GeoDataFrame School location records. Empty DataFrame if no data is returned.

Fetch and process school profile data from the Giga School Profile API.

Includes connectivity information and other school details.

Parameters

country : str Country name or ISO 3166 alpha-2/alpha-3 code. api_key : str Bearer token for the Giga API. Defaults to GIGA_SCHOOL_PROFILE_API_KEY. page_size : int Number of records per API page. Defaults to 1000. max_retries : int Maximum retry attempts on transient errors. Defaults to 3. giga_id_school : str, optional Filter results to a specific school by Giga ID. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.

Examples

fetcher = GigaSchoolProfileFetcher(country="Kenya") df = fetcher.fetch_profiles() df = fetcher.fetch_profiles(max_pages=5) df = fetcher.fetch_profiles(giga_id_school="abc-123")

fetch_profiles(max_pages=None, giga_id_school=None)

Fetch school profiles including connectivity information.

Parameters

max_pages : int, optional Limit the number of pages fetched. Useful for testing. giga_id_school : str, optional Override the instance-level school ID filter.

Returns

pd.DataFrame School profiles. Empty DataFrame if no data is returned.

Fetch and process school daily realtime connectivity measurements from the Giga API.

Includes download/upload speeds, latency, and connectivity performance data.

Parameters

country : str Country name or ISO 3166 alpha-2/alpha-3 code. start_date : str, date, or datetime Start of the measurement date range (inclusive). end_date : str, date, or datetime End of the measurement date range (inclusive). api_key : str Bearer token for the Giga API. Defaults to GIGA_SCHOOL_MEASUREMENTS_API_KEY. page_size : int Number of records per API page. Defaults to 1000. max_retries : int Maximum retry attempts on transient errors. Defaults to 3. giga_id_school : str, optional Filter results to a specific school by Giga ID. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.

Examples

fetcher = GigaSchoolMeasurementsFetcher( ... country="Kenya", ... start_date="2024-01-01", ... end_date="2024-01-31", ... ) df = fetcher.fetch_measurements() df = fetcher.fetch_measurements(max_pages=3)

fetch_measurements(max_pages=None, giga_id_school=None, start_date=None, end_date=None)

Fetch school connectivity measurements.

Parameters

max_pages : int, optional Limit the number of pages fetched. Useful for testing. giga_id_school : str, optional Override the instance-level school ID filter. start_date : str, date, or datetime, optional Override the instance-level start date. end_date : str, date, or datetime, optional Override the instance-level end date.

Returns

pd.DataFrame Measurements with processed performance columns. Empty DataFrame if no data is returned.

Fetch and process country data from the Giga Country API.

Parameters

api_key : str Bearer token for the Giga API. Defaults to GIGA_COUNTRIES_API_KEY from global config. page_size : int Number of records per API page. Defaults to 100. max_retries : int Maximum number of retry attempts on transient errors. Defaults to 3. logger : logging.Logger, optional Logger instance. Defaults to the global config logger.

Examples

fetcher = GigaCountryFetcher() df = fetcher.fetch_countries() df_mng = fetcher.fetch_countries(country_iso3_code="MNG")

fetch_countries(id=None, country_iso3_code=None)

Fetch countries from the Giga API.

Parameters

id : int, optional Filter by country ID. country_iso3_code : str, optional Filter by country ISO3 code (e.g., "BRA", "MNG").

Returns

pd.DataFrame Country records. Empty DataFrame if no data is returned.

Healthsites

Fetch and process health facility data from the Healthsites.io API.

Supports both GeoJSON and JSON output formats, bounding box filtering, date range filtering, and per-facility lookups.

Parameters

country : str, optional Country name or ISO 3166 code. Converted to OSM English name internally. api_key : str Healthsites API key. Defaults to HEALTHSITES_API_KEY from global config. extent : tuple of float, optional Bounding box as (minLng, minLat, maxLng, maxLat). page_size : int Number of records per API page. Defaults to 100. flat_properties : bool Return properties in flat format. Defaults to True. tag_format : str Tag format, either 'osm' or 'hxl'. Defaults to 'osm'. output_format : str Response format, either 'geojson' or 'json'. Defaults to 'geojson'. max_retries : int Maximum retry attempts on transient errors. Defaults to 3. logger : logging.Logger, optional Logger instance. Defaults to global config logger.

Examples

fetcher = HealthSitesFetcher(country="Kenya") gdf = fetcher.fetch_facilities() gdf = fetcher.fetch_facilities(max_pages=3, output_format="geojson") df = fetcher.fetch_facilities(output_format="json")

fetch_facilities(max_pages=None, country=None, extent=None, output_format=None, flat_properties=None, from_date=None, to_date=None)

Fetch health facility locations.

Parameters

max_pages : int, optional Limit the number of pages fetched. Useful for testing. country : str, optional Override the instance-level country filter. extent : tuple, optional Override the instance-level bounding box. output_format : str, optional Override the instance-level output format ('geojson' or 'json'). flat_properties : bool, optional Override the instance-level flat_properties setting. from_date : str, date, or datetime, optional Return facilities modified after this timestamp. to_date : str, date, or datetime, optional Return facilities modified before this timestamp.

Returns

gpd.GeoDataFrame When output_format is 'geojson' (default). pd.DataFrame When output_format is 'json'.

fetch_statistics(country=None, extent=None, from_date=None, to_date=None)

Fetch aggregate statistics for health facilities.

Parameters

country : str, optional Override the instance-level country filter. extent : tuple, optional Override the instance-level bounding box. from_date : str, date, or datetime, optional Filter by modification timestamp start. to_date : str, date, or datetime, optional Filter by modification timestamp end.

Returns

dict Statistics returned by the API.

fetch_facility_by_id(osm_type, osm_id)

Fetch a single facility by its OSM type and ID.

Parameters

osm_type : str OSM element type: 'node', 'way', or 'relation'. osm_id : str OSM element ID.

Returns

dict Facility detail record.

Overture

Fetch and process amenity locations from the Overture Places theme.

This handler queries the Overture Places GeoParquet on S3, filters by a country boundary, and returns a GeoDataFrame of point locations for the requested amenity categories.

Attributes:

Name Type Description
release Optional[str]

Overture release version (e.g., "2024-11-13.1").

base_url Optional[str]

S3 pattern for the Places theme.

country str

ISO 3166-1 alpha-2 or alpha-3 country code.

amenity_types List[str]

List of primary categories to retrieve.

geom Union[Polygon, MultiPolygon]

Optional spatial boundary for filtering.

data_store DataStore

Optional storage interface for boundary retrieval.

country_geom_path Optional[Union[str, Path]]

Path to local country boundary file.

fetch_locations(match_pattern=False, **kwargs)

Execute the query and return locations as a GeoDataFrame.

Parameters:

Name Type Description Default
match_pattern bool

If True, uses ILIKE for category matching.

False
**kwargs

Additional execution parameters.

{}

Returns:

Type Description
GeoDataFrame

A GeoDataFrame of retrieved POIs.

UNICEF

Fetch UNICEF child welfare indicators from the UNICEF SDMX Data Warehouse.

Wraps the unicefdata Python package, providing access to 700+ indicators across categories such as child mortality, education, nutrition, immunisation, WASH, and SDGs.

Parameters

indicator : str SDMX indicator code, e.g. "CME_MRY0T4" (Under-5 mortality rate). Use :func:search_indicators or :func:list_categories from unicefdata to discover available codes. countries : list of str, optional ISO 3166 alpha-3 country codes (e.g. ["KEN", "NGA", "BRA"]). If None, fetches data for all available countries. year : str or int, optional Single year (2020), range ("2015:2023"), or list. If None, fetches all available years. sex : str, optional Sex disaggregation: "_T" (total), "F", "M", or "ALL". Defaults to "_T". format : str, optional Output format: "long" (default), "wide", or "wide_indicators". latest : bool, optional If True, keep only the most recent value per country. mrv : int, optional Keep the N most recent values per country. circa : bool, optional If True, return the closest available year when an exact match is not found. logger : logging.Logger, optional Logger instance. Defaults to the global GigaSpatial logger.

Examples

fetcher = UnicefDataFetcher( ... indicator="ED_CR_L1", ... countries=["KEN", "NGA", "ETH"], ... year="2015:2023", ... ) df = fetcher.fetch() df = fetcher.fetch(latest=True)

Spatial enrichment with admin boundaries:

gdf = fetcher.fetch_as_geodataframe(boundaries_gdf=admin_gdf, boundaries_iso3_col="iso3")

fetch_as_geodataframe(boundaries_gdf, boundaries_iso3_col='iso3', indicator_iso3_col='REF_AREA')

Fetch indicator data and spatially enrich it by joining to admin boundaries.

Parameters

boundaries_gdf : gpd.GeoDataFrame GeoDataFrame with administrative boundary geometries (e.g. country polygons). boundaries_iso3_col : str Column in boundaries_gdf containing ISO3 country codes. Defaults to "iso3". indicator_iso3_col : str Column in the fetched indicator DataFrame containing ISO3 codes. UNICEF SDMX uses "REF_AREA" by default.

Returns

gpd.GeoDataFrame Boundaries GeoDataFrame enriched with UNICEF indicator values, preserving the original CRS.

Raises

ValueError If boundaries_iso3_col is missing from boundaries_gdf or indicator_iso3_col is missing from the fetched data.

M-Lab

Fetcher for querying M-Lab NDT7 network measurement data from BigQuery.

Composes BigQueryClient for all query execution against the measurement-lab.ndt.ndt7 table. Returns results as DataFrames or GeoDataFrames using client.Geo.Latitude / client.Geo.Longitude available in the ndt7 table schema.

Parameters

config : MLabConfig, optional Dataset-level configuration. Defaults to MLabConfig(). client_config : BigQueryClientConfig, optional Auth/project configuration for the underlying BigQuery client. Defaults to BigQueryClientConfig() which reads from global_config.

Examples

fetcher = MLabFetcher() df = fetcher.query_ndt7("TR", start_date="2023-01-01") df = fetcher.query_ndt7("KE", measurement="upload", start_date="2023-06-01")

query_ndt7(country_code, start_date=None, end_date=None, measurement='download')

Query the NDT7 table and return results as a DataFrame.

Parameters

country_code : str ISO 3166-1 alpha-2 country code (e.g. "TR", "KE"). start_date : str, optional ISO date lower bound (YYYY-MM-DD). Defaults to config.default_start_date. end_date : str, optional ISO date upper bound (YYYY-MM-DD). No upper bound if not provided. measurement : {"download", "upload", "both"} Which measurement direction to query. Defaults to "download".

Returns

pd.DataFrame NDT7 measurements with id, date, derived a.* fields, and raw measurement fields for the selected direction.

query_ndt7_gdf(country_code, start_date=None, end_date=None, measurement='download')

Query the NDT7 table and return results as a GeoDataFrame.

Convenience wrapper around query_ndt7 that constructs point geometry from client.Geo.Latitude / client.Geo.Longitude. Rows where lat/lon are null are dropped before geometry construction.

Parameters

country_code : str ISO 3166-1 alpha-2 country code (e.g. "TR", "KE"). start_date : str, optional ISO date lower bound (YYYY-MM-DD). Defaults to config.default_start_date. end_date : str, optional ISO date upper bound (YYYY-MM-DD). measurement : {"download", "upload", "both"} Which measurement direction to query. Defaults to "download".

Returns

gpd.GeoDataFrame Point GeoDataFrame in EPSG:4326.