Skip to content

Clients

A client is a connection used to query a database, a warehouse, or a web API. It is not a data store, and it is not a dataset: a fetcher or a handler may use one.

from gigaspatial import BigQueryClient, BigQueryClientConfig, DBConnection

A generalizable Google BigQuery client for GigaSpatial.

Provides generic, dataset-agnostic methods for schema inspection and query execution. Dataset-specific fetchers (e.g. MLabFetcher) should compose this client rather than reimplement it.

Parameters

config : BigQueryClientConfig, optional Validated configuration object. If not provided, a default config is constructed from global_config.

Examples

client = BigQueryClient() # uses global_config defaults client.list_datasets() df = client.query_to_dataframe("SELECT * FROM project.dataset.table LIMIT 10")

custom = BigQueryClient(BigQueryClientConfig(project="my-gcp-project"))

_resolve_credentials()

Resolve credentials using the following priority:

  1. Service account key file (config.service_account_key_path)
  2. Application Default Credentials (ADC)

Returns

google.auth.credentials.Credentials

get_query_cost_estimate(sql)

Estimate the cost of a BigQuery query.

Parameters

sql : str Standard SQL query string.

Returns

float Estimated cost in USD based on on-demand pricing (${_BQ_COST_PER_TIB_USD}/TiB). For flat-rate pricing this will not reflect actual cost.

get_table_schema(dataset_id, table_id)

Retrieve the schema for a specific table.

Parameters

dataset_id : str The BigQuery dataset ID. table_id : str The BigQuery table ID.

Returns

list[dict] List of field descriptors with keys: name, type, mode, description.

list_datasets(project_id=None)

List all available datasets in the configured project.

Parameters

project_id : str, optional The BigQuery project ID. Defaults to config.project.

Returns

list[str] Dataset IDs within the given project.

list_tables(dataset_id)

List all tables within a dataset.

Parameters

dataset_id : str The BigQuery dataset ID.

Returns

list[str] Table IDs within the given dataset.

query(sql, **kwargs)

Execute a SQL query and return a raw result iterator.

Parameters

sql : str Standard SQL query string. **kwargs Additional keyword arguments forwarded to bigquery.Client.query().

Returns

google.cloud.bigquery.table.RowIterator

query_to_dataframe(sql, dtypes=None, max_gb_allowed=10, is_ci=False, **kwargs)

Execute a SQL query and return results as a pandas DataFrame.

Parameters

sql : str Standard SQL query string. dtypes : dict, optional Column dtype overrides passed to to_dataframe(). max_gb_allowed : int, optional Maximum number of gigabytes to bill for the query. Defaults to 10. is_ci : bool, optional Whether the query is running in a CI environment. Defaults to False. **kwargs Additional keyword arguments forwarded to bigquery.Client.query().

Returns

pd.DataFrame DataFrame containing the query results.

Bases: BaseModel

Configuration for authenticating and initializing a BigQuery client.

Defaults are resolved from GigaSpatial's global_config, mirroring the pattern used by the Google Earth Engine profiler.

Credential resolution priority: 1. Service account key file (service_account_key_path) 2. Application Default Credentials (ADC) — for GCP-hosted environments

Parameters

project : str, optional GCP project ID to bill queries against. Defaults to global_config.GOOGLE_CLOUD_PROJECT. service_account : str, optional Service account email address. Defaults to global_config.GOOGLE_SERVICE_ACCOUNT. service_account_key_path : str, optional Path to the service account JSON key file. Defaults to global_config.GOOGLE_SERVICE_ACCOUNT_KEY_PATH. use_bq_storage : bool Enable BigQuery Storage API for faster query_to_dataframe reads. Defaults to False.

A unified database connection class supporting both Trino and PostgreSQL.

__init__(db_type=DB_CONFIG.get('db_type', 'postgresql'), host=DB_CONFIG.get('host', None), port=DB_CONFIG.get('port', None), user=DB_CONFIG.get('user', None), password=DB_CONFIG.get('password', None), catalog=DB_CONFIG.get('catalog', None), database=DB_CONFIG.get('database', None), schema=DB_CONFIG.get('schema', 'public'), http_scheme=DB_CONFIG.get('http_scheme', 'https'), sslmode=DB_CONFIG.get('sslmode', 'require'), **kwargs)

Initialize a database connection.

Parameters:

Name Type Description Default
db_type Literal['postgresql', 'trino']

Database type ("trino" or "postgresql").

 get('db_type', 'postgresql')
host Optional[str]

Server hostname or IP address.

 get('host', None)
port Union[int, str]

Database port number.

 get('port', None)
user Optional[str]

Username for authentication.

 get('user', None)
password Optional[str]

Password for authentication.

 get('password', None)
catalog Optional[str]

Catalog name (Trino only).

 get('catalog', None)
database Optional[str]

Database name (PostgreSQL only).

 get('database', None)
schema str

Default schema to use.

 get('schema', 'public')
http_scheme str

HTTP scheme for Trino ("http" or "https").

 get('http_scheme', 'https')
sslmode str

SSL mode for PostgreSQL (e.g., "require").

 get('sslmode', 'require')
**kwargs

Additional connection parameters for SQLAlchemy.

 {}

Raises:

Type Description
ImportError

If 'sqlalchemy' is not installed.
ValueError

If db_type is unsupported.

_add_event_listener()

Add event listeners for schema setting.

_create_postgresql_engine(**kwargs)

Create a PostgreSQL SQLAlchemy engine.

Parameters:

Name Type Description Default
**kwargs

Additional arguments for create_engine.

 {}

Returns:

Type Description
Engine

SQLAlchemy Engine.

_parse_schema_table(table_name, schema=None)

Parse a table name that might contain schema or catalog prefixes.

Parameters:

Name Type Description Default
table_name str

The table name (e.g., 'table', 'schema.table', or 'catalog.schema.table').

 required
schema Optional[str]

Optional explicit schema.

 None

Returns:

Type Description
tuple[Optional[str], str]

A tuple of (schema, table_name).

execute_query(query, fetch_results=True, params=None)

Executes a SQL query.

Parameters:

Name Type Description Default
query str

SQL query string to execute.

 required
fetch_results bool

Whether to fetch and return results.

 True
params Optional[Dict]

Optional dictionary of parameters for the query.

 None

Returns:

Type Description
Union[List[tuple], None]

List of result tuples if fetch_results is True, else None.

Raises:

Type Description
SQLAlchemyError

If query execution fails.

get_column_names(table_name, schema=None)

Get column names for a specific table.

Parameters:

Name Type Description Default
table_name str

Name of the table.

 required
schema Optional[str]

Schema name.

 None

Returns:

Type Description
List[str]

List of column names.

get_connection_string()

Returns the connection string used to create the engine.

Returns:

Type Description
str

The raw connection string.

get_extensions()

Get list of installed PostgreSQL extensions.

Returns:

Type Description
List[str]

List of extension names.

Raises:

Type Description
NotImplementedError

If database type is not PostgreSQL.

get_primary_keys(table_name, schema=None)

Get primary key columns for a table.

Parameters:

Name Type Description Default
table_name str

Name of the table.

 required
schema Optional[str]

Schema name.

 None

Returns:

Type Description
List[str]

List of primary key column names.

get_schema_names()

Get list of all schema names in the database.

Returns:

Type Description
List[str]

List of schema names.

get_table_info(table_name, schema=None)

Get detailed column information for a table.

Parameters:

Name Type Description Default
table_name str

Name of the table.

 required
schema Optional[str]

Schema name.

 None

Returns:

Type Description
List[Dict]

List of column metadata dictionaries.

get_table_names(schema=None)

Get list of table names in a schema.

Parameters:

Name Type Description Default
schema Optional[str]

Schema name. Defaults to default_schema.

 None

Returns:

Type Description
List[str]

List of table names.

get_view_names(schema=None)

Get list of view names in a schema.

Parameters:

Name Type Description Default
schema Optional[str]

Schema name. Defaults to default_schema.

 None

Returns:

Type Description
List[str]

List of view names.

read_sql_to_dask_dataframe(table_name, index_col, columns=None, limit=None, **kwargs)

Reads data into a Dask DataFrame.

Parameters:

Name Type Description Default
table_name str

Table name (schema.table or just table).

 required
index_col str

Column to use as index for partitioning.

 required
columns Optional[List[str]]

List of columns to select.

 None
limit Optional[int]

Maximum rows to return.

 None
**kwargs

Additional arguments for dd.read_sql_query.

 {}

Returns:

Type Description
DataFrame

Dask DataFrame with results.

Raises:

Type Description
ImportError

If 'dask' is not installed.
ValueError

If reading fails.

read_sql_to_dataframe(query, params=None)

Executes query and returns results as pandas DataFrame.

Parameters:

Name Type Description Default
query str

SQL query string to execute.

 required
params Optional[Dict]

Optional parameters for the query.

 None

Returns:

Type Description
DataFrame

pandas DataFrame containing the query results.

Raises:

Type Description
SQLAlchemyError

If query execution fails.

table_exists(table_name, schema=None)

Check if a table exists in the database.

Parameters:

Name Type Description Default
table_name str

Name of the table.

 required
schema Optional[str]

Schema name.

 None

Returns:

Type Description
bool

True if table exists, False otherwise.

test_connection()

Tests the database connection.

Returns:

Type Description
bool

True if connection successful, False otherwise.