Data stores¶
A data store is where a handler keeps its files. LocalDataStore is the default. A client is not a data store.
read_dataset and write_dataset read and write a file through whichever store you pass.
from gigaspatial import LocalDataStore, ADLSDataStore, read_dataset, write_dataset
from gigaspatial.core.io import DataStore
Every store implements the abstract DataStore. To support another storage service, subclass it.
Bases: DataStore
Implementation for local filesystem storage.
__init__(base_path='') ¶
Initialize the local data store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_path | Union[str, Path] | Base directory for relative paths. Defaults to current directory. | '' |
copy_file(src, dst) ¶
Copy a file from src to dst.
exists(path) ¶
Checks if path exists.
file_exists(path) ¶
Checks if file exists at path.
is_dir(path) ¶
Checks if path is a directory.
is_file(path) ¶
Checks if path is a file.
list_directories(path) ¶
List immediate subdirectories in path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Directory to list. | required |
Returns:
| Type | Description |
|---|---|
List[str] | List of subdirectory names. |
list_files(path) ¶
List all files in a directory, returning relative paths from base_path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Directory to list. | required |
Returns:
| Type | Description |
|---|---|
List[str] | List of relative file paths. |
mkdir(path, exist_ok=False) ¶
Creates a directory and its parents.
open(path, mode='r', encoding=None) ¶
Open a file-like object using the local filesystem.
read_file(path, encoding=None) ¶
Read contents of a file. Returns bytes unless encoding is given.
remove(path) ¶
Removes a file.
rmdir(directory) ¶
Removes a directory.
walk(top) ¶
Walk through directory tree.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
top | str | Starting directory. | required |
Yields:
| Type | Description |
|---|---|
Tuple[str, List[str], List[str]] | Tuple of (relative_root, dirnames, filenames). |
write_file(path, data, encoding=None) ¶
Write data (str, bytes, or a readable binary stream) to a file. Automatically creates parent directories if they don't exist.
Bases: DataStore
An implementation of DataStore for Azure Data Lake Storage.
__init__(container=config.ADLS_CONTAINER_NAME, connection_string=config.ADLS_CONNECTION_STRING, account_url=config.ADLS_ACCOUNT_URL, sas_token=config.ADLS_SAS_TOKEN) ¶
Initialize the ADLS data store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
container | str | The name of the container in ADLS to interact with. | ADLS_CONTAINER_NAME |
connection_string | str | Azure storage connection string. | ADLS_CONNECTION_STRING |
account_url | str | Azure storage account URL. | ADLS_ACCOUNT_URL |
sas_token | str | SAS token for authentication if connection_string is not provided. | ADLS_SAS_TOKEN |
Raises:
| Type | Description |
|---|---|
ImportError | If 'azure-storage-blob' is not installed. |
ValueError | If authentication credentials are missing. |
_normalize_path(path) ¶
Normalize a path for Azure Blob Storage.
DEPRECATED: Use _to_blob_key() instead for better type support. This method is kept for backward compatibility.
Ensures the path: - Ends with '/' if it's a directory - Doesn't start with '/' (Azure doesn't use leading slashes) - Uses forward slashes
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Path to normalize | required |
Returns:
| Type | Description |
|---|---|
str | Normalized path |
Examples:
_normalize_path('data') -> 'data/' _normalize_path('data/') -> 'data/' _normalize_path('/data/') -> 'data/' _normalize_path('data\subdir') -> 'data/subdir/'
_to_blob_key(path, *, ensure_dir=False) ¶
Convert an input path (str or PathLike) to a normalized ADLS blob key.
Rules: - Accepts str, pathlib.Path, PurePath, etc. - Converts Windows backslashes to '/' via PurePosixPath. - Strips any leading '/'. - Optionally enforces a trailing '/' for directory prefixes.
copy_directory(source_dir, destination_dir) ¶
Copies all files from a source directory to a destination directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_dir | Pathish | The source directory path in blob storage. | required |
destination_dir | Pathish | The destination directory path in blob storage. | required |
copy_file(source_path, destination_path, overwrite=False) ¶
Copies a single file from source to destination.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_path | Pathish | The source file path in blob storage. | required |
destination_path | Pathish | The destination file path in blob storage. | required |
overwrite | bool | If True, overwrite the destination file if it already exists. | False |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If source file is missing. |
FileExistsError | If destination exists and overwrite=False. |
count_files(dir_path) ¶
Count total files in a directory (memory efficient).
More memory-efficient than len(list_files()) for large directories.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir_path | Pathish | Directory path | required |
Returns:
| Type | Description |
|---|---|
int | Number of files |
Example
count = store.count_files('data/') print(f"Total files: {count}")
Performance
- Must iterate all files (no shortcuts)
- Memory: ~2MB (one page at a time)
- Time: ~30-60s for 1M files (Azure rate limit)
count_files_with_extension(dir_path, extension) ¶
Count files with specific extension (memory efficient).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir_path | Pathish | Directory path | required |
extension | str | File extension (e.g., '.png' or 'png') | required |
Returns:
| Type | Description |
|---|---|
int | Number of files with given extension |
Example
png_count = store.count_files_with_extension('images/', '.png') print(f"Found {png_count} PNG files")
download_directory(blob_dir_path, local_dir_path) ¶
Downloads all files from a directory in Azure Blob Storage to a local directory.
exists(path) ¶
Checks if a blob exists at the prefix.
file_exists(path) ¶
Checks if a blob is a file and exists.
get_file_metadata(path) ¶
Retrieve comprehensive file metadata.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | Pathish | File path in blob storage. | required |
Returns:
| Type | Description |
|---|---|
dict | Dictionary containing metadata like 'size_bytes', 'last_modified', 'etag'. |
has_files_with_extension(dir_path, extension) ¶
Check if ANY file with given extension exists (early exit).
Much faster than filtering list_files() result when you only need to know if at least one file exists.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir_path | Pathish | Directory path | required |
extension | str | File extension (e.g., '.png' or 'png') | required |
Returns:
| Type | Description |
|---|---|
bool | True if at least one file with extension exists |
Example
if store.has_files_with_extension('images/', '.png'): print("Folder contains PNG images")
Performance
- Returns immediately on first match
- Best case: <1s (if match in first page)
- Worst case: ~30-60s (no matches, must check all)
is_dir(path) ¶
Checks if path is a directory.
is_file(path) ¶
Checks if path is a file.
list_directories(path) ¶
List only directory names using Azure's hierarchical listing.
list_files(dir_path) ¶
List all files in a directory.
For large directories (100K+ files), consider using list_files_iter() to avoid memory overhead.
Performance improvements in this version: - Uses list_blob_names() internally (8-14x faster) - Lower memory usage (strings vs objects)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir_path | Pathish | Directory path | required |
Returns:
| Type | Description |
|---|---|
list | List of file paths |
list_files_iter(dir_path, results_per_page=None) ¶
Iterate over files with lazy evaluation (memory efficient).
Recommended for large directories. Returns generator that yields file paths one at a time.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir_path | Pathish | Directory path | required |
results_per_page | Optional[int] | Number of results to request per page. | None |
Yields:
| Type | Description |
|---|---|
str | File paths |
Example
for file in store.list_files_iter('data/'): if file.endswith('.png'): process(file) break # Early exit possible
mkdir(path, exist_ok=False) ¶
Create a virtual directory in ADLS.
Directories are conceptual in blob storage and represented by a placeholder blob.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | Pathish | Path of the directory to create. | required |
exist_ok | bool | If False, raise an error if the directory already exists. | False |
Raises:
| Type | Description |
|---|---|
FileExistsError | If directory exists and exist_ok=False. |
open(path, mode='r', encoding=None) ¶
Context manager for file operations with enhanced mode support.
open_stream(path) ¶
Stream blob content from ADLS without loading it fully into memory. Yields an object supporting .read(size); reads are fetched from Azure lazily in chunks, not buffered up front.
read_file(path, encoding=None) ¶
Read file contents from blob storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | Pathish | Path to the file in blob storage. | required |
encoding | Optional[str] | Optional string encoding (e.g., 'utf-8'). If None, returns bytes. | None |
Returns:
| Type | Description |
|---|---|
Union[str, bytes] | File contents as a string if encoding is provided, otherwise bytes. |
Raises:
| Type | Description |
|---|---|
IOError | If there's an error downloading the blob. |
remove(path) ¶
Removes a file.
rename(source_path, destination_path, overwrite=False, delete_source=True, wait=True, timeout_seconds=300, poll_interval_seconds=1) ¶
Rename (move) a single file by copying to the new path and deleting the source.
:param source_path: Existing blob path :param destination_path: Target blob path :param overwrite: Overwrite destination if it already exists :param delete_source: Delete original after successful copy :param wait: Wait for the copy operation to complete :param timeout_seconds: Max time to wait for copy to succeed :param poll_interval_seconds: Polling interval while waiting
rmdir(dir) ¶
Removes a directory.
upload_directory(dir_path, blob_dir_path) ¶
Uploads all files from a directory to Azure Blob Storage.
upload_file(file_path, blob_path) ¶
Uploads a single file to Azure Blob Storage.
walk(top) ¶
Walk through directory tree yielding (dirpath, dirnames, filenames).
Optimized to use list_files_iter() for better performance.
write_file(path, data, overwrite=True, encoding=None) ¶
Write data (str, bytes, or a readable binary stream) to a blob.
Bases: DataStore
An implementation of DataStore for Snowflake internal stages. Uses Snowflake stages for file storage and retrieval.
connection property ¶
Property accessor for connection (for backward compatibility).
This allows existing code that accesses self.connection to continue working, while using lazy initialization internally.
__enter__() ¶
Context manager entry.
__exit__(exc_type, exc_val, exc_tb) ¶
Context manager exit.
__getstate__() ¶
Custom pickling: exclude connection and lock to allow multiprocessing.
When pickled, only store connection parameters (all picklable). The connection and lock will be recreated in the worker process.
__init__(account=config.SNOWFLAKE_ACCOUNT, user=config.SNOWFLAKE_USER, password=config.SNOWFLAKE_PASSWORD, warehouse=config.SNOWFLAKE_WAREHOUSE, database=config.SNOWFLAKE_DATABASE, schema=config.SNOWFLAKE_SCHEMA, stage_name=config.SNOWFLAKE_STAGE_NAME) ¶
Initialize the Snowflake data store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
account | str | Snowflake account identifier. | SNOWFLAKE_ACCOUNT |
user | str | Snowflake username. | SNOWFLAKE_USER |
password | str | Snowflake password. | SNOWFLAKE_PASSWORD |
warehouse | str | Snowflake warehouse name. | SNOWFLAKE_WAREHOUSE |
database | str | Snowflake database name. | SNOWFLAKE_DATABASE |
schema | str | Snowflake schema name. | SNOWFLAKE_SCHEMA |
stage_name | str | Name of the Snowflake stage to use for storage. | SNOWFLAKE_STAGE_NAME |
Note
Connection is created lazily to support multiprocessing. This allows the DataStore to be pickled and sent to worker processes.
Raises:
| Type | Description |
|---|---|
ImportError | If 'snowflake-connector-python' is not installed. |
ValueError | If connection parameters are missing. |
__setstate__(state) ¶
Custom unpickling: restore state without connection.
Connection will be created lazily on first use in the worker process.
_create_connection() ¶
Create and return a Snowflake connection.
_ensure_connection() ¶
Ensure the connection is active, reconnect if needed.
_get_connection() ¶
Get or create the Snowflake connection (lazy initialization).
This method ensures the connection is created on first use, which allows the DataStore to be pickled and sent to worker processes. Each process creates its own connection.
_get_lock() ¶
Get or create the lock for thread safety (created per process).
_get_stage_path(path) ¶
Convert a file path to a Snowflake stage path.
_normalize_path(path) ¶
Normalize path for Snowflake stage operations.
close() ¶
Close the Snowflake connection.
copy_directory(source_dir, destination_dir) ¶
Copies all files from source to destination directory within the stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_dir | str | Source directory path in the stage. | required |
destination_dir | str | Destination directory path in the stage. | required |
copy_file(source_path, destination_path, overwrite=False) ¶
Copies a single file within the Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_path | str | Source file path in the stage. | required |
destination_path | str | Destination file path in the stage. | required |
overwrite | bool | If True, overwrite the destination file if it already exists. | False |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If source file is missing. |
FileExistsError | If destination exists and overwrite=False. |
download_directory(stage_dir_path, local_dir_path) ¶
Downloads all files from a Snowflake stage directory to a local directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stage_dir_path | str | Source directory path in the stage. | required |
local_dir_path | str | Destination local directory path. | required |
exists(path) ¶
Check if a path exists (file or directory).
file_exists(path) ¶
Check if a file exists in the Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Path to check in the stage. | required |
Returns:
| Type | Description |
|---|---|
bool | True if file exists, False otherwise. |
file_size(path) ¶
Get the size of a file in kilobytes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | File path in the stage. | required |
Returns:
| Type | Description |
|---|---|
float | File size in kilobytes. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If file is not found. |
get_file_metadata(path) ¶
Retrieve comprehensive file metadata from Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | File path in the stage. | required |
Returns:
| Type | Description |
|---|---|
dict | Dictionary with keys: 'name', 'size_bytes', 'last_modified', 'md5'. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If file is not found. |
is_dir(path) ¶
Check if path points to a directory.
is_file(path) ¶
Check if path points to a file.
list_directories(path) ¶
List immediate directory names from a given path in the stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Directory path to list. | required |
Returns:
| Type | Description |
|---|---|
List[str] | List of subdirectory names. |
list_files(path) ¶
List all files in a directory within the Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Directory path to list. | required |
Returns:
| Type | Description |
|---|---|
List[str] | List of relative file paths from the stage root. |
mkdir(path, exist_ok=False) ¶
Create a directory in Snowflake stage. Directories are created implicitly on upload; this creates a placeholder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Path of the directory to create. | required |
exist_ok | bool | If False, raise error if directory exists. | False |
Raises:
| Type | Description |
|---|---|
FileExistsError | If directory exists and exist_ok=False. |
open(path, mode='r', encoding=None) ¶
Context manager for opening files in Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | File path in stage. | required |
mode | str | File mode ('r', 'rb', 'w', 'wb'). | 'r' |
encoding | Optional[str] | Optional string encoding. | None |
Returns:
| Type | Description |
|---|---|
| Context manager yielding a file-like object. |
Raises:
| Type | Description |
|---|---|
ValueError | If mode is unsupported. |
read_file(path, encoding=None) ¶
Read file contents from Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Path to the file in the stage. | required |
encoding | Optional[str] | Optional string encoding. If None, returns bytes. | None |
Returns:
| Type | Description |
|---|---|
Union[str, bytes] | File contents as a string if encoding is provided, otherwise bytes. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If file is not found in the stage. |
IOError | If there's an error during download or read. |
remove(path) ¶
Remove a file from the Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Path to the file to remove in stage. | required |
Raises:
| Type | Description |
|---|---|
IOError | If removal fails. |
rename(source_path, destination_path, overwrite=False, delete_source=True) ¶
Rename (move) a single file by copying to the new path and deleting source.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_path | str | Existing file path in the stage. | required |
destination_path | str | Target file path in the stage. | required |
overwrite | bool | Overwrite destination if it exists. | False |
delete_source | bool | Delete original after successful copy. | True |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If source is missing. |
FileExistsError | If destination exists and overwrite=False. |
rmdir(dir) ¶
Remove a directory and all its contents from the Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir | str | Path to the directory to remove in stage. | required |
Raises:
| Type | Description |
|---|---|
IOError | If removal fails. |
upload_directory(dir_path, stage_dir_path) ¶
Uploads all files from a local directory to Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir_path | str | Local directory path. | required |
stage_dir_path | str | Destination directory path in the stage. | required |
Raises:
| Type | Description |
|---|---|
NotADirectoryError | If dir_path is not a directory. |
upload_file(file_path, stage_path) ¶
Uploads a single file from local filesystem to Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path | str | Local file path. | required |
stage_path | str | Destination path in the stage. | required |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If local file is missing. |
walk(top) ¶
Walk through directory tree in Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
top | str | Starting directory for the walk. | required |
Yields:
| Type | Description |
|---|---|
Tuple[str, List[str], List[str]] | Tuples of (dirpath, dirnames, filenames). |
write_file(path, data, encoding=None) ¶
Write data (string or bytes) to a Snowflake stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Destination path in the stage. | required |
data | Union[bytes, str] | Data to write (str or bytes). | required |
encoding | Optional[str] | Optional string encoding. | None |
Raises:
| Type | Description |
|---|---|
ValueError | If data type is unsupported. |
IOError | If there's an error during the upload process. |
General-purpose Delta Sharing data accessor.
cache_size_mb property ¶
Total memory used by cache in megabytes.
client property ¶
Lazy-load Delta Sharing client.
clear_cache(table_name=None) ¶
Clear cached data.
get_cached_tables() ¶
Get list of currently cached table names.
get_table_metadata(table_name) ¶
Retrieve metadata for a table.
list_tables(schema_filter=None, sort=True) ¶
List all available tables in the configured schema.
load_multiple_tables(table_names, filters=None) ¶
Load and concatenate multiple tables.
load_table(table_name, filters=None, use_cache=None) ¶
Load a table from Delta Sharing with optional filtering.
Read data from various file formats stored in local or cloud-based storage.
Supports plain and compressed formats. For compound extensions such as .csv.gz or .geojson.gz, the inner format is detected automatically.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | str | Path to the file in data storage. | required |
data_store | DataStore | DataStore instance for accessing storage. If None, local storage is used. | None |
compression | str | Explicit compression type ("gzip", "bz2", "xz"). If None, inferred from the file extension. | None |
validate_crs | bool | If True, emit a UserWarning when a GeoDataFrame has no CRS. | True |
**kwargs | Additional arguments forwarded to the underlying reader function (e.g., | {} |
Returns:
| Type | Description |
|---|---|
Union[DataFrame, GeoDataFrame] | The data read from the file as a DataFrame or GeoDataFrame. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If the file does not exist in the storage backend. |
ValueError | If the file type is unsupported or the file cannot be parsed. |
RuntimeError | For unexpected errors during the read process. |
Write DataFrame, GeoDataFrame, or a generic object to various file formats.
Supported formats include .csv, .xlsx, .json, .parquet for DataFrames, and .geojson, .gpkg, .parquet for GeoDataFrames.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data | The data to write. Can be pd.DataFrame, gpd.GeoDataFrame, or a JSON-serializable object. | required | |
data_store | DataStore | Instance of DataStore for accessing data storage. If None, local storage is used. | None |
path | Path where the file will be written. | None | |
**kwargs | Additional arguments passed to the specific writer function (e.g., index=False). | {} |
Raises:
| Type | Description |
|---|---|
ValueError | If the file type is unsupported or if there's an error writing the file. |
TypeError | If input data type is incompatible with the file extension. |
RuntimeError | For unexpected errors during the write process. |