Platforms

base

Abstract base class for platform (file system and secrets) operations.

Every concrete platform — Local, Fabric, Databricks, AWS — inherits from :class:BasePlatform and implements all abstract methods.

BasePlatform

BasePlatform(cache_ttl: int = 300, **kwargs: Any)

Bases: BaseSecretProvider

Abstract interface for file/directory operations and secret retrieval.

Platforms encapsulate the storage layer differences (local FS, ADLS, S3, DBFS) behind a uniform API so that the rest of the framework never touches os, shutil, or cloud SDKs directly.

Each platform also implements :meth:_fetch_secret so it can serve as its own :class:~datacoolie.core.secret_provider.BaseSecretProvider, using its existing SDK handle (env vars / notebookutils / dbutils / boto3).

16 abstract methods across five categories:

• File I/O — read / write / append / delete text content
• Directory Ops — create / delete / list files / list folders
• Existence Checks — file_exists / folder_exists
• File Management — upload / download / copy / move / get_file_info
• Secrets — _fetch_secret (inherited requirement from BaseSecretProvider)

append_file abstractmethod

append_file(path: str, content: str) -> None

Append text content to an existing file.

If the file does not exist, create it.

Parameters:

Name	Type	Description	Default
path	str	Absolute path or URI for the target file.	required
content	str	Text to append.	required

Raises:

Type	Description
PlatformError	On I/O failure.

copy_file abstractmethod

copy_file(
    src: str, dest: str, *, overwrite: bool = False
) -> None

Copy a file from src to dest.

A normalized source/destination pair that identifies the same file is an idempotent no-op and must not issue a destructive self-copy.

Parameters:

Name	Type	Description	Default
src	str	Source file path.	required
dest	str	Destination file path.	required
overwrite	bool	Allow overwriting an existing file at dest.	False

Raises:

Type	Description
PlatformError	If source does not exist or dest exists without overwrite.

create_folder abstractmethod

create_folder(path: str) -> None

Create a directory (including any missing parents).

No-op if the directory already exists.

Parameters:

Name	Type	Description	Default
path	str	Absolute path or URI.	required

Raises:

Type	Description
PlatformError	On failure.

delete_file abstractmethod

delete_file(path: str) -> None

Delete a file.

No-op if the file does not exist (idempotent).

Parameters:

Name	Type	Description	Default
path	str	Absolute path or URI to delete.	required

Raises:

Type	Description
PlatformError	On I/O failure.

delete_folder abstractmethod

delete_folder(
    path: str, *, recursive: bool = False
) -> None

Delete a directory.

Parameters:

Name	Type	Description	Default
path	str	Absolute path or URI.	required
recursive	bool	If True, delete contents recursively.	False

Raises:

Type	Description
PlatformError	If recursive is False and the directory is non-empty, or on I/O failure.

download_file abstractmethod

download_file(src: str, dest: str) -> None

Download a file from this platform to the local filesystem.

Parameters:

Name	Type	Description	Default
src	str	Source path on this platform.	required
dest	str	Absolute path on the local OS filesystem to write to.	required

Raises:

Type	Description
PlatformError	If the source does not exist or the download fails.

file_exists abstractmethod

file_exists(path: str) -> bool

Return True if the path refers to an existing file.

folder_exists abstractmethod

folder_exists(path: str) -> bool

Return True if the path refers to an existing directory.

get_file_info abstractmethod

get_file_info(path: str) -> FileInfo

Return metadata about a single file or directory.

Returns:

Name	Type	Description
A	FileInfo	class:FileInfo instance with name, path, size,
	FileInfo	modification_time (UTC-aware datetime or None), and
	FileInfo	is_dir.

Raises:

Type	Description
PlatformError	If the path does not exist.

list_files abstractmethod

list_files(
    path: str,
    *,
    recursive: bool = False,
    extension: str | None = None,
) -> list[FileInfo]

List files under path.

Each entry is a :class:FileInfo instance.

Parameters:

Name	Type	Description	Default
path	str	Directory path or URI.	required
recursive	bool	Descend into sub-directories.	False
extension	str | None	Filter by suffix (e.g. ".parquet").	None

Returns:

Type	Description
list[FileInfo]	List of :class:FileInfo objects (directories excluded).

Raises:

Type	Description
PlatformError	If the path does not exist or is not a directory.

list_folders abstractmethod

list_folders(
    path: str, *, recursive: bool = False
) -> list[str]

List immediate (or recursive) sub-directories.

Parameters:

Name	Type	Description	Default
path	str	Directory path or URI.	required
recursive	bool	Descend into sub-directories.	False

Returns:

Type	Description
list[str]	List of directory paths.

Raises:

Type	Description
PlatformError	If path is not a directory.

move_file abstractmethod

move_file(
    src: str, dest: str, *, overwrite: bool = False
) -> None

Move (rename) a file from src to dest.

A normalized source/destination pair that identifies the same file is an idempotent no-op.

Implementations create missing destination parent directories. A failed move retains the source. When overwrite is False, an existing destination is preserved and causes an error.

Parameters:

Name	Type	Description	Default
src	str	Source file path.	required
dest	str	Destination file path.	required
overwrite	bool	Allow overwriting an existing file at dest.	False

Raises:

Type	Description
PlatformError	If source does not exist or dest exists without overwrite.

read_bytes abstractmethod

read_bytes(path: str) -> bytes

Return the entire file contents as bytes.

Implementations MUST NOT use bounded head()-style APIs that truncate large files. Prefer the platform's fastest native unbounded primitive (e.g. s3.get_object for AWS, direct open(path, "rb") for Unity Catalog Volumes, fs.cp to a temp file for other cloud paths).

Suitable for files that fit comfortably in memory.

Raises:

Type	Description
PlatformError	If the file does not exist or cannot be read.

read_file abstractmethod

read_file(path: str) -> str

Read the entire content of a text file.

Parameters:

Name	Type	Description	Default
path	str	Absolute path or URI to the file.	required

Returns:

Type	Description
str	File content as a string.

Raises:

Type	Description
PlatformError	If the file does not exist or cannot be read.

upload_file abstractmethod

upload_file(
    local_path: str, dest: str, *, overwrite: bool = False
) -> None

Upload a file from the local filesystem to the platform.

Parameters:

Name	Type	Description	Default
local_path	str	Absolute path on the local OS filesystem.	required
dest	str	Destination path on this platform.	required
overwrite	bool	Allow overwriting an existing file at dest.	False

Raises:

Type	Description
PlatformError	If local_path does not exist or dest exists without overwrite.

write_bytes abstractmethod

write_bytes(
    path: str, data: bytes, *, overwrite: bool = False
) -> None

Write raw bytes to path.

Symmetric counterpart of :meth:read_bytes. Prefer the platform's fastest native upload primitive. For very large payloads, write to a temp file and call :meth:upload_file directly.

Parameters:

Name	Type	Description	Default
path	str	Destination path or URI.	required
data	bytes	Bytes to write.	required
overwrite	bool	Allow overwriting an existing file at path.	False

Raises:

Type	Description
PlatformError	If the file exists and overwrite is False, or on I/O failure.

write_file abstractmethod

write_file(
    path: str, content: str, *, overwrite: bool = False
) -> None

Write text content to a file.

Parameters:

Name	Type	Description	Default
path	str	Absolute path or URI for the target file.	required
content	str	Text to write.	required
overwrite	bool	If True, overwrite an existing file; otherwise raise when the file already exists.	False

Raises:

Type	Description
PlatformError	If the file exists and overwrite is False, or on I/O failure.

FileInfo dataclass

FileInfo(
    name: str,
    path: str,
    modification_time: Optional[datetime],
    size: int = 0,
    is_dir: bool = False,
)

Metadata for a single file-system entry.

Returned by :meth:BasePlatform.list_files and :meth:BasePlatform.get_file_info. frozen=True makes instances immutable and hashable; slots=True reduces per-instance memory.

local_platform

Local filesystem platform implementation.

Uses pathlib, os, and shutil for all operations. Suitable for local development, testing, and single-node environments.

LocalPlatform

LocalPlatform(
    base_path: str | None = None,
    cache_ttl: int = 300,
    **kwargs: Any,
)

Bases: BasePlatform

Platform backed by the local filesystem.

Also implements :meth:_fetch_secret via os.environ, so it can serve as its own secret provider in local / test environments.

Parameters:

Name	Type	Description	Default
base_path	str | None	Optional root directory. When set, all relative paths are resolved against this root.	None
cache_ttl	int	Secret cache time-to-live in seconds (default 300). Pass 0 to disable caching.	300

read_bytes

read_bytes(path: str) -> bytes

Zero-overhead native read — no temp file.

write_bytes

write_bytes(
    path: str, data: bytes, *, overwrite: bool = False
) -> None

Zero-overhead native write — no temp file.

aws_platform

AWS-first platform facade backed by boto3 and S3-compatible storage.

The implementation is split into private components under platforms._aws. The public class remains stable for AWS S3, while endpoint_url allows the same S3 filesystem contract to be used with MinIO and LocalStack.

AWSPlatform

AWSPlatform(
    bucket: str = "",
    region: str | None = None,
    profile: str | None = None,
    endpoint_url: str | None = None,
    cache_ttl: int = 300,
    **kwargs: Any,
)

Bases: BasePlatform

Platform backed by AWS S3 via boto3.

The storage portion also works with S3-compatible endpoints such as MinIO when endpoint_url is supplied. Secrets Manager, Glue, and Athena remain AWS service integrations and use their normal AWS endpoints by default.

Parameters:

Name	Type	Description	Default
bucket	str	Default S3 bucket name.	''
region	str | None	AWS region for S3 and AWS service operations.	None
profile	str | None	Named AWS profile from ~/.aws/credentials.	None
endpoint_url	str | None	Custom S3 endpoint for MinIO, LocalStack, or similar.	None
cache_ttl	int	Secret cache time-to-live in seconds (default 300). Pass 0 to disable caching.	300

s3 property

s3: Any

Return the cached boto3 S3 client.

boto3_client

boto3_client(service: str, **kwargs: Any) -> Any

Create a boto3 service client using the platform's credentials.

The platform endpoint_url is a storage override and is injected automatically only for S3. Callers may pass an explicit endpoint for any service through kwargs.

clear_cache

clear_cache() -> None

Clear secret field and source-payload caches.

fabric_platform

Portable Microsoft Fabric platform facade.

NotebookUtils remains the preferred backend inside Fabric. External Python runtimes use the Azure Data Lake and Key Vault SDKs with Microsoft Entra credentials while callers keep the same :class:FabricPlatform API.

FabricPlatform

FabricPlatform(
    cache_ttl: int = 300,
    *,
    runtime: FabricRuntime = "auto",
    azure_credential: TokenCredential | None = None,
    **kwargs: Any,
)

Bases: BasePlatform

Access OneLake or ADLS through the best backend for the current runtime.

Parameters:

Name	Type	Description	Default
cache_ttl	int	Secret cache time-to-live in seconds. Pass 0 to disable it.	300
runtime	FabricRuntime	"auto" prefers NotebookUtils when it is usable; "fabric" requires NotebookUtils; "external" always uses Azure SDK clients.	'auto'
azure_credential	TokenCredential | None	Optional Azure TokenCredential for external mode. When omitted, DefaultAzureCredential is created lazily.	None

fs property

fs: Any

Return notebookutils.fs in the native Fabric runtime.

notebookutils property

notebookutils: Any

Return native NotebookUtils; unavailable in explicit external mode.

databricks_platform

Portable Databricks platform facade.

Native Databricks runtimes prefer dbutils and direct POSIX content I/O for Unity Catalog Volumes. Other Python runtimes use WorkspaceClient.files with Databricks unified authentication.

DatabricksPlatform

DatabricksPlatform(
    dbutils: Any | None = None,
    cache_ttl: int = 300,
    *,
    runtime: DatabricksRuntime = "auto",
    workspace_client: Any | None = None,
    **kwargs: Any,
)

Bases: BasePlatform

Access Databricks storage through the best backend for the runtime.

Parameters:

Name	Type	Description	Default
dbutils	Any | None	Optional native Databricks utilities handle.	None
cache_ttl	int	Secret cache time-to-live in seconds. Pass 0 to disable.	300
runtime	DatabricksRuntime	"auto" prefers native Databricks, "databricks" requires it, and "external" always uses the Databricks SDK.	'auto'
workspace_client	Any | None	Optional injected WorkspaceClient for external execution. When omitted, unified authentication is used lazily.	None

dbutils property

dbutils: Any

Return native dbutils; unavailable in external mode.

fs property

fs: Any

Return native dbutils.fs; unavailable in external mode.