Connectors

In the following we provide the reference for all currently available connectors.

Amazon

You can install the Amazon connectors through the amazon extra:

pip install stormware[amazon]

Note

All Amazon connectors use the AWS credentials that are obtained through the authentication mechanism described in the Amazon Web Services section of the authentication documentation.

AWS Secrets Manager

class stormware.amazon.secrets.SecretsManager(organization: str | None = None, auth: AWSAuth | None = None)

AWS Secrets Manager connector.

Parameters:

organization – The default organization to use.
auth – The Amazon Web Services authentication manager to use.

__getitem__(key: str) → str: Retrieve the secret under the given key.

Facebook

You can install the Facebook connectors through the facebook extra:

pip install stormware[facebook]

class stormware.facebook.FacebookAds(account_name: str | None = None, secret_key: str = 'stormware-facebook', secret_store: SecretStore | None = None)

Facebook Ads connector.

Parameters:

account_name – The name of the ad account to use.
secret_key – The key of the credentials in the secret store.
secret_store – The secret store to use for retrieving the credentials. Uses the default secret store when not provided.

Authentication

The session credentials are loaded from the secret store using the provided key. The secret must be a string-encoded JSON object with the app_id, app_secret and access_token keys. The app ID and app secret can be obtained by creating a Facebook App (under https://developers.facebook.com/apps/), after which we recommend creating a system user (under https://business.facebook.com/settings/system-users/), which can be then used to generate an access token. Note that the system user must have access to the necessary ad accounts and it must be also added to the appropriate app as an app tester.

account_id(account_name: str | None = None) → str: Return the account ID for a given account name.

Return a Facebook report.

Parameters:

metrics – Numeric fields, see https://developers.facebook.com/docs/marketing-api/insights/parameters#fields.
dimensions – Dimensional fields, see https://developers.facebook.com/docs/marketing-api/insights/parameters#fields.
statistics – Ads action statistics fields, see https://developers.facebook.com/docs/marketing-api/reference/ads-action-stats/.
parameters – Report parameters, see https://developers.facebook.com/docs/marketing-api/insights/parameters#param.
account_name – The ad account name to use.
account_id – The ad account to use. Takes precedence over the account_name argument.

Google

You can install the Google connectors through the google extra:

pip install stormware[google]

Note

All Google connectors use the Google credentials that are obtained through the authentication mechanism described in the Google Cloud Platform section of the authentication documentation.

Google BigQuery

class stormware.google.bigquery.BigQuery(organization: str | None = None, project: str | None = None, auth: GCPAuth | None = None)

Google BigQuery connector.

Must be used with a context manager.

Parameters:

organization – The organization to use.
project – The project to use.
auth – The Google Cloud Platform authentication manager to use.

get_table(name: str) → DataFrame: Return the given table as a data frame.

set_table(name: str, data: DataFrame) → None

Upload the given data frame to a table.

Existing data in the table is dropped.

Google Drive

class stormware.google.drive.DrivePath(*args, **kwargs)

Bases: PurePath

Path object representing Google Drive paths.

Parameters:: *pathsegments – The segments of the path. The first segment must start with // (in case of a shared drive) or / (in case of the user’s own “My Drive” drive).

property drive: str: Name of the shared drive or the empty string (in case of a user drive).

property root: str: Root of the path.

as_uri() → str: Return the path as a ‘file’ URI.

class stormware.google.drive.Drive(organization: str | None = None, project: str | None = None, auth: GCPAuth | None = None)

Google Drive connector.

Must be used with a context manager.

Parameters:

organization – The organization to use.
project – The project to use.
auth – The Google Cloud Platform authentication manager to use. Note that the credentials must be authorized for the https://www.googleapis.com/auth/drive scope.

exists(path: DrivePath, in_trash: bool = False) → bool

Return True if the given path exists.

Parameters:

path – The path to use.
in_trash – Whether to check in the trash or not.

mkdir(path: DrivePath) → DrivePath

Create a folder at the given path (including parent folders).

Parameters:: path – The path to use.

remove(path: DrivePath, missing_ok: bool = False, use_trash: bool = True, in_trash: bool = False) → None

Remove a file or folder in the given path.

Parameters:

path – The path to use.
missing_ok – Whether to raise a FileNotFoundError if the path does not exist.
use_trash – Whether to trash the file or folder instead of permanently deleting it.
in_trash – Whether to work on files or folders that are trashed or not.

upload(src: Path, dst: DrivePath, overwrite: bool | None = None) → DrivePath

Upload a file or directory to Google Drive and return the Google Drive path pointing to it.

Parameters:

src – The local source path to use.
dst – The destination Google Drive folder to use.
overwrite – Whether to trash existing files first.

Google Secret Manager

class stormware.google.secrets.SecretManager(organization: str | None = None, project: str | None = None, auth: GCPAuth | None = None)

Google Cloud Secret Manager connector.

Must be used with a context manager.

Parameters:

organization – The default organization to use.
project – The default project to use.
auth – The Google Cloud Platform authentication manager to use.

__getitem__(key: str) → str: Retrieve the secret under the given key.

Google Sheets

class stormware.google.sheets.Spreadsheet(key: str, organization: str | None = None, project: str | None = None, auth: GCPAuth | None = None)

Google Sheets connector.

Must be used with a context manager.

Parameters:

key – The spreadsheet ID to use.
organization – The organization to use.
project – The project to use.
auth – The Google Cloud Platform authentication manager to use. Note that the credentials must be authorized for the https://www.googleapis.com/auth/spreadsheets scope.

add_sheet(name: str, properties: dict[str, Any] | None = None) → int: Add a new sheet to the spreadsheet.

delete_sheet(name: str, ignore_missing: bool = True) → None: Delete a sheet from the spreadsheet.

get_sheet(name: str) → DataFrame: Return the given sheet as a data frame.

set_sheet(name: str, data: DataFrame) → None

Upload the given data frame to a sheet.

Create a new sheet if necessary. Existing data in the sheet is dropped.

Gmail

Gmail API connector.

class stormware.google.gmail.Label(id: str, name: str | None = None): Represents a label.

class stormware.google.gmail.Attachment(id: str, message_id: str, filename: str, mime_type: str | None): Represents an email message attachment.

class stormware.google.gmail.Message(id: str, thread_id: str | None = None, timestamp: datetime | None = None, labels: list[Label] | None = None, attachments: list[Attachment] | None = None): Represents an email message.

class stormware.google.gmail.Query(text: str = '', sender: str | None = None, to: str | None = None, cc: str | None = None, subject: str | None = None, timestamp_from: datetime | None = None, timestamp_to: datetime | None = None, label: str | None = None, labels: list[Label] | None = None, attachment: bool | None = None): Represents a Gmail search query.

class stormware.google.gmail.Gmail(organization: str | None = None, project: str | None = None, auth: GCPAuth | None = None)

Gmail connector.

Must be used with a context manager.

Parameters:

organization – The organization to use.
project – The project to use.
auth – The Google Cloud Platform authentication manager to use. Note that the credentials must be authorized for the https://www.googleapis.com/auth/gmail.readonly scope.

messages(query: Query, *, user_id: str = 'me') → list[Message]

Load messages that match a given query.

Parameters:

query – The query to use.
user_id – The user ID to use.

message(message: Message, *, user_id: str = 'me') → Message: Load a specific message.

download_attachment(attachment: Attachment, *, dst: Path, filename: str | None = None, overwrite: bool | None = None, user_id: str = 'me') → Path

Download an attachment and return a path to it.

Parameters:

attachment – The attachment information object.
dst – The destination folder to use.
filename – The filename to use. Defaults to using the attachment filename.
overwrite – Whether to overwrite existing files. Raises a FileExistsError by default.
user_id – The user ID to use.