genvid.toolbox.SDK

class genvid.toolbox.SDK(cluster_id='', **kwargs)

Bases: genvid.toolbox.aws.AWSTool, genvid.toolbox.consul_template.ConsulTemplateTool

Common commands for setting up the SDK.

This class wraps the ClusterAPI and BastionAPI for setting up a new Genvid SDK into the current cluster.

Changed in version 1.29.0: Moved load_config_template() to base class ConsulTemplateTool.

Changed in version 1.29.0: Moved merge_dict() to base class BaseTool.

Changed in version 1.30.0: Derived from ProjectTool instead of its ConsulTemplateTool ancestor.

Changed in version 1.30.0: Removed VaultTool and ConsulTool from the ancestry since they were already part of ConsulTemplateTool.

Changed in version 1.30.0: Moved load_nomad_job_template() to base class ProjectTool.

Changed in version 1.31.0: Moved update_config() to base class ProjectTool.

Changed in version 1.33.0: Loaded the binaries paths for a local cluster from ROOTDIR/local-services/init folder, instead of hard coded. Also move the initialization from the static folders from setup_jobs() to setup().

Changed in version 1.33.0: Replace ProjectTool with ConfigurationLoader, now inheriting from ConsulTemplateTool instead.

NAME = 'sdk'
SDK_CONFIG_VERSION = '1.7.0'
cluster_id = None

The Cluster ID.

bastionAPI = None

Interface to the Bastion-API instance.

clusterAPI = None

Interface to the cluster Cluster-API instance.

cluster_vault_secret = None

Key in the bastion’s Vault where the cluster vault keys are saved.

azure
cloud_provider
get_remote_cluster_conn()

Return the cluster IP and private key.

get_remote_cluster_secrets()

Returns the cluster vault initialization secrets.

setup(bucket: str = None, path: str = '/', *, region=None)

Starts the base SDK services.

Parameters:
  • bucket – The name of the bucket. Defaults to the return of get_s3_bucket_id().
  • path – The prefix key in the bucket for the images.
setup_remote(cluster_id: str, server_ip: str, ssh_private_key_pem_file: str, ssh_user: str)
clean()

Stops the base SDK services, removes jobs and clean the config.

get_sdk_config() → dict

Get the default SDK configuration.

The configuration is found either in ROOTDIR/local-services/config if the cluster_id is local, or ROOTDIR/cloud-services/config otherwise. The config is loaded and parsed by consul-template first, and the jobs templates are expected to be found in a sibling folder named templates.

Changed in version 1.31.0: Load config from a folder.

get_config() → dict

Get the current cluster configuration.

set_config(config: dict = None)

Merge the specified configuration with the current one. :param config: The configuration to merge.

clean_config()

Clean the configuration.

remove_config(config: dict)

Remove values from the configuration. :param config: The configuration to remove values from.

static update_dict_no_create(source, dest)
is_jobs_running()

Return True if a job is configured to be running.

start_jobs(jobs: typing.Union[typing.List[str], NoneType] = None)

Start all jobs or the specified job. :param jobs: The list of jobs to start.

stop_jobs(jobs: typing.Union[typing.List[str], NoneType] = None)

Stop all jobs or the specified job.

Parameters:jobs – The list of jobs to stop.
default_images_sdk_path

Get default S3 key prefix for SDK images.

New in version 1.20.0.

upload_images_sdk(bucket=None, path=None, update_config: bool = False, cloud_provider: typing.Union[str, NoneType] = None, *, region=None)

Upload the images for the SDK.

Parameters:
  • bucket (str) – Bucket ID (default: get_s3_bucket_id()).
  • path – Key prefix to use in the bucket.
  • update_config (bool) – Update cluster images configuration.
  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Default path changes from ‘/’ to ‘/images/sdk-{version}/’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

    Changed in version 1.26.0:
  • Added parameter region to target another region for the S3 bucket where the image will be uploaded.

  • If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.

  • If it already exists but in a different region, the underlying code will raise an exception.

  • By default, the region used to create the bucket will be the region of the current AWS profile.

  • It can also be overriden by the AWS_DEFAULT_REGION environment variable.

  • If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

get_s3_bucket_id()

Get the S3 bucket ID for images.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

get_s3_bucket_with_bastion_id(bastion_id: str)

Get the S3 bucket ID for images using the current AWS account and the Bastion ID.

Note

This method makes a global STS request to AWS. We disallow setting the region for this call to avoid confusion with the region of the bucket itself.

update_images_config(bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', *, delete: bool = False, region: typing.Union[str, NoneType] = None)

Update the images configuration on the cluster.

Parameters:
  • bucket – Bucket ID (default: get_s3_bucket_id()).
  • path – Key prefix to use in the bucket.
  • delete – Do not update configuration, delete it instead.
  • region – Region containing the S3 bucket.

Warning

This method can only be called on a non-local cluster.

New in version 1.20.0.

upload_images(prefixes: typing.List[str], bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', update_config: bool = False, basedir: typing.Union[pathlib.Path, str, NoneType] = None, cloud_provider: typing.Union[str, NoneType] = None, *, region: typing.Union[str, NoneType] = None)

Upload the images with the specified prefixes.

Note

The image names must match DockerTool.RE_IMAGE_IMAGEID.

Parameters:
  • prefixes – Only upload the files with these prefixes.
  • bucket (str) – Bucket ID (default: get_s3_bucket_id()).
  • path – Key prefix to use in the bucket.
  • update_config (bool) – Update cluster images configuration.
  • basedir – Source directory (default: ARTIFACTS_DIR).
  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

    Changed in version 1.20.0:
  • Added the basedir parameter.

  • Augmented error handling.

  • New region parameter following the conventions in AWSTool.

  • Factored the update_config part of the method into its own method.

    Changed in version 1.26.0:
  • Added parameter region to target another region for the S3 bucket where the image will be uploaded.

  • If the bucket, named after the Bastion Id, does not exist, it will be created in the targeted region.

  • If it already exists but in a different region, the underlying code will raise an exception.

  • By default, the region used to create the bucket will be the region of the current AWS profile.

  • It can also be overriden by the AWS_DEFAULT_REGION environment variable.

  • If the region parameter is None and the bucket region already exists but its region differs from the one of the profile or AWS_DEFAULT_REGION environment variable, it will simply be used.

create_nats_window_package()

Creates nats-window package if it does not exist.

Changed in version 1.32.0: Make it run on Linux using the binaries file.

get_images_config(bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', *, region: typing.Union[str, NoneType] = None) → dict
Retrieve images configurations from an S3 bucket or azure container
based on cloud provider.
Parameters:
  • bucket – Bucket ID .
  • path – Key prefix in the bucket (default: ‘/images/custom’).
  • region (str) – Region containing the S3 bucket/Azure Container.

Important

The image names must match DockerTool.RE_IMAGE_IMAGEID to be selected.

Returns:Images configurations as a dict.

Changed in version 1.14.0: Changed the default path from ‘/’ to ‘/images/custom’.

Changed in version 1.20.0: New region parameter following the conventions in AWSTool.

clean_images(bucket: typing.Union[str, NoneType] = None, path: str = '/images/custom', update_config: bool = False, *, region: typing.Union[str, NoneType] = None)

Remove all images from an S3 bucket.

Parameters:
  • bucket – Bucket ID (default: get_s3_bucket_id()).
  • path – Key prefix to use in the bucket.
  • update_config (bool) – Update cluster images configuration.
  • region (str) – Region containing the S3 bucket.

Warning

update_config can only be set for non-local clusters.

Changed in version 1.14.0: Change the default path from ‘/’ to ‘/images/custom’.

    Changed in version 1.20.0:
  • New region parameter following the conventions in AWSTool.

  • Now delegates to update_images_config() for update_config.

update_archive(archive: typing.Union[pathlib.Path, str], version: str = 'dev', destdir: typing.Union[pathlib.Path, str, NoneType] = None, clean: bool = True, keep: bool = False)

Update a binary archive to conform to the image format expected.

param archive:The path to the archive file. The filename should always be in the form of name.archfmt.
param version:Version of the archive.
param destdir:Destination directory. Default to the same as the original archive.
param clean:Clean old versions of the archive (the ones with the same name).
param keep:Keep the original archives. You should almost always used it with a different destdir.

New in version 1.20.0.

Deprecated since version 1.30.0: Method moved to PackageArchiver

load_config(source: typing.Union[pathlib.Path, str], *, job_template_dir: typing.Union[pathlib.Path, str, NoneType], with_consul_template: bool, dryrun: bool = False) → dict

Loads the configuration file(s) (all files in a directory or a specific file) from the source.

Parameters:
  • source – Path to the configuration file/folder.
  • with_consul_template – Execute consul-template on the configuration file(s) (if source is a directory) before using it only if it is set to true.
  • job_template_dir – Path to Folder from where nomad templates should be loaded. If not mentioned, no jobs will be loaded.
  • dryrun – If True, only parse the source and returns the configuration.
Returns:

The configuration as a dictionary that can be used directly with set_config().

:raises FileNotFoundError : If the job_template_dir is not None and
is not a valid directory.

Changed in version 1.26.0: Fix support for pathlib.Path for arguments. Add load parameter

Changed in version 1.30.0: Refactored to use load_config_folder() instead of creating a new instance of ProjectTool.

Changed in version 1.33.0: Use the new render_template command instead of the local consul-template.

get_output_public_ips() → list

Retrieves all public ip’s from output of terraform module.

Returns:List of all public IP Addressses
get_cloud_provider() → str

Retrieves the cloud provider from output of terraform module.

If no cloud provider is found in terraform’s cluster output, aws is returned as default.

Returns:Cloud provider.
load_config_template(template: typing.Union[pathlib.Path, str], env: typing.Union[typing.Mapping[str, str], NoneType] = None) → typing.Any

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

consul_template_once(*, use_consul: bool = False, use_vault: bool = False, env: typing.Union[typing.Mapping[str, str], NoneType] = None, **kwargs) → genvid.toolbox.consul_template.ConsulTemplateOutput

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

consul_template

Deprecated since version 1.33.0: ConsulTemplateTool will no longer be part of the SDK ancestors. Use render_template() instead.

class sdk.SDK

Implementation of genvid.toolbox.SDK