genvid.toolbox.AWSTool

class genvid.toolbox.AWSTool(*, default_region=None, **kwargs)

Bases: genvid.toolbox.basetool.BaseTool

Utilities to access and manipulate AWS resources.

Parameters:default_region (str) – Override the default AWS region.

Changed in version 1.20.0: * Updated documentation. * Sessions and instances are now lazily-instantiated as needed. * Every applicable methods now have an optional keyword-only parameter region=None to specify in which region the operation should be carried in. * The class default region can be overridden from the default region defined by the AWS configuration.

Changed in version 1.33.0: Make the AWS dependencies optional.

NAME = 'aws'
DESCRIPTION = 'Utilities to access and manipulate AWS resources.'
AWS_GLOBAL_REGION = 'global'

Key used to refer to AWS global region.

AMI_NAME_TMPL = '{prefix}-{type}-{version}-{timestamp}'

Template of the AMI name.

AMI_NAME_RE = re.compile('(?P<prefix>.*)-(?P<type>[^-]*)-(?P<version>[^-]*)-(?P<timestamp>\\d{8}-\\d{6})')

Regex to match an AMI.

EXITCODE_MESSAGE_RE = re.compile('ExitCode of User Data with PID: (?P<PID>\\d+) is (?P<ExitCode>\\d+)')

Regex to match exit codes of user data scripts.

aws_default_region
Returns:The default region name.
aws_session
Returns:The default AWS session.

Note

For more information on the object returned by this method, see the boto3 Session API documentation.

aws_ec2
Returns:an AWS EC2 resource service client in the default region.

Note

For more information on the object returned by this method, see the boto3 EC2 API documentation.

aws_s3
Returns:an AWS S3 resource service client in the default region.

Note

For more information on the object returned by this method, see the boto3 S3 API documentation.

aws_sts
Returns:an AWS STS client in the default region.

Warning

The request is sent in the default region, not globally. If the request should be sent to sts.amazonaws.com use the aws_sts_global() instead.

Note

For more information on the object returned by this method, see the boto3 STS API documentation.

aws_sts_global
Returns:an AWS STS global client.

Note

For more information on the object returned by this method, see the boto3 STS API documentation.

aws_session_in(region=None)
Returns:An AWS session.
Parameters:region (str) – The region in which the session should operate.

Note

For more information on the object returned by this method, see the boto3 Session API documentation.

aws_ec2_in(region=None)
Returns:an AWS EC2 resource service client.
Parameters:region (str) – The region in which the client should operate.

Note

For more information on the object returned by this method, see the boto3 EC2 API documentation.

aws_s3_in(region=None)
Returns:an AWS S3 resource service client.
Parameters:region (str) – The region in which the client should operate.

Note

For more information on the object returned by this method, see the boto3 S3 API documentation.

aws_sts_in(region=None)
Returns:an AWS STS client.
Parameters:region (str) – The region in which the client should operate.

Warning

If region is None, the request is sent in the default region, not globally. If the request should be sent to sts.amazonaws.com use the aws_sts_global() instead.

Note

For more information on the object returned by this method, see the boto3 STS API documentation.

aws_tag_ami_snapshot(ami, ec2=None, *, region=None)

Copy some AMI tags onto its EBS Snapshots.

The tags that are copied are:

  • Name
  • Version
  • InstanceType
Parameters:
  • ami

    EC2 image whose tags are to be copied to its snapshots.

    Note

    The API for AMIs (EC2 Images) can be accessed from boto3 EC2.Image API documentation.

  • ec2

    An EC2 resource service client to use.

    Deprecated since version 1.20.0: This parameter is redundant with just setting the region parameter and letting AWSTool provide the correct client.

  • region (str) –

    The region in which the snapshots reside.

    Note

    The region parameter is only used if ec2 is None.

static aws_publish_image(ami)

Make an AMI public.

Parameters:ami

EC2 image to operate on.

Note

The API for AMIs (EC2 Images) can be accessed from boto3 EC2.Image API documentation.

aws_list_amis(ami_type: str, owner: str = '', prefix: str = '', version: str = '', *, region=None)

List the available AMIs.

Parameters:
  • ami_type (str) –

    The type of AMI. Can be either:

    • wingame
    • server
  • owner (str) –

    If provided, filters by owner.

    Note

    The value genvidtech is a special alias for the Genvid Technologies Account.

  • prefix (str) –

    Prefix to filter. Can be ‘*’.

    Important

    If left unspecified, it will be set to either:

    • default for the wingame AMI.
    • genvidtech in all other cases.
  • version (str) – Version to filter. Defaults to GENVID_TOOLBOX_VERSION.
  • region (str) – Specify in which region to do the lookup.
Returns:

A list of EC2 Images.

aws_copy_ami(ami_id, regions, public=False, force=False, *, region=None)

Copy an AMI into multiple regions.

Parameters:
  • ami_id – AMI identifier to be looked up for the source.
  • regions – An iterable over destination regions.
  • public (bool) –

    Whether to publish all AMIs.

    Warning

    This also make the source AMI public.

  • force (bool) – Will override destination if it already exists.
  • region (str) – Used to specify the source AMI’s region.
Returns:

A list containing the copies.

aws_rename_ami(ami_id, prefix, public=False, force=False, *, region=None)

Copy an AMI and change its initial prefix.

Parameters:
  • ami_id – AMI identifier to be looked up for the source.
  • prefix

    New prefix for the resulting copy.

    Warning

    The new prefix is going to be used with the AMI_NAME_TMPL for formatting. The resulting name must match the regex defined by AMI_NAME_RE.

  • public (bool) – Whether to publish the resulting AMI.
  • force (bool) – Will override destination if it already exists.
  • region (str) – Used to specify the source AMI’s region.
Returns:

The resulting AMI.

aws_ami_snapshots(ami)

Generator that yields EBS snapshots associated with an EC2 image.

Parameters:ami

EC2 image to operate on.

Note

The API for AMIs (EC2 Images) can be accessed from boto3 EC2.Image API documentation.

New in version 1.13.0.

aws_modify_ami_permissions(ami_id: str, user_ids: typing.Union[typing.List[str], NoneType] = None, *, public: bool = False, region: typing.Union[str, NoneType] = None, remove: bool = False, reset: bool = False)

Set the access permissions on an AMI.

Parameters:
  • ami_id – The ID of the AMI.
  • user_ids – An optional list of Account IDs to set the access to it.
  • public – Add the Group: all permission.
  • region – The region of the AMI. Default to the current region.
  • remove – Removing instead of adding permissions.
  • reset – Reset permissions prior to adding them. Invalid if remove is True.

New in version 1.30.0.

aws_describe_ami_permissions(ami_id: str, *, region: typing.Union[str, NoneType] = None) → typing.Tuple[bool, typing.List[str]]

Set the access permissions on an AMI.

Parameters:
  • ami_id – The ID of the AMI.
  • region – The region of the AMI. Default to the current region.
Returns:

A tuple with a list (possibly empty) of user ids as first member, and if the AMI is public as second member.

New in version 1.30.0.

aws_delete_ami(ami, dryrun: bool = False, *, region=None)

Deregister an image and its associated snapshots.

Note

This method logs all operations.

Parameters:
  • ami

    EC2 image to operate on.

    Note

    The API for AMIs (EC2 Images) can be accessed from boto3 EC2.Image API documentation.

  • dryrun (bool) – Display only. Do not actually remove anything.
  • region (str) – Used to specify the source AMI’s region.

New in version 1.13.0.

aws_format_bucket_key(key: str) → str

Ensure the format of the bucket key is correct.

More specifically, we remove any leading “/” and add a trailing “/” if the key is not empty.

Parameters:key (str) – Key to format.
Returns:Potentially reformated key.

New in version 1.20.0.

aws_upload_file(bucket, key: str, filepath: str, *, region=None)

Upload a file to the specified S3 bucket.

Parameters:
  • bucket

    An S3 bucket to upload the file to.

    Note

    The API for S3 buckets can be accessed from boto3 S3.Bucket API documentation.

    Note

    It is possible to provider a bucket identifier as a str instead. If so, a lookup for that bucket will be handled automatically.

  • key (str) – Key to give to the new file in the bucket.
  • filepath (str) – Path to the file to be uploaded.
  • region (str) – Lookup region if a bucket ID is used.
update_s3_images(prefixes: typing.List[str], bucket_id: str, key_prefix: str, images_folder: typing.Union[pathlib.Path, str, NoneType] = None, *, region=None)

Update images in an S3 bucket.

Warning

File sizes are used to determine if an update is needed. In the unlikely event the new file has the same size as an existing file, it will fail to update.

Parameters:
  • bucket – The S3 bucket identifier to operate on.
  • key_prefix – The prefix key where to upload the files.
  • prefixes

    Update only files with these prefixes.

    Warning

    Every file in the bucket matching one of the prefixes is going to be removed even if it is not being replaced by a new version.

  • images_folder – The directory containing the source images.
  • region (str) – Region in which the bucket resides.

Changed in version 1.20.0: Added the images_folder parameter.

bucket_exist(boto_client, bucket_id: str) → bool

Checks if a bucket exists.

Retrieves a list of all the buckets and check if the one named after the bucket_id parameter.

Parameters:
  • boto_client – A boto3 client instance must be supplied
  • bucket_id – The name of the bucket whose existence we need to validate.
Returns:

Returns True if the bucket exists.

get_bucket_region(bucket_id, boto_client=None) → str

Get the region of a S3 bucket

Parameters:
  • bucket_id – Name of the bucket
  • boto_client – Optional. Will create one if none is supplied.
Returns:

Returns region of a S3 bucket

get_s3_images_bucket_config(bucket: str, key_prefix='/', *, region=None) → dict

Retrieve images configurations from an S3 bucket.

Important

The image names must match DockerTool.RE_IMAGE_IMAGEID to be selected.

Parameters:
  • bucket – The bucket identifier to operate on.
  • key_prefix – Bucket key prefix to match against.
  • region (str) – The region in which the bucket resides.
Returns:

The list of image configurations retrieved.

resolve_s3_domain(bucket_name, region) → str
aws_wait_for_windows_instance(iid: str, timeout: float = 1800, expect: typing.Union[typing.Callable[[str], typing.Union[int, NoneType]], NoneType] = None, *, region=None) → int

Wait for a Windows instance to be ready.

The following conditions must be met for a Windows instance to be considered ready:

  • The instance must be running.
  • The instance must be ok.
  • The provided expect callable should return a value other than None when provided with the latest output from the instance console.

Important

If the callable is None (not provided), then the output to the instance console should match EXITCODE_MESSAGE_RE so that this method can retrieve the exit code of the user data script.

Parameters:
  • iid – The AWS instance ID.
  • timeout – Seconds to wait before raising a RuntimeError.
  • expect – Callable taking the console output as argument and returning either None, if not ready, or an exit code.
  • region (str) – Lookup region for the Windows instance.
Returns:

The return value of expect or the exit code retrieved using the EXITCODE_MESSAGE_RE regex.

New in version 1.19.0.

class aws.AWSTool

Implementation of genvid.toolbox.AWSTool