When to use

Batch jobs are ideal for long-running and resource-demanding tasks, such as data-processing and ETL pipelines, training a machine-learning model, etc.

Advantages

• Pay-per-use - You only pay for the compute resources your jobs use.
• Resource flexibility - Whether your job requires 1 CPU or 50 CPUs, 1GiB or 128Gib of memory, the self-managed compute environment always meets your needs by spinning up the optimal instance to run your job.
• Time flexibility - Unlike functions, batch jobs can run indefinitely.
• Secure by default - Underlying environment is securely managed by AWS.
• Easy integration - batch-job can be invoked by events from a wide variety of services.

Disadvantages

• Slow start time - After a job execution is triggered, the job instance is put into an execution queue and can take anywhere from few seconds up to few minutes to start.

Basic usage

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: schedule
          properties:
            scheduleRate: cron(0 14 * * ? *) # every day at 14:00 UTC

(async () => {
  const event = JSON.parse(process.env.STP_TRIGGER_EVENT_DATA);

  // process the event
})();

Container

• Batch jobs execution runs a Docker container inside a fully managed batch environment.
• You can configure the following properties of the container:

Image

• Docker container is a running instance of an Docker image.
• The image for your container can be supplied in 3 different ways:

Built from source code

• Stacktape bundles your source code with all of its dependencies and builds the image for you.
• During deployment, Stacktape pushes the image to the stack's private AWS container registry.

resources:
  mybatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts

Built from a Dockerfile

• Stacktape builds the image using the specified Dockerfile.
• During the deployment, Stacktape pushes the image to the stack's private AWS container registry.
• If you are not familiar with Docker or writing Dockerfiles, you can refer to Docker's official guide on writing Dockerfiles.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          dockerfilePath: src/Dockerfile
          buildContextPath: src

Pre-built ahead of time

• Pre-built image from the Docker registry is used. Stacktape does not build the image for you.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          image: mypublicrepo/my-container
          command: ['index.js']

Environment variables

Most commonly used types of environment variables:

• static - any value (will be stringified)
• result of a custom directive Learn more about directives
• referenced property of another resource (using $Param directive) Learn more about referencing resources
• secret (using $Secret directive) Learn more about using secrets

environment:
  STATIC_ENV_VAR: 'my-env-var'
  DYNAMICALLY_SET_ENV_VAR: "$MyCustomDirective('input-for-my-directive')"
  DB_HOST: "$Param('myPgSql', 'DbInstance::Endpoint.Address')"
  DB_PASSWORD: "$Secret('dbSecret.password')"

Pre-set environment variables

Stacktape pre-sets the following environment variables:

Logging

• Every time your code outputs (prints) something to the stdout or stderr, your log will be captured and stored in a AWS CloudWatch log group.
• You can browse your logs in 2 ways:
  • go to your batch job's log-group in the AWS CloudWatch console. You can use stacktape stack-info command to get a direct link.
  • use stacktape logs command that will print logs to the console

Computing resources

• You can configure the amount of resource your batch job will have access to.
• In addition to CPU and RAM, batch jobs also allow you to configure GPU. To learn more about GPU instances, refer to AWS Docs.
• Behind the scenes, AWS Batch selects an instance type (from the C4, M4, and R4 instance families) that best fits the needs of the jobs with a preference for the lowest-cost instance type (BEST_FIT strategy).

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: batch-jobs/js-batch-job.js
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: schedule
          properties:
            scheduleRate: 'cron(0 14 * * ? *)' # every day at 14:00 UTC

Spot instances

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      useSpotInstances: true

Retries

Timeout

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      timeout: 1200

Storage

• Each batch job instance has access to its own ephemeral storage. It's removed after the batch job finishes processing or fails.
• It has a fixed size of 20GB.
• To store data persistently, consider using Buckets.

Lifecycle process

The lifecycle of your batch job is fully managed. Stacktape leverages 2 extra resources to achieve this:

• Trigger function

  • Stacktape-managed AWS lambda function used to connect event integration to the batch job and start the execution of the batch job state machine

• Batch job state machine

  • Stacktape-managed AWS State machine used to control the lifecycle of the batch job container.

Batch job execution flow:

1. Trigger function receives the event from one of its integrations
2. Trigger function starts the execution of the batch job state machine
3. Batch job state machine spawns the batch job instance (Docker container) and controls its lifecycle.

Trigger events

• Batch jobs are invoked ("triggered") in a reaction to an event.
• Each batch job can have multiple event integrations.
• Payload (data) received by the batch job depends on the event integration. It is accessible using the STP_TRIGGER_EVENT_DATA environment variable as a JSON stringified value.

HTTP Api event

• The batch job is triggered in a reaction to an incoming request to the specified HTTP API Gateway.
• HTTP API Gateway selects the route with the most-specific match. To learn more about how paths are evaluated, refer to AWS Docs

resources:
  myHttpApi:
    type: http-api-gateway

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: http-api-gateway
          properties:
            httpApiGatewayName: myHttpApi
            path: /hello
            method: GET

Cognito authorizer

• Using Cognito authorizer allows only the users authenticated with User pool to access your batch job.
• Request must include access token (specified as a bearer token, { Authorization: "<<your-access-token>>"" })
• If the request is successfully authorized, your batch job will receive some authorization claims in its payload. To get more information about the user, you can use getUser API Method
• HTTP API uses JWT(JSON Web token)-based authorization. To lean more about how requests are authorized, refer to AWS Docs.

resources:
  myGateway:
    type: http-api-gateway

  myUserPool:
    type: user-auth-pool
    properties:
      userVerificationType: email-code

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: http-api-gateway
          properties:
            httpApiGatewayName: myGateway
            path: /some-path
            method: '*'
            authorizer:
              type: cognito
              properties:
                userPoolName: myUserPool

import { CognitoIdentityProvider } from '@aws-sdk/client-cognito-identity-provider';

const cognito = new CognitoIdentityProvider({});

(async () => {
  const event = JSON.parse(process.env.STP_TRIGGER_EVENT_DATA);
  const userData = await cognito.getUser({ AccessToken: event.headers.authorization });
  // do something with your user data
})();

Lambda authorizer

• When using Lambda authorizer, a special lambda function determines if the client can access your batch job.
• When a request arrives to the HTTP API Gateway, lambda authorizer function is invoked. It must return either a simple response indicating if the client is authorized

{
  "isAuthorized": true,
  "context": {
    "exampleKey": "exampleValue"
  }
}

or an IAM Policy document (when the iamReponse property is set to true, you can further configure permissions of the target batch job)

{
  "principalId": "abcdef", // The principal user identification associated with the token sent by the client.
  "policyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "execute-api:Invoke",
        "Effect": "Allow|Deny",
        "Resource": "arn:aws:execute-api:{regionId}:{accountId}:{apiId}/{stage}/{httpVerb}/[{resource}/[{child-resources}]]"
      }
    ]
  },
  "context": {
    "exampleKey": "exampleValue"
  }
}

• Data returned in the context property will be available to the batch job.
• You can configure identitySources that specify the location of data that's required to authorize a request. If they are not included in the request, the Lambda authorizer won't be invoked, and the client receives a 401 error. The following identity sources are supported: $request.header.name, $request.querystring.name and $context.variableName.
• When caching is enabled for an authorizer, API Gateway uses the authorizer's identity sources as the cache key. If a client specifies the same parameters in identity sources within the configured TTL, API Gateway uses the cached authorizer result, rather than invoking the authorizer Lambda function.
• By default, API Gateway uses the cached authorizer response for all routes of an API that use the authorizer. To cache responses per route, add $context.routeKey to your authorizer's identity sources.
• To learn more about Lambda authorizers, refer to AWS Docs

Schedule event

The batch job is triggered on a specified schedule. You can use 2 different schedule types:

• Fixed rate - Runs on a specified schedule starting after the event integration is successfully created in your stack. Learn more about rate expressions
• Cron expression - Leverages Cron time-based scheduler. Learn more about Cron expressions

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        # invoke function every two hours
        - type: schedule
          properties:
            scheduleRate: rate(2 hours)
        # invoke function at 10:00 UTC every day
        - type: schedule
          properties:
            scheduleRate: cron(0 10 * * ? *)

Event Bus event

The batch job is triggered when the specified event bus receives an event matching the specified pattern.

2 types of event buses can be used:

• Default event bus

  • Default event bus is pre-created by AWS and shared by the whole AWS account.
  • Can receive events from multiple AWS services. Full list of supported services.
  • To use the default event bus, set the useDefaultBus property.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: event-bus
          properties:
            useDefaultBus: true
            eventPattern:
              source:
                - 'aws.autoscaling'
              region:
                - 'us-west-2'

• Custom event bus
  • Your own, custom Event bus.
  • This event bus can receive your own, custom events.
  • To use custom event bus, specify either eventBusArn or eventBusName property.

resources:
  myEventBus:
    type: event-bus

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: event-bus
          properties:
            eventBusName: myEventBus
            eventPattern:
              source:
                - 'mycustomsource'

SNS event

The batch job is triggered every time a specified SNS topic receives a new message.

• Amazon SNS is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.
• Messages (notifications) are published to the topics
• To add your custom SNS topic to your stack, add Cloudformation resource to the cloudformationResources section of your config.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: sns
          properties:
            topicArn: $Param('mySnsTopic', 'Arn')
            onDeliveryFailure:
              sqsQueueArn: $Param('mySnsTopic', 'Arn')
              sqsQueueUrl: $Param('mySqsQueue', 'QueueURL')

cloudformationResources:
  mySnsTopic:
    Type: AWS::SNS::Topic

  mySqsQueue:
    Type: AWS::SQS::Queue

SQS event

The function is triggered whenever there are messages in the specified SQS Queue.

• Messages are processed in batches
• If the SQS queue contains multiple messages, the batch job is invoked with multiple messages in its payload
• A single queue should always be "consumed" by a workload. SQS message can only be read once from the queue and while it's being processed, it's invisible to other workloads. If multiple different workloads are processing messages from the same queue, each will get their share of the messages, but one message won't be delivered to more than one workload at a time. If you need to consume the same message by multiple consumers (Fanout pattern), consider using EventBus integration or SNS integration.
• To add your custom SQS queue to your stack, simply add Cloudformation resource to the cloudformationResources section of your config.

Batching behavior can be configured. The batch job is triggered when any of the following things happen:

• Batch window expires. Batch window can be configured using maxBatchWindowSeconds property.
• Maximum Batch size (amount of messages in the queue) is reached. Batch size can be configured usingbatchSize property.
• Maximum Payload limit is reached. Maximum payload size is 6 MB.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: sqs
          properties:
            queueArn: $Param('mySqsQueue', 'Arn')

cloudformationResources:
  mySqsQueue:
    Type: AWS::SQS::Queue

Kinesis event

The batch job is triggered whenever there are messages in the specified Kinesis Stream.

• Messages are processed in batches.
• If the stream contains multiple messages, the batch job is invoked with multiple messages in its payload.
• To add a custom Kinesis stream to your stack, simply add Cloudformation resource to the cloudformationResources section of your config.
• Similarly to SQS, Kinesis is used to process messages in batches. To learn the differences, refer to AWS Docs

Batching behavior can be configured. The batch job is triggered when any of the following things happen:

• Batch window expires. Batch window can be configured using maxBatchWindowSeconds property.
• Maximum Batch size (amount of messages in the queue) is reached. Batch size can be configured usingbatchSize property.
• Maximum Payload limit is reached. Maximum payload size is 6 MB.

Consoming messages from a kinesis stream can be done in 2 ways:

• Consuming directly from the stream - polling each shard in your Kinesis stream for records once per second. Read throughput of the kinesis shard is shared with other stream consumers.
• Consuming using a stream consumer - To minimize latency and maximize read throughput, use "stream consumer" with enhanced fan-out. Enhanced fan-out consumers get a dedicated connection to each shard that doesn't impact other applications reading from the stream. You can either pass reference to the consumer using consumerArn property, or you can let Stacktape auto-create consumer using autoCreateConsumer property.

resources:
  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: kinesis
          properties:
            autoCreateConsumer: true
            maxBatchWindowSeconds: 30
            batchSize: 200
            streamArn: $Param('myKinesisStream', 'Arn')
            onFailure:
              arn: $Param('myOnFailureSqsQueue', 'Arn')
              type: sqs

cloudformationResources:
  myKinesisStream:
    Type: AWS::Kinesis::Stream
    Properties:
      ShardCount: 1

  myOnFailureSqsQueue:
    Type: AWS::SQS::Queue

DynamoDb event

The batch job is triggered whenever there are processable records in the specified DynamoDB streams.

• DynamoDB stream captures a time-ordered sequence of item-level modifications in a DynamoDB table and durably stores the information for up to 24 hours.
• Records from the stream are processed in batches. This means that multiple records are included in a single batch job invocation.
• DynamoDB stream must be enabled in a DynamoDB table definition. Learn how to enable streams in dynamo-table docs

resources:
  myDynamoDbTable:
    type: dynamo-db-table
    properties:
      primaryKey:
        partitionKey:
          attributeName: id
          attributeType: string
      dynamoStreamType: NEW_AND_OLD_IMAGES

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: dynamo-db
          properties:
            streamArn: $Param('myDynamoDbTable', 'DynamoTable::StreamArn')
            # OPTIONAL
            batchSize: 200

S3 event

The batch job is triggered when a specified event occurs in your bucket.

• Supported events are listed in the s3EventType API Reference.

• To learn more about the even types, refer to AWS Docs.

resources:
  myBucket:
    type: bucket

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: s3
          properties:
            bucketArn: $Param('myBucket', 'Bucket::Arn')
            s3EventType: 's3:ObjectCreated:*'
            filterRule:
              prefix: order-
              suffix: .jpg

Cloudwatch Log event

The batch job is triggered when a log record arrives to the specified log group.

• Event payload arriving to the batch job is BASE64 encoded and has the following format: { "awslogs": { "data": "BASE64ENCODED_GZIP_COMPRESSED_DATA" } }
• To read access the log data, event payload needs to be decoded and decompressed first.

resources:
  myLogProducingLambda:
    type: function
    properties:
      packageConfig:
        filePath: lambdas/log-producer.ts

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: cloudwatch-log
          properties:
            logGroupArn: $Param('myLogProducingLambda', 'LogGroup::Arn')

Application Load Balancer event

The batch job is triggered when a specified Application load Balancer receives an HTTP request that matches the integration's conditions.

• You can filter requests based on HTTP Method, Path, Headers, Query parameters, and IP Address.

resources:
  # load balancer which routes traffic to the function
  myLoadBalancer:
    type: application-load-balancer
    properties:
      listeners:
        - port: 80
          protocol: HTTP

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      resources:
        cpu: 2
        memory: 1800
      events:
        - type: application-load-balancer
          properties:
            # referencing load balancer defined above
            priority: 1
            loadBalancerName: myLoadBalancer
            listenerPort: 80
            paths:
              - /invoke-my-lambda
              - /another-path

Accessing other resources

• For most of the AWS resources, resource-to-resource communication is not allowed by default. This helps to enforce security and resource isolation. Access must be explicitly granted using IAM (Identity and Access Management) permissions.

• Access control of Relational Databases is not managed by IAM. These resources are not "cloud-native" by design and have their own access control mechanism (connection string with username and password). They are accessible by default, and you don't need to grant any extra IAM permissions. You can further restrict the access to your relational databases by configuring their access control mode.

• Stacktape automatically handles IAM permissions for the underlying AWS services that it creates (i.e. granting functions permission to write logs to Cloudwatch, allowing functions to communicate with their event source and many others).

If your workload needs to communicate with other infrastructure components, you need to add permissions manually. You can do this in 2 ways:

Using allowAccessTo

• List of resource names that this function will be able to access (basic IAM permissions will be granted automatically). Granted permissions differ based on the resource.
• Works only for resources managed by Stacktape (not arbitrary Cloudformation resources)
• This is useful if you don't want to deal with IAM permissions yourself. Handling permissions using raw IAM role statements can be cumbersome, time-consuming and error-prone.

resources:
  photosBucket:
    type: bucket

  myBatchJob:
    type: batch-job
    properties:
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      accessControl:
        allowAccessTo:
          - photosBucket

Using iamRoleStatements

• List of raw IAM role statement objects. These will be appended to the function's role.
• Allow you to set granular control over your function's permissions.
• Can be used to give access to any Cloudformation resource

resources:
  myBatchJob:
    type: batch-job
    properties:
      resources:
        cpu: 2
        memory: 1800
      container:
        imageConfig:
          filePath: path/to/my/batch-job.ts
      accessControl:
        iamRoleStatements:
          - Resource:
              - $Param('NotificationTopic', 'Arn')
            Effect: Allow
            Action:
              - 'sns:Publish'

cloudformationResources:
  NotificationTopic:
    Type: AWS::SNS::Topic

Default VPC connection

• Certain AWS services (such as MongoDb Atlas Clusters) must be connected to a VPC (Virtual private cloud) to be able to run. For stacks that include these resources, Stacktape does 2 things:
  • creates a default VPC
  • connects the VPC-requiring resources to the default VPC.
• Batch jobs are connected to the default VPC of your stack by default. This means that batch jobs can communcate with resources that have their accessibility mode set to vpc without any extra configuration.
• To learn more about VPCs and accessibility modes, refer to VPC docs, accessing relational databases, accessing redis clusters and accessing MongoDb Atlas clusters.

Pricing

• You are charged for the instances running in your batch job compute environment.
• Instance sizes are automatically chosen to best suit the needs of your batch jobs.
• You are charged only for the time your batch job runs. After your batch job finishes processing, the instances are automatically killed.
• Price depends on region and instance used. (https://aws.amazon.com/ec2/pricing/on-demand/)
• You can use spot instances to save costs. These instances can be up to 90% cheaper. (https://aws.amazon.com/ec2/spot/pricing/)
• You are also paying a very neglibile price for lambda functions and state machines used to manage the execution and integrations of your batch job.

API reference