How to schedule Docker One-Off Tasks on AWS Fargate using Terraform in 4 steps

In this article I would like to describe how you can create a Docker container that can be run as a Task on an AWS Fargate Cluster which basically kills itself after running.

We would all like to schedule reoccuring tasks as easy and efficient as possible. In every complex project this is needed for updating data, ingesting data or similar use cases. With AWS Fargate it is possible to have orchestrated clusters on which you can run your tasks in Docker Containers. Furthermore it is possible to create the infrastructure of a cluster, service and corresponding tasks with terraform as in infrastructure as code. In this article I will show how this is done, how your Dockerfile has to look like in order to just run the python script and therefore also kills the task it is running in after having finished the job. Those are also known as One-Off-Tasks. The whole code base to build and deploy this can be found in the three following repos

• one-off-python-docker-template
• terraform-fargate-one-off-task
• terraform-one-off-task-module

on GitHub and is free to use.

1. Create Docker Image

First we have to create a Docker-image. For this we will need to have a look at how to structure our Dockerfile. As in our case we wanted to run a Python-Procedure we used an image with preinstalled anaconda/miniconda of continuum. Furthermore we need to copy the following files to our docker-container

• run.py (contains your python script)
• environment.yml (contains name and dependencies run.py needs

FROM continuumio/miniconda3:4.6.14
RUN apt-get update
ADD environment.yml /tmp/environment.yml
RUN conda env create -f /tmp/environment.yml
RUN echo "source activate $(head -1 /tmp/environment.yml | cut -d' ' -f2) && conda deactivate" > ~/.bashrc
COPY python  ./home/python/
ENV PATH /opt/conda/envs/$(head -1 /tmp/environment.yml | cut -d' ' -f2)/bin:$PATH
ENV PYTHONPATH="$PYTHONPATH:/home/python-scripts"
RUN bash -c "source ~/.bashrc"
RUN bash -c "source activate $(head -1 /tmp/environment.yml | cut -d' ' -f2)"
ENTRYPOINT ["python", "home/python/run.py"]

In the Dockerfile we create a new conda environment and install all its packages defined in the environment.yml in it.

name: my_env
channels:
  - conda-forge
dependencies:
  - python=3.7
  - numpy=1.16.1
  - pandas=0.24.1
  - psycopg2=2.7.6.1
  - sqlalchemy=1.2.18
  - pip:
    - pg8000==1.13.1

On Entry we start the run.py script. The Docker container will exit after Python script was executed and you will be back at your command line

Lastly, build and test your docker container locally with the following commands

SERVICE_NAME=my_service
IMAGE_TAG="latest"
docker build -t ${SERVICE_NAME}:${IMAGE_TAG} .
docker run -e ENVIRONMENT=local_docker \
           --network="host"\
           -it -t ${SERVICE_NAME}:${IMAGE_TAG}

which are executing the following script (run.py) on entering

import os
ENVIRONMENT = os.environ["ENVIRONMENT"]
print("Docker works in {env}.".format(env=ENVIRONMENT))

with output:

Docker works in local_docker.

The — network flag is here not needed but if you would like to know how you can use your hosting machines connections, this is how its done.

2. Upload Docker-Image to AWS ECR

So, now that you were able to run your Container locally, you might want to push it to ECR to get it up and running on the Fargate Cluster. For that you have to create a new repository on AWS ECR and push the just created image. Then it will be available to launch tasks using it on Fargate.

For the below steps you need to have the AWS CLI installed.

If you setup your aws-profile on your linux machine like this:

AWS_KEY=KEY AWS_SECRET=SECRET AWS_REGION=eu-west-1 LOCAL_PROFILE=my_profile && printf "${AWS_KEY}\n${AWS_SECRET}\n${AWS_REGION}\njson" | aws configure --profile ${LOCAL_PROFILE}

you should be able to run the following code successfully

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')
AWS_REGION=$(aws configure get region)
SERVICE_NAME=my_service
IMAGE_TAG="latest"
docker build -t ${SERVICE_NAME}:${IMAGE_TAG} .
echo "Logging into AWS ECR..."
loggin=$(aws ecr get-login --region ${AWS_REGION} --no-include-email)
${loggin}
aws ecr create-repository --repository-name ${SERVICE_NAME} \
  --region ${AWS_REGION} || echo 'AWS ECR Repository exists: ' \
  && echo '' && aws ecr describe-repositories
  
echo "Tagging image..."
docker tag $SERVICE_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$SERVICE_NAME:$IMAGE_TAG
echo "Pushing into AWS ECR image repository ${SERVICE_NAME}"
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$SERVICE_NAME:$IMAGE_TAG

you can now see the progress of uploading it. You can also confirm in the Amazon Management Console if it was uploaded when you go to the ECR Service.

3. Create Fargate Cluster, Service and Task with Terraform

We have now everything setup regarding the Docker Container. However the most essential part is still missing to run this as a Task on the Fargate Cluster. Yes, you’re right, it is the Fargate Cluster! 😉

The infrastructure we need to run the Docker as a Task is consisting of the following three components:

• Fargate cluster
• service
• task

We have set up all this with terraform and are executing it using a Makefile. If you have already set up a VPC in your AWS account and created private subnets, it should be possible for you to execute the code with only a few changes:

• Bucket name must be changed as it is unique
• add the path to the terraform remote state file
• define variables in variables.tf which needs to be set.
• set your aws default region

You can then use the terraform template to create a cluster, service and tasks corresponding to your needs by simply adjusting the variables it uses. Be aware that you have to already set the image name of your docker-Image such that event scheduler will now from which repository it has to create the task from. You can then apply your changes to your environment in order to deploy.

If you want to access any other resources like RDS Instances make sure to open the ingress of the security group to Fargate tasks

4. Schedule Task using Cloud Watch Event Rules using Terraform

The above repo also contains the code which is needed to schedule the just created task on a regular basis.
Now, your task should be already up and running. Hope it worked out for you, otherwise let me know :)