Working of Step Functions
After careful consideration of multiple workflow and data management pattern, we selected Step Functions for our purposes. We have the following important requirements from the selected framework:
- Service Orchestration
- Retry Mechanisms
- Catcher Mechanisms
- Saga Implementation*
- External REST API
- High Throughput
- Overall Sustainable Cost
- Existing Cloud Platform
AWS Step Function supports all the above requirements and is also, already existing on the AWS Cloud. AWS Step Functions supports creation of a use-case or workflow visually using the Workflow Studio. It provided multiple constructs like Start, End, Pass, Parallel, Invoke API Gateway and so on… We may invoke any AWS Service via configuration. Also, It provides a very intuitive way to configure each step in the step function. It also allows to manipulate input, output, and setup the retry and catcher mechanisms very easily. Also, each of the Step Function translated in code or definition to Amazon States Languages (ASL). ASL is similar to JSON.
Access Step Functions via [Login] > [AWS Console] > [Search for Services] > [Step Functions]
[Scope of the Article]
Note: The scope of this document is not to cover every step to create a simple workflow or step function but to understand the most important features that were explored during this Proof of Concept (PoC). We will be discussing Step Functions in terms of its integration with the following:
- API Gateway
- Elastic Beanstalk (EC2)
- Spring Boot / REST
[Deployment Context]
We will discuss this in context of deploying Spring Boot/REST endpoints on Elastic Beanstalk (EC2) that is further exposed via API Gateway.
[Sample Workflow/ASL in AWS Step Function]
Click on [Workflow Studio] to launch the Step Functions visual editor. You may use the drag and drop features here to design your Step Function. It is straightforward, and I will not digress from the scope of the article.
Fig.1 : Sample Workflow (Mock Business Use-Case) / Amazon States Language
[Configuration]
Once you click on any step of the workflow, you will get any or all the following tabs: Configuration, Input, Output, Error Handling. In the above example, you can observe the configuration for the API Gateway REST Endpoint. Each of the specific integration endpoints have a specific way to integrate which can be referenced from: AWS Step Functions - Official Documentation - Use AWS with Other Services
Fig.2 : Mock Transaction (Configuration of AWS API Gateway Invocation)
There is exhaustive documentation available for all facets of AWS Step Functions including the API. Hence, please use Google Search to locate all the required documentation as per your integration need.
[Retry, Backoff Rates]
For each of the steps, the fault tolerance and failure recovery in terms of retrying a service. From the below screen, you can see that we can specify the ‘Interval’, ‘Max Attempts’ and ‘Back off Rate’ for the purposes of a retrying a service.
Fig.3 : Mock Transaction (Retry, Back-Off for AWS API Gateway Invocation)
Retrying a service maybe required due to temporary service downtime, specific business exception, runtime exception, temporary environment error or an error in a dependent service.
[Catcher]
Continuing the same topic – In case of a Total Failure or an Unrecoverable Error, we may need to take some form of a final action of a form notification or clean-up before we proceed out of the step function. Just as in normal programming, Step Functions allows us to perform the same.
Fig.4 : Mock Transaction (Catcher for AWS API Gateway Invocation)
You may observe from the below diagram, you can catch the error and transfer control to the required fallback state. We can also filter/manipulate the result path as shown below.
[Parallel Flow]
Sometimes, there is a need to execute multiple flows in parallel and execution moves to next step only once all the parallel flows are complete. If any of the step in parallel step fails, it may be deemed as the failure of the parallel step. Rest of the options of Parallel Flow are like other actions – Configuration, Retry, Catcher, Input (Filter/Manipulation) and Output (Filter/Manipulation).
Fig.5 : Mock Transaction (Parallel Flow of Execution)
[JSON Path Usage]
JSON Path is used to manipulate/filter input and output in various actions or flows in Step Functions. To learn, understand and test our JSON Path for usage in AWS Step Functions – Developer/Architect may use the following handy link: https://jsonpath.com/
Fig.6 : JSON Path Usage (Show Above is a Mock Business Workflow)
Watch this Space for Part-02 of this Article!
