Argo Workflows: Container-native workflow orchestration

Container-Native Workflow Orchestration with Argo Workflows

Client Details

The customer is a major German financial institution having its own data analytics and data services platform.

Challenges

The customer had the following use case cases for container-native orchestration:

The customer needed batch processing jobs where large datasets are processed in batches. The solution needs to read data from a Google Cloud SQL database and perform transformations by using R-, Java- and Python code. The solution should be triggered event based as well as with schedule.

The customer wished to generate automated cost reports based on Google Billing data. The workflows needed to orchestrate the necessary steps to gather data, process it, generate insights through analytics, and even distribute these insights via reports or dashboards at scheduled intervals or on-demand.

The solution should be cost-effective, provide a GUI and have multi-tenancy.

Solution

Argo Workflows was used as an orchestration engine. Dedicated Argo Workflows instances have been setup for the different stages (Test, Integration, Production). Argo Workflows have been integrated into the Google Cloud ecosystems by means of Argo Events. Events can be triggered by means of Pub/Sub messages or Cloud Storage object notifications. SSO has been enabled with Azure Entra ID integration. The solution provides multi-tenancy with different teams running their own set of workflows. The solution is deployed highly available and is scalable to perform many workflows runs at the same time. The whole deployment is automated with Terraform and Helm charts and can be deployed in a couple of minutes. In addition, the Argo instance is persisted to Cloud SQL.

The following picture shows the Deployment Architecture

Why Our Clients Trust Us

  • Tailored Solutions that Deliver Real Business Value
  • Nearly 15 years of experience in implementing, operating, and building expertise across a wide range of IT solutions.
  • Award-Winning Services: High commitment and comprehensive skills.

Get in Touch

Do you have questions or want to work with us? Our team is available Monday to Friday from 9:00 AM to 6:00 PM!