Container-Native Workflow Orchestration with Argo Workflows
Client Details
The customer is a major German financial institution having its own data analytics and data services platform.
Challenges
The customer had the following use case cases for container-native orchestration:
The customer needed batch processing jobs where large datasets are processed in batches. The solution needs to read data from a Google Cloud SQL database and perform transformations by using R-, Java- and Python code. The solution should be triggered event based as well as with schedule.
The customer wished to generate automated cost reports based on Google Billing data. The workflows needed to orchestrate the necessary steps to gather data, process it, generate insights through analytics, and even distribute these insights via reports or dashboards at scheduled intervals or on-demand.
The solution should be cost-effective, provide a GUI and have multi-tenancy.
Solution
Argo Workflows was used as an orchestration engine. Dedicated Argo Workflows instances have been setup for the different stages (Test, Integration, Production). Argo Workflows have been integrated into the Google Cloud ecosystems by means of Argo Events. Events can be triggered by means of Pub/Sub messages or Cloud Storage object notifications. SSO has been enabled with Azure Entra ID integration. The solution provides multi-tenancy with different teams running their own set of workflows. The solution is deployed highly available and is scalable to perform many workflows runs at the same time. The whole deployment is automated with Terraform and Helm charts and can be deployed in a couple of minutes. In addition, the Argo instance is persisted to Cloud SQL.
The following picture shows the Deployment Architecture
Warum unsere Kunden uns vertrauen
Maßgeschneiderte Lösungen, die echte Geschäftswerte schaffen
Fast 15 Jahre Erfahrung in der Implementierung, dem Betrieb und dem Wissensaufbau von einer breiten Palette an IT-Lösungen
Preisgekrönte Dienstleistungen: Hohes Engagement und umfassende Fähigkeiten
Nehmen Sie Kontakt mit uns auf
Haben Sie Fragen oder wollen mit uns zusammenarbeiten? Unser Team steht Ihnen Montag bis Freitag von 09:00 – 18:00 Uhr zur Verfügung!
- +49 911 424 688 30
- anfrage@soeldner-consult.de
