Capgemini
Knutsford, United Kingdom
Salary : 15,500 - 19000
Deploy comprehensive cloud infrastructure for various products, including Astronomer Airflow and AccelData environments.- Facilitate cross-functional integration between vendor products and other systems, such as data lakes, storage, and compute services.
- Establish best practices for cloud security, scalability, and performance.
- Manage and configure vendor product deployments, ensuring the setup and maintenance of environments.
- Ensure high availability, scalability, and fault tolerance of Airflow clusters.
- Implement monitoring, alerting, and logging for Airflow and related components.
- Perform upgrades and patches for platform-related components.
- Oversee capacity planning, resource allocation, and optimization of Airflow workers.
- Maintain and configure integrations with source control systems (e.g., GitHub, GitLab) for version control.
- Collaborate with cloud providers (e.g., AWS) for pipeline integration and scaling requirements.
- Identify, design, and implement internal process improvements, including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes.
- Develop infrastructure for optimal extraction, transformation, and loading of data from various sources using AWS and SQL technologies.
- Work with stakeholders, including design, product, and executive teams, to address platform-related technical issues.
- Build analytical tools to leverage the data pipeline, providing actionable insights into key business performance metrics, such as operational efficiency and customer acquisition.