Cloud Platform Engineer

Job Title: Cloud Platform EngineerExperience: 7+ YearsEmployment Type: Full-TimeWork Mode: Remote

Job Summary

We are seeking a highly skilled Cloud Platform Engineer / Cloud Data Engineer to join our team for a Cloud Data Modernization initiative.

The ideal candidate will have strong hands-on experience with Databricks, Azure Cloud data services, Apache Spark, PySpark, and Snowflake, with a proven background in migrating and modernizing on-premises ETL workloads to cloud-based data platforms.

Databricks is the primary data technology, with Snowflake as a secondary data platform. Candidates should also have experience with Azure and/or AWS cloud infrastructure and data services.

The role will involve designing and implementing modern cloud data solutions, developing scalable data pipelines, supporting ETL migrations, implementing DevOps and CI/CD practices, and ensuring data quality, security, observability, and performance.

Key Responsibilities

Contribute to the migration and modernization of on-premises ETL workloads to Azure Cloud, Databricks, and Snowflake.

Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Spark, PySpark, and Python.

Develop data pipelines using Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, ADLS Gen2, Blob Storage, and Snowflake.

Review, analyze, and document existing on-premises ETL processes and data workflows.

Translate legacy ETL processes into scalable and maintainable cloud-native solutions.

Develop and optimize data ingestion, transformation, integration, and processing pipelines.

Implement DevOps practices and CI/CD pipelines using GitHub Actions.

Follow Git branching strategies, pull requests, code reviews, and software engineering best practices.

Collaborate with architects, data engineers, application teams, QA, business stakeholders, and other cross-functional teams.

Tune and optimize Databricks, Spark, PySpark, and Snowflake workloads for performance and scalability.

Implement data quality validation, monitoring, reconciliation, and observability processes.

Support automated data validation and reconciliation frameworks.

Implement appropriate logging, monitoring, alerting, and operational processes for data pipelines.

Ensure data solutions comply with organizational security, privacy, governance, and regulatory standards.

Participate in Agile ceremonies, sprint planning, technical discussions, and delivery activities.

Troubleshoot production issues and proactively identify performance, reliability, and data-quality problems.

Leverage AI-assisted development tools to improve engineering productivity, code quality, testing, and documentation.

Work independently, manage ambiguity, prioritize tasks, and deliver high-quality solutions within established timelines.

Primary Skills

Databricks

Azure Data Engineering

Azure Data Factory (ADF)

Self-hosted Integration Runtime (SHIR)

Azure Data Lake Storage Gen2 (ADLS Gen2)

Azure Blob Storage

Logic Apps

Apache Spark

PySpark

Python

Snowflake

ETL / ELT Development

Cloud Data Migration

Data Quality & Validation

GitHub Actions

CI/CD

Secondary Skills

Delta Lake

Lakehouse Architecture

Medallion Architecture – Bronze / Silver / Gold

Snowflake Data Engineering

Data Modeling

Database Design

Airflow

Data Governance

Data Observability

Performance Optimization

AWS Cloud

Azure Cloud

AI/ML for Data Engineering

AI-Assisted Development Tools

Healthcare Payer Data

Required Qualifications

Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.

Minimum 3+ years of experience as a Cloud Data Engineer, Cloud Platform Engineer, or similar role.

Strong hands-on experience with Databricks in enterprise data engineering environments.

Experience with Azure data services, particularly ADF, ADLS Gen2, Blob Storage, Logic Apps, and SHIR.

Strong experience with Apache Spark and PySpark.

Hands-on experience with Snowflake for data engineering or analytics workloads.

Experience developing and supporting data migration and cloud modernization initiatives.

Strong experience with ETL development using on-premises databases and ETL technologies.

Strong SQL development skills, including:

Complex queries

Stored procedures

Views

Joins

Data transformations

Performance optimization

Experience with Python or other scripting languages for data processing and automation.

Hands-on experience with GitHub, including branching, pull requests, and code reviews.

Experience implementing GitHub Actions and CI/CD pipelines.

Experience supporting automated testing and data quality validation processes.

Strong understanding of Agile/Scrum delivery methodologies.

Ability to learn new technologies quickly and adapt to changing project requirements.

Strong analytical, troubleshooting, communication, and collaboration skills.

Ability to work independently and effectively manage ambiguity and competing priorities.

Ability to work under tight deadlines and proactively escalate critical technical or delivery issues.

Willingness to travel when required based on business needs and applicable travel conditions.

Preferred Qualifications

Experience with Azure and/or AWS cloud platforms.

Experience implementing Bronze/Silver/Gold (Medallion) architecture.

Strong understanding of Delta Lake and Lakehouse architecture.

Experience with Delta Lake optimization techniques.

Experience with Databricks performance tuning and optimization.

Experience with Snowflake performance optimization.

Knowledge of enterprise data governance and data quality best practices.

Experience with data modeling and database architecture.

Experience with Airflow or other data orchestration frameworks.

Experience developing Snowflake-based data engineering and analytics solutions.

Experience leveraging AI/ML solutions for data engineering automation and workflow optimization.

Familiarity with AI-assisted development tools such as GitHub Copilot, Databricks Assistant, ChatGPT, or Claude.

Experience working with healthcare payer data, including:

Claims

Membership

Enrollment

Provider

Clinical

Financial data

Azure or Databricks certification is a plus.

DevOps & Engineering Practices

GitHub

GitHub Actions

CI/CD

Git branching and pull requests

Code reviews

Automated testing

Infrastructure and deployment automation

Monitoring and observability

Agile/Scrum

Key Competencies

Cloud Data Engineering

Databricks Engineering

ETL Modernization

Cloud Migration

Azure Data Platform

Spark & PySpark

Snowflake

Data Pipeline Development

Data Quality & Reconciliation

DevOps & CI/CD

Performance Optimization

Data Governance

Problem Solving

Technical Collaboration

Work Location: Remote

Apply for this position

Full Name *Email *Phone *Bio *Upload CV/Resume *Allowed Type(s): .pdf, .doc, .docxBy using this form you agree with the storage and handling of your data by this website. *