Pyspark Engineer

• Architect, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Python to ingest data from diverse batch and streaming sources (Kafka, relational databases, REST APIs, Object Storage).

• Architect, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Python to ingest data from diverse batch and streaming sources (Kafka, relational databases, REST APIs, Object Storage).
• Design and implement Lakehouse architectures using formats such as Delta Lake, Apache Iceberg, or Apache Hudi .
• Build reusable frameworks and data ingestion libraries to standardize enterprise data processing.

• Diagnose and tune Spark application performance, managing memory allocation, partitions, execution plans, broadcast joins, data skews, and garbage collection.

• Diagnose and tune Spark application performance, managing memory allocation, partitions, execution plans, broadcast joins, data skews, and garbage collection.
• Optimize query execution plans using Catalyst Optimizer and Tungsten execution engine.
• Monitor and troubleshoot production Spark jobs across cluster management environments (YARN, Kubernetes, or Databricks).

• Collaborate with Data Architects to design dimensional (Star/Snowflake), normalized, and Data Vault schemas optimized for analytical workloads.

• Collaborate with Data Architects to design dimensional (Star/Snowflake), normalized, and Data Vault schemas optimized for analytical workloads.
• Ensure efficient data storage, partitioning strategies, and file format selection (Parquet, ORC, Avro).

• Schedule and orchestrate complex workflow DAGs using workflow orchestrators such as Apache Airflow , Prefect, or Dagster.

• Schedule and orchestrate complex workflow DAGs using workflow orchestrators such as Apache Airflow , Prefect, or Dagster.
• Implement CI/CD pipelines (Git, Jenkins, GitLab CI, or GitHub Actions) for automated testing, packaging, and deployment of data pipelines.
• Write unit tests, integration tests, and data validation routines (e.g., Great Expectations, PyTest).

• Implement enterprise data governance, lineage tracking, metadata management, and data quality frameworks.

• Implement enterprise data governance, lineage tracking, metadata management, and data quality frameworks.
• Ensure strict compliance with data security, masking, tokenization, RBAC, and regulatory privacy standards (e.g., GDPR, CCPA).

• Core Technologies - Apache Spark, PySpark (Spark SQL, DataFrames, Datasets, Structured Streaming), Python 3.x

• Core Technologies - Apache Spark, PySpark (Spark SQL, DataFrames, Datasets, Structured Streaming), Python 3.x
• Database & Querying - Advanced SQL, Query optimization, Indexing, Window functions
• Data Lake & Lakehouse - Delta Lake, Apache Iceberg, Apache Hudi, Parquet, Avro, ORC
• Workflow & Orchestration - Apache Airflow, Luigi, Prefect, or Databricks Workflows
• Cloud Platforms - AWS (EMR, Glue, S3, Redshift), Azure (Synapse, ADLS, HDInsight), or GCP (Dataproc, BigQuery, GCS)
• Streaming Technologies - Apache Kafka, Spark Structured Streaming, Event Hubs, or Kinesis

• Distributed Computing Fundamentals: In-depth knowledge of Spark internals (RDD, Catalyst Optimizer, DAG execution, Shuffling, Narrow vs. Wide dependencies).

• Distributed Computing Fundamentals: In-depth knowledge of Spark internals (RDD, Catalyst Optimizer, DAG execution, Shuffling, Narrow vs. Wide dependencies).
• Production Experience: Minimum 4+ years of hands-on experience building and deploying production-grade PySpark applications at enterprise scale.
• Problem Solving: Proven track record in debugging memory bottlenecks, Out-Of-Memory (OOM) errors, and resolving data skew issues.
• Data Engineering Best Practices: Experience implementing modular code architectures, logging, monitoring (Prometheus, Grafana, CloudWatch), and alerting.

• Hands-on experience with Databricks platform (Delta Lake, Unity Catalog, Photon engine, MLflow).

• Hands-on experience with Databricks platform (Delta Lake, Unity Catalog, Photon engine, MLflow).
• Familiarity with Scala/Java in addition to Python.
• Knowledge of DataOps and Infrastructure as Code ( Terraform , CloudFormation).
• Experience with modern analytical data warehouses (Snowflake, Google BigQuery, Amazon Redshift).
• Relevant certifications: Databricks Certified Data Engineer Associate / Professional AWS Certified Data Engineer / Azure Data Engineer Associate (DP-203)CCA Spark and Hadoop Developer (or equivalent)