Job Title: Data Engineer

Job Summary

We are seeking a Data Engineer to design, build, and maintain scalable data pipelines and data platforms that support enterprise analytics, reporting, and AI initiatives. The role involves developing ETL/ELT pipelines, integrating multiple data sources, and transforming raw data into high-quality, analytics-ready datasets.

The Data Engineer will work closely with data architects, data engineering leads, analytics teams, and business stakeholders to build reliable, efficient, and secure data systems across Azure, Microsoft Fabric, Databricks, Snowflake, and AWS environments.

Key Responsibilities

Data Pipeline Development

• Develop and maintain ETL/ELT pipelines using tools such as Azure Data Factory, Databricks, Snowflake, or similar data integration platforms.

• Build scalable data ingestion pipelines to collect structured, semi-structured, and unstructured data from multiple enterprise sources.

• Implement batch and streaming data pipelines to support analytics and reporting requirements.

• Transform and process raw data into clean, structured, and analytics-ready datasets.

Data Platform Development

• Support the development and maintenance of data lakes, data warehouses, and lakehouse architectures.

• Implement Medallion Architecture (Bronze, Silver, Gold) to organize and refine datasets for analytics consumption.

• Develop data processing workflows using Python, PySpark, Spark, and SQL.

• Work with distributed data processing frameworks to handle large-scale datasets efficiently.

Data Quality and Reliability

• Implement data validation, cleansing, and transformation logic within data pipelines.

• Monitor data pipelines to ensure data accuracy, consistency, and reliability.

• Troubleshoot pipeline failures and optimize workflows for performance.

Collaboration and Delivery

• Work with data architects and data engineering leads to implement scalable data architectures.

• Collaborate with analytics teams and business stakeholders to understand data requirements.

• Document data pipelines, data flows, and data models to ensure maintainability.

• Follow best practices for data engineering, version control, and deployment automation.

Required Skills and Experience

• Strong programming skills in Python and SQL.

• Experience working with data engineering tools such as Azure Data Factory, Databricks, Snowflake, or similar platforms.

• Experience building ETL/ELT pipelines and data integration workflows.

• Familiarity with Spark, PySpark, or distributed data processing frameworks.

• Understanding of data lakes, data warehouses, and lakehouse architectures.

• Experience working with cloud platforms such as Azure or AWS.

• Understanding of data modeling and data transformation techniques.

Preferred Qualifications

• Experience with streaming technologies such as Kafka or Spark Streaming.

• Familiarity with Microsoft Fabric, Synapse, or Snowflake Snowpipe.

• Knowledge of data governance, metadata management, and data quality frameworks.

• Experience working in Agile development environments.

• Exposure to AI/ML data pipelines or feature engineering workflows.

Experience: 2-5 Years

Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

Location: Bangalore