数据工程师（Databricks）

Senior Data Engineer (Databricks + CosmosDB)

Location: Bangalore (Hybrid - 3 days from client office)

Experience: 8-10 Years

Job Summary: We are seeking a highly skilled Senior Data Engineer with deep expertise in Azure Databricks and Cosmos DB. The ideal candidate will be responsible for designing, building, and optimizing scalable, cloud-native data platforms and data pipelines. This role requires a hands-on approach to developing high-performance data solutions that support analytics, AI/ML initiatives, real-time processing, and enterprise reporting requirements while ensuring data quality, security, and operational excellence.

• Responsibilities:1. Data Engineering & Pipeline Development
• Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and Apache Spark.
• Build robust ETL/ELT processes to ingest, transform, and curate large volumes of structured and unstructured data.
• Develop reusable data engineering frameworks and accelerators.
• Optimize data processing jobs for performance, scalability, and cost efficiency.

• Databricks Platform Development
• Build and maintain data solutions using Azure Databricks.
• Develop notebooks and workflows using Apache Spark RDD, Apache Spark SQL, and Python.
• Implement Delta Lake architectures for reliable and efficient data management.
• Configure and optimize Databricks clusters and workloads.
• Support CI/CD implementation for Databricks deployments.

• Cosmos DB Development & Optimization
• Design and implement scalable solutions using Azure Cosmos DB (NoSQL).
• Model data effectively for NoSQL workloads.
• Optimize partition strategies, indexing policies, and query performance.
• Manage data replication, consistency models, and throughput provisioning.
• Integrate Cosmos DB with analytics and downstream systems.

• Azure Data Platform Integration
• Develop and integrate end-to-end solutions using Azure services such as Azure Data Factory, Azure Functions, and Azure Logic Apps.

• Real-Time & Streaming Data Processing
• Build streaming data pipelines using Spark Structured Streaming.
• Process event-driven workloads from Azure Event Hubs, Kafka Stream, or IoT sources.
• Implement near real-time analytics solutions integrated with Cosmos DB and Databricks.

• DevOps & CI/CD
• Implement Devsecops practices for Data Engineering solutions.
• Automate deployment pipelines using Azure DevOps or GitHub Actions.
• Manage source control and release processes.

Mandatory Skills

• Azure Databricks
• Apache Spark RDD
• Apache Spark SQL
• Delta Lake
• Azure Cosmos DB (NoSQL)

Preferred Skills

• Delta Live Tables (DLT)
• Azure Databricks
• Azure Databricks

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 8-10 years of experience in data engineering or related roles.
• Strong understanding of cloud engineering principles and modern data architectures.
• Excellent problem-solving skills and ability to work in a fast-paced environment.

根据职位描述，该公司专注于数据工程解决方案，使用Azure Databricks、Cosmos DB等云原生技术，提供云-native数据平台和数据管道开发服务。