Azure Databricks Engineer - Assistant Manager

We are seeking an experienced Azure Data Engineer/Developer with strong hands-on expertise in Azure Databricks, Apache Spark, Scala, Snowflake, SQL, and Azure DevOps.

The successful candidate will bring strong data-engineering expertise, experience with automated CI/CD pipelines and cloud migration, and the ability to deliver effectively within Agile teams.

This role is central to building and modernizing scalable cloud-based data solutions. The engineer will design, develop, test, deploy, and support data platforms that improve processing efficiency, reliability, automation, and enterprise data delivery.

Data Engineering and Platform Development

• Design and develop scalable, secure, and maintainable data-engineering solutions on Microsoft Azure.
• Build and support high-volume data-processing workloads using Azure Databricks, Apache Spark, and Scala.
• Develop reliable ETL/ELT pipelines for data ingestion, transformation, processing, and delivery.
• Design and implement Snowflake data-processing and data-warehouse solutions.
• Develop and optimize SQL queries, stored procedures, functions, and transformation logic.
• Integrate Databricks with Snowflake and other enterprise data sources.
• Create reusable, modular, and maintainable Scala and Spark components.
• Develop batch and streaming data pipelines using Lakeflow Connect, Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, Auto Loader, and Structured Streaming.
• Apply Unity Catalog capabilities for data governance, access control, lineage, catalog and schema management, and secure data sharing.
• Design and optimize Delta Lake solutions using ACID transactions, schema enforcement and evolution, time travel, change data feed, and table-maintenance practices.
• Build incremental and event-driven processing solutions using streaming tables, materialized views, Azure Event Hubs, or Kafka.
• Develop and tune Databricks SQL workloads using SQL warehouses, query profiling, Photon acceleration, and cost-optimization practices.
• Configure serverless computing, cluster policies, Databricks Runtime versions, workload isolation, and autoscaling based on workload requirements.
• Automate Databricks deployments using Declarative Automation Bundles, the Databricks CLI, REST APIs, service principals, and OAuth-based authentication.
• Monitor pipeline health, data quality, lineage, streaming backlogs, and resource usage using event logs, system tables, query history, and operational alerts.
• Implement data-quality controls, manage schema drift and quarantined records, and build resilient pipelines with checkpoint recovery, idempotent processing, and restart capabilities.
• Apply enterprise security practices, including managed identities, private networking, secure secret management, and Azure Key Vault integration.
• Support ML flow 3 capabilities for experiment tracking, model evaluation, observability, deployment, and model lifecycle management in Unity Catalog, where applicable.

Performance, Reliability, and Support

• Tune and optimize Spark/Databricks, SQL, and Snowflake workloads.
• Troubleshoot applications, pipeline, deployment, and data-processing issues.
• Identify technical risks and production issues and drive them through resolution.

DevOps and Release Management

• Implement source-control, build, release, and deployment processes using Azure DevOps.
• Develop and maintain automated CI/CD pipelines for data-engineering applications.
• Support automated deployments across development, test, and production environments.
• Participate in application and data-workload migrations to Azure and other modern cloud platforms.

Engineering Delivery and Collaboration

• Collaborate with engineering, infrastructure, and operations teams to deliver solutions successfully.
• Contribute to solution design, technical reviews, code reviews, and release activities.
• Work within Agile, Scrum, or Kanban delivery models and manage deliverables against committed schedules.

Required Technical Skills

• Strong hands-on experience with Azure Databricks, Apache Spark, Scala, Snowflake, and SQL.
• Proven SQL development and query-optimization skills.
• Solid understanding of data warehousing and ETL/ELT concepts.
• Experience with large-scale data processing and transformation.

Azure Platform

• Experience with Microsoft Azure, including Azure Databricks and Azure data-integration services.
• Experience migrating cloud applications and data workloads.

DevOps and Release Engineering

• Hands-on experience with Azure DevOps and Git-based source control.
• Ability to design and implement CI/CD pipelines.
• Experience automating build and deployment processes.
• Experience managing environment-based deployments and releases.

Engineering Practices

• Data-pipeline design and development.
• Performance tuning and optimization.
• Debugging, root-cause analysis, and production issue resolution.
• Code reviews and adherence to engineering standards.
• Experience with Agile development methodologies.

Additional Technologies and Experience

• Azure Data Factory (ADF).
• Python.
• Azure SQL or Azure SQL Managed Instance.
• Oracle or SQL Server.
• Jira and Confluence.

Domain Experience and Certifications

• Experience migrating legacy or on-premises workloads to Azure.
• Financial services technology experience.
• Azure Developer Associate, Azure Data Engineer, or Snowflake certification.

• Six or more years of overall IT or software-engineering experience preferred.
• Strong practical experience developing enterprise-scale data solutions.
• Demonstrated hands-on experience with Databricks/Spark, Scala, Snowflake, and SQL.
• Experience with enterprise CI/CD and DevOps practices.
• Bachelor’s degree in computer science, Engineering, Information Technology, or a related discipline preferred.

• Strong analytical, problem-solving, troubleshooting, and root-cause-analysis skills.
• Ability to independently own technical deliverables and meet committed timelines.
• Ability to collaborate effectively across engineering, infrastructure, and operations teams.
• Clear written and verbal communication skills.
• Ability to translate business and data requirements into scalable technical solutions.
• Strong focus on code quality, automation, reliability, and maintainability.

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at StateStreet.com/careers

Read our CEO Statement