Databricks Engineer

Salary: $123,000 - 163,000 per year Requirements: We seek a bachelors degree in Computer Science, Information Technology, or a related discipline, or equivalent professional experience. We require 5-8 years of experience in data engineering, with hands-on work in Databricks and Hadoop environments. We need strong proficiency in Spark (Scala or PySpark), ETL pipeline development, and data migration methods. We value experience with Lakehouse architecture, Delta Lake, and advanced Databricks capabilities such as Lakehouse Federation, Liquid Clustering, and Unity Catalog. We look for a solid understanding of Azure cloud platforms and data governance principles. We expect strong analytical and troubleshooting abilities with close attention to detail. We require the ability to collaborate effectively across functions and communicate clearly with both technical and non-technical stakeholders. Preferred qualifications include experience with Databricks Migration Accelerator or similar migration tools. Preferred qualifications include familiarity with synthetic data generation and pipeline health monitoring. Preferred qualifications include understanding of performance tuning, query optimization, and cost management on cloud data platforms. Preferred qualifications include the ability to document migration patterns and help establish best practices for broader adoption. Responsibilities: We identify and classify existing Hadoop jobs and data sources, including ETL, batch, and streaming workloads, to support migration planning. We help select scalable and repeatable migration scenarios for MVP initiatives. We provision and configure Databricks workspaces, ensuring Lakehouse architecture and federation capabilities are enabled. We execute pilot migrations of representative Hadoop workloads using Databricks Migration Accelerator or partner solutions. We validate and monitor post-migration performance, cost efficiency, and data integrity. We enable and test advanced Databricks capabilities, such as Liquid Clustering, Lakehouse AI monitoring, Serverless warehouse, and Unity Catalog, to improve data layout and pipeline health. We collaborate with data engineering, analytics, and governance teams to gather feedback and document learnings, blockers, and feature gaps. We track and report key success metrics, including migration time, query latency, cost savings, and feature adoption. We contribute to a phased roadmap for full-scale migration and the rollout of advanced Databricks features. Technologies: AI Azure Cloud Databricks ETL Hadoop Support Liquid PySpark Scala Serverless Spark Unity GameDev LESS More: We are a client-facing hiring team seeking a Mid-Level Databricks Engineer in Purchase, NY to join our data engineering group. In this role, we support the migration of Hadoop workloads to the Databricks Lakehouse platform and help build scalable, repeatable data solutions. We offer comprehensive benefits to eligible employees, including medical, dental, and vision insurance, HSA, FSA, 401(k), and life, disability, and ADD coverage. Salaried personnel receive paid time off, and hourly employees may be eligible for paid sick leave depending on assignment and applicable law. last updated 37 week of 2026