Databricks Architect

Salary: $123,000 - 163,000 per year Requirements: We require a bachelors or masters degree in Computer Science, Engineering, or a related discipline. We are looking for 10+ years of experience in data architecture, with substantial hands-on work in Hadoop and Databricks environments. We need strong expertise in cloud data platforms, especially Azure, along with data engineering and ETL practices. We expect a solid grasp of Lakehouse architecture, data federation, and modern data governance frameworks such as Unity Catalog. We value experience leading large-scale migration programs, including MVP scoping and iterative delivery. We require advanced proficiency with Databricks capabilities such as Delta Lake, Liquid Clustering, AI monitoring, and serverless compute. We need excellent communication, leadership, and stakeholder management skills. We look for the ability to mentor and support cross-functional teams in adopting best practices and innovative data solutions. Preferred experience includes using Databricks Migration Accelerator or similar migration tools. Preferred hands-on knowledge includes testing advanced features such as dynamic clustering, Lakehouse Federation, and Unity Catalog. Preferred knowledge includes data security, access controls, and compliance in cloud environments. Preferred experience includes generating synthetic data and maintaining data governance during migration efforts. Responsibilities: We lead the assessment and classification of existing Hadoop workloads and data sources for Databricks migration. We design and deliver scalable, repeatable migration patterns, with a focus on MVP-based approaches to accelerate value. We provision and architect Databricks environments, including sandbox workspaces built on Lakehouse architecture and federation capabilities. We enable connectivity to external sources such as Hive and oversee pilot migrations using Databricks Migration Accelerator or partner solutions. We validate migrated workloads for performance, cost efficiency, and data integrity using features such as Z-ordering, Liquid Clustering, Lakehouse AI monitoring, and serverless warehouses. We monitor query performance, storage efficiency, and pipeline health using Databricks best practices and advanced platform features. We collaborate across data engineering, analytics, and governance teams to confirm outcomes and incorporate feedback. We document learnings, blockers, and feature gaps to guide broader rollout and continuous improvement. We define and track success metrics such as migration duration, query latency, cost reduction, and feature adoption. We develop a phased roadmap for full-scale migration, deeper feature adoption, and future platform optimization. Technologies: AI Architect Azure Cloud Databricks ETL Hadoop Hive Support Liquid Security Serverless Unity GameDev LESS More: We are a company in Purchase, NY seeking a Lead Databricks Architect to drive our migration from Hadoop to Databricks and establish scalable Lakehouse solutions. This role is at the center of our digital transformation, helping us unlock advanced analytics, AI capabilities, and modern data governance. We offer comprehensive benefits to eligible employees, including medical, dental, and vision insurance, HSA, FSA, 401(k), and life, disability, and ADD coverage. Salaried personnel receive paid time off, and we are an equal opportunity employer committed to fair consideration for all qualified applicants. last updated 37 week of 2026