Machine Learning Engineer

Who we are At Halcyon Labs , we’re shaping the future of communications, all from the comfort of our homes. We deliver innovative solutions to hundreds of thousands of businesses and empower millions of developers worldwide to craft personalized customer experiences. Our dedication to remote-first work, and strong culture of connection and global inclusion means that no matter your location, you’re part of a vibrant team with diverse experiences making a global impact each day. As we continue to revolutionize how the world interacts, we’re acquiring new skills and experiences that make work feel truly rewarding. Your career at Halcyon Labs is in your hands. We use Artificial Intelligence (AI) to help make our hiring process efficient. That said, every hiring decision is made by real Halcyon Labs ns! . See yourself at Halcyon Labs Join the team as Halcyon Labs ’s next Machine Learning Engineer. About the job This position is needed to drive innovation and the development of cutting-edge products that serve developers, builders, and operators within Halcyon Labs ’s Data & Observability Substrate organization. This is a hands-on, builder-focused engineering role that bridges Product, Design, and Engineering to develop, evaluate, and maintain scalable, low-latency, ML-based systems for real-time applications. You will lead rapid research-to-production cycles that translate business ideas into solutions for complex problems—such as streaming anomaly detection, recommendation systems, predictive modeling, and agentic AI frameworks—with the goal of delivering personalized customer experiences. You will collaborate closely with a cross-functional team of engineers, architects, product managers, UI/UX designers, and ML/data science partners to deliver robust, reliable solutions that power customer success. Responsibilities In this role, you’ll: Partner with product, UX, and technical stakeholders to analyze business problems, clarify requirements, define scope, and translate them into measurable ML problem statements. Design, implement, and maintain scalable, enterprise-grade ML solutions in production. Build reproducible ML workflows for data preparation, training, evaluation, and inference using modern orchestration and MLOps tooling. Implement monitoring and evaluation frameworks to continuously improve data quality, model performance, latency, and cost through feedback loops. Partner cross-functionally with Product, Data Science/ML, Engineering, and Security to deliver resilient, scalable, and compliant ML-powered services. Demonstrate end-to-end systems understanding and articulate the “why” behind model and system design choices. Own operational excellence: SLAs, on-call, incident response, customer feedback triage, and blameless post-mortems. Drive engineering excellence via AI-assisted SDLC, code reviews, automated testing, MLOps best practices, knowledge-sharing, and mentoring. Actively adopt AI-assisted practices to improve implementation and collaboration efficiency. Qualifications Halcyon Labs values diverse experiences from all kinds of industries, and we encourage everyone who meets the required qualifications to apply. If your career is just starting or hasn't followed a traditional path, don't let that stop you from considering Halcyon Labs . We are always looking for people who will bring something new to the table! *Required: Strong foundation in ML/AI (statistics, probability, optimization) with the ability to apply these concepts to real-world problems. 5+ years of experience building, deploying, and operating data and ML systems in production. Proficient in Python, Java, and SQL; strong software engineering fundamentals (system design, testing, version control, code reviews). Hands-on experience with workflow orchestration and data pipelines (e.g., Airflow, Kubeflow) and cloud data platforms/storage (e.g., SageMaker Feature Store, Snowflake, DynamoDB, OpenSearch). Experience with the ML lifecycle and MLOps tooling (e.g., MLflow, Metaflow, SageMaker; LLM/agent frameworks such as LangChain/LangGraph; model evaluation/observability tools such as Galileo or similar). Working knowledge of containerization and cloud infrastructure, including Docker and Kubernetes, GitOps/CI/CD tools (e.g., Argo CD), and at least one major cloud platform (AWS, GCP, or Azure). Understanding of data modeling and scalable systems, including distributed computing and streaming frameworks (e.g., Spark/EMR, Flink, Kafka Streams); familiarity with GPU-based implementation is a plus. Demonstrated ability to ramp up quickly and operate effectively in new application/business domains. Strong written and verbal communication skills: able to document and present designs and decisions, and comfortable giving/receiving feedback in an Agile environment. Desired: Familiarity with ML problem areas and techniques, including recommendation systems (e.g., graph-based approaches, two-tower models), time-series modeling (classical and deep learning), representation learning (e.g., embedd