Databricks Architect

We're seeking a skilled, hands-on Databricks Architect to serve as our top technical expert and lead practitioner for the Databricks Platform. This role combines architectural leadership with practical implementation. You will define the architectural vision and develop core, reusable reference architectures and patterns that accelerate our teams. The ideal candidate is a Databricks master who leads by example through hands-on development and empowers teams to build robust, scalable data solutions.

Key Responsibilities

· Develop and maintain production-quality reference architectures and reusable patterns that showcase best practices and accelerate engineering teams.

· Architect and build proofs-of-concept for end-to-end solutions on the Databricks Lakehouse Platform, validating complex designs hands-on.

· Serve as the primary Databricks consultant, providing expert guidance beyond diagrams, including code, best practices, and hands-on support.

· Provide thought leadership around big data, machine learning, and cloud-native solutions.

· Define, document, and evangelize Databricks development standards for data modeling, performance tuning, security, and cost management.

· Mentor engineers through architectural reviews, code reviews, paired programming, and design sessions, boosting technical proficiency across the organization.

Core Qualifications:

· Bachelor's or Master's degree in Computer Science or related field.

· 8+ years of experience in data architecture or software engineering, data engineering and data warehousing including 3+ years with Databricks.

· Expert-level knowledge of Apache Spark, Delta Lake, Python/Scala, and SQL.

· Proven hands-on technical architect and advisor on large-scale data projects.

· Excellent communication skills with ability to influence and guide technical teams and stakeholders.

· Strategic thinker passionate about solving complex data challenges and driving business outcomes.

· Strong critical thinking and well-reasoned architectural decision-making skills.

Technical Expertise:

Databricks Mastery (Including):

· Unity Catalog: data governance and security design and implementation.

· Delta Lake & Delta Live Tables: architecting scalable, reliable data pipelines.

· Experience with data mesh, real-time streaming (e.g., Kafka), and MLOps.

· Performance & Cost Optimization: Spark job tuning, cluster optimization, cost management.

· MLOps: practical ML lifecycle management with MLflow.

· Databricks SQL: analytical workload design and optimization.

· Strong understanding of DBT

· Cloud & Infrastructure: deep AWS/Azure knowledge and expertise with Infrastructure as Code (Terraform, YAML).

· Data Engineering & Programming: strong data modeling, ETL/ELT development, and advanced Python and SQL skills.

· CI/CD & Automation: experience designing and implementing CI/CD pipelines (GitHub Actions preferred) for data and ML workloads.

· Observability: familiarity with monitoring, logging, and alerting implementations for data platforms.

· Automation: platform is ephemeral; all changes via Terraform and Python. Expertise in both is required.