Senior Databricks Solution Architect

We are seeking a Senior Databricks Architect to design, implement, and optimize enterprise-scale data platforms using theDatabricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions.

Key Responsibilities

• Design and architectDatabricks Lakehouse solutions for large-scaleenterprise data platforms.
• Lead end-to-end Databricks implementation, including architecture, development, deployment, and optimization.
• Define Databricks workspace architecture, clusterstrategies, security models, and operational standards.
• Design and developscalable data pipelinesusingDatabricks, Apache Spark, PySpark, SQL, andDelta Lake .
• Implement data ingestionframeworks supporting batchand real-timedata processing.
• Design Delta Lake architectures including:
• Delta tables
• Medallion architecture (Bronze/Silver/Gold layers)
• Data quality frameworks
• Schema evolution andoptimization
• Perform Databricks performancetuning:
• Cluster optimization
• Spark job optimization
• Query tuning
• Cost optimization
• Implement Databricks governanceand security using:
• Unity Catalog
• Access controls
• Data lineage
• Audit capabilities
• Lead migration projectsfrom legacy dataplatforms to DatabricksLakehouse.
• Define best practicesfor Databricks CI/CD, DevOps, andenvironment management.
• Collaborate with dataengineers, data scientists, analysts, and enterprisearchitects.
• Provide technical guidance, architecture documentation, and design reviews.

Skills & Experience

• Strong experience in dataengineering, big data, or solution architecture.
• Strong hands-onexperience withDatabricks Platform .
• Expert knowledge of:
• Apache Spark
• PySpark / Scala
• Delta Lake
• SQL
• Data Lakehouse architecture
• ETL/ELT development
• Experience designing anddelivering enterprise Databrickssolutions.
• Strong experience withcloud platforms:
• Azure Databricks (preferred)
• AWS Databricks
• Google Cloud Databricks
• Experience with:
• Databricks Workflows
• Jobs orchestration
• Notebooks
• Repos
• Cluster management
• Unity Catalog

Preferred Skills

• Databricks Certified DataEngineer Professional /Databricks Certified Architect.
• Experience with:
• MLflow
• Databricks Machine Learning
• Feature Store
• Model deployment
• MLOps
• Knowledge of streamingtechnologies:
• Structured Streaming
• Kafka
• Experience with datagovernance, compliance,and enterprise securityframeworks.

Key Competencies

• Strong Databricks architectureand implementation expertise.
• Ability to designscalableand cost-efficientLakehouse solutions.
• Strong troubleshooting andperformance optimizationskills.
• Ability to leadtechnical discussions withenterprise stakeholders.

#J-18808-Ljbffr

Full job description from WhatJobs. Apply for this role →