Senior Data Engineer - Databricks

Mobile Wave Solutions is a professional services company specializing in software development as a service. With a team of over 120 engineers, we deliver scalable, high-quality software that empowers our global clients to innovate and grow. We value collaboration, technical excellence, and a pragmatic approach to solving complex problems. About the Role We are seeking a seasoned, hands-on Senior Data Engineer to serve as the subject matter expert and technical anchor for our next-generation Databricks Lakehouse platform. In this role, you will design and build scalable ingestion pipelines, establish production best practices, set up CI/CD standards, and elevate the team’s technical capabilities. Our current environment is hosted on AWS , but we welcome strong Databricks experts with background in Azure or other major cloud providers, as your core Lakehouse mastery is what matters most. We need a confident, self-directed engineer with deep production experience across Databricks and Apache Spark who can tackle unstructured data sources, optimize distributed workloads, and drive decisive architectural standards in a fast-paced rebuild environment. Key Responsibilities

• Platform Standards & Best Practices: Lead the architecture, design, and implementation of robust Lakehouse ingestion and transformation pipelines using Databricks and PySpark. Establish standards that level up the broader team's engineering maturity.
• CI/CD & Operational Excellence: Set up automated delivery pipelines using Databricks Asset Bundles (DABs) and modern orchestration (Airflow or similar) to ensure seamless, reliable deployments.
• Complex Data Integration: Architect pipelines to ingest, process, and structure both structured (OLTP) and unstructured/semi-structured data (APIs, logs, cloud storage streams) into high-quality analytical datasets.
• Delta Lake & Governance: Enforce Delta Lake principles (CDC, schema evolution, optimization) and integrate data quality frameworks (e.g., Great Expectations) and Unity Catalog governance into daily development.
• AI Agent Direction: Direct and evaluate AI agents for rapid code generation, ensuring generated code meets strict quality, security, and performance standards.
• Technical Mentorship & Execution: Clearly articulate complex technical trade-offs to stakeholders, mentor peers, and confidently drive pragmatic solutions without needing step-by-step guidance. Qualifications
• Deep Databricks & Spark Expertise: 5+ years of dedicated data engineering experience with proven, large-scale production exposure to Databricks, PySpark performance tuning, and distributed processing architectures.
• Cloud Ecosystem Experience: Strong proficiency in major cloud environments (AWS primary, but candidate expertise in Azure or GCP with Databricks is fully welcomed and transferable).
• Advanced CI/CD Knowledge: Hands-on experience implementing enterprise CI/CD for Databricks workflows, specifically utilizing Databricks Asset Bundles (DABs) or equivalent infrastructure-as-code patterns.
• Unstructured & Diverse Data Processing: Demonstrated history of turning raw, unstructured, or semi-structured data sources into structured, production-ready Lakehouse layers.
• Production Discipline: Expert SQL optimization, deep Python proficiency, and practical experience with workflow orchestrators (Airflow) and quality/governance tooling (Unity Catalog, Great Expectations).
• Senior Communication: Concise, direct communicator capable of explaining technical concepts clearly and mentoring team members on best practices.
• Pragmatic & Autonomous: Comfortable tackling ambiguous, legacy datasets in a fast-paced rebuild environment; excels at guiding AI-assisted coding tools while enforcing high stability standards. Our Benefits
• Remote Office – Flexible hybrid form of working
• Parking Space – We provide free parking spots
• Fun Office Space – We offer a game zone and a relaxation area
• Health Insurance – Additional private health insurance, including a dental care plan
• Personal Development – Company-sponsored training budget to further develop your skills
• Employee Referral Program – Receive a bonus for referring a friend
• Holidays – Enjoy an extra 5 days after your 1st and 5th year
• Social Events – We love to celebrate our success together
• Family Insurance – Add insurance to a family member
• Offering sports cards – 100% sponsored by the company