Databricks Data Engineer

Come join us

The Databricks Data Engineer will be responsible for the design, development, and maintenance of scalable and high-performance data pipelines within the Databricks Lakehouse Platform. This role involves using Apache Spark, Delta Lake, and various Databricks services to process large volumes of batch and streaming data, ensuring data quality, reliability, and accessibility for data consumers.

Skills and experiences

Required Skills & Experience:

• 5+ years of hands-on experience in Data Engineering.
• 3+ years of dedicated experience building solutions on the Databricks Lakehouse Platform .
• Expert proficiency in Python (PySpark) and SQL for data manipulation and transformation.
• In-depth knowledge of Apache Spark and distributed computing principles.
• Experience with Delta Lake and Lakehouse architecture.
• Strong understanding of ETL/ELT processes, data warehousing, and data modeling concepts.
• Familiarity with at least one major cloud platform (AWS, Azure, or GCP) and its relevant data services.

Preferred Skills & Certifications:

• Experience with Databricks features like Delta Live Tables (DLT), Databricks Workflows, and Unity Catalog .
• Experience with streaming technologies (e.g., Kafka, Spark Streaming).
• Familiarity with CI/CD tools and Infrastructure-as-Code (e.g., Terraform, Databricks Asset Bundles).
• Databricks Certified Data Engineer Associate or Professional certification.

About you

Characteristics that can spell success for this role

· Consultative, collaborative, curious

· Resilient, adaptable, flexible

· Humble leader, strong negotiator, relationship builder

· Passionate about tech and engaging with and advising clients

What you'll do

• Data Pipeline Development: Design, build, and maintain robust and scalable ETL/ELT pipelines using Databricks, PySpark/Scala, and SQL to ingest, transform, and load data from diverse sources (e.g., databases, APIs, streaming services) into Delta Lake.
• Databricks Ecosystem Utilization: Utilize core Databricks features such as Delta Lake, Databricks Workflows (or Jobs), Databricks SQL, and Unity Catalog for pipeline orchestration, data management, and governance.
• Performance Optimization: Tune and optimize Spark jobs and Databricks clusters for maximum efficiency, performance, and cost-effectiveness.
• Data Quality and Governance: Implement data quality checks, validation rules, and observability frameworks. Adhere to data governance policies and leverage Unity Catalog for fine-grained access control.
• Collaboration: Work closely with Data Scientists, Data Analysts, and business stakeholders to translate data requirements into technical solutions and ensure data is structured to support analytics and machine learning use cases.
• Automation & DevOps: Implement CI/CD and DataOps principles for automated deployment, testing, and monitoring of data solutions.
• Documentation: Create and maintain technical documentation for data pipelines, data models, and processes.
• Troubleshooting: Monitor production pipelines, troubleshoot complex issues, and perform root cause analysis to ensure system reliability and stability.

Enjoy your career

Some of the best things about working at Avanade

• • Opportunity to work for Microsoft's Global Alliance Partner of the Year, with exceptional development and training (minimum 80 hours per year for training and paid certifications)
• Real-time access to technical and skilled resources globally
• Dedicated career advisor to encourage your growth
  Engaged and helpful coworkers genuinely interested in you

Find out more about some of our benefits

Employee Benefits at Avanade | Avanade

A great place to work

As you bring your skills and abilities to Avanade, you'll get distinctive experiences, limitless learning, and ambitious growth in return. As we continue to build our diverse and inclusive culture, we become even more innovative and creative, helping us better serve our clients and communities. You'll join a community of smart, supportive collaborators to lift, mentor, and guide you, and to lean on your expertise . You get a company purpose-built for business-critical, leading-edge technology solutions, committed to improving the way humans work, interact, and live. It's all here, so take a closer look

We work hard to provide an inclusive, diverse culture with a deep sense of belonging for all our employees. Visit our Inclusion & Diversity page.

Create a future for our people that focuses on

• Expanding your thinking
• Experimenting courageously
• Learning and pivoting

Inspire greatness in our people by

• Empowering every voice
• Encouraging boldness
• Celebrating progress

Accelerate the impact of our people by

• Amazing the client
• Prioritizing what matters
• Acting as one