KEY RESPONSIBLITIES :

• Defining, designing, and implementing data pipelines using ADF and Databricks which pull data from ADLS and other cloud and on-prem data sources.
• Design instrumentation of Application Insights logs for Data Quality and events.
• Play a consultant define the new architecture landscape as they move from ADW to
• Databricks + ADLS. This includes defining the best possible file formats, ETL/ELT methodology, and the right infrastructure requirements.
• Advice and Implement strategies for cost-effective implementation of Databricks and best practices for coding and performance.
• Guide the team extending the current Azure DevOps CICD into Databricks deployments as well.
• Suggest innovative ideas which can help reduce current infrastructure costs for Synapse / ADLA etc.
• Set guidelines and governance rules around usage of Azure tech by Analysts and Scientists.
• Do Proof of concepts that will solve issues that the team is currently facing using other technologies like Cosmos DB. Synapse etc.
• Present solutions and next ideas to their senior leadership.

PREFFERED QUALIFICATIONS & EXPERIENCE:

• Experience in data warehousing.
• Experience in writing complex SQL and ETL processes
• Experience with schema design and dimensional data modelling Technologies
• Synapse/ Data Warehouse
• Azure Data Lake, Blob Storage
• Databricks
• Python/Scala/Spark
• Azure DevOps - CI/CD Job Requirements: Azure Synapse Analytics, Data Analysis, Data Modelling Document.