Lead Consultant - Azure data pipelines Engineer
Location: United Kingdom, GB
Positon: Lead Consultant - Azure data pipelines Engineer
Job Title: Lead Consultant - Azure data pipelines Engineer
Salary: Competitive, plus benefits
Contract type: Permanent, Full-time
- Construct and manage a sophisticated data platform on Azure, tailored to manage extensive and intricate data sets.
- Develop solutions for Data Lakes and Data Warehouses.
- Employ Data Vault and Dimensional modelling techniques for data model construction.
- Establish both batch and streaming data pipelines that are automated, efficient, and
- Engage collaboratively with Data Science teams to fulfill their data-related needs.
- Coordinate with teams focused on Reporting and Data Visualization to deliver comprehensive data visualization solutions.
- Devise strategies for managing Personal and Sensitive data securely.
- Adopt Agile methodologies and DevOps practices in daily work.
- Proficiency with Azure Data Lake Storage (ADLS Gen2).
- Experience with Synapse Dedicated SQL Pool, including table creation, stored procedures, and more.
- Familiarity with Azure Data Factory, including the creation and management of metadata-driven pipelines.
- Skilled in using Azure Databricks for both batch and streaming data processing.
- Understanding of various Data Lake storage formats such as Parquet and Delta Lake.
- Capability to process complex data types, including JSON and XML.
- Experience with Azure Event Hubs and Event Grids.
- Proficiency in Azure Functions and Cognitive Search.
- Programming skills in a structured language, such as Python or C#.
- Knowledge of implementing CI/CD pipelines.
- Experience in feature engineering and ML Ops.
- Capability to develop reporting solutions using Power BI.
- Familiarity with tools within the Synapse Analytics Workspace.
- Understanding of SQL and Data Streaming technologies.