Job Details
Experience Needed:
Career Level:
Education Level:
Salary:
Job Categories:
Skills And Tools:
Job Description
- Data Engineering with Microsoft Fabrics:
- Utilize Microsoft Fabrics to design, implement, and manage scalable, high-performance data pipelines that ingest and process large volumes of data.
- Leverage Microsoft Fabrics features such as Lakehouse architecture, OneLake, and other data fabric components to integrate data from multiple sources into a unified platform.
- Apply best practices for data engineering to ensure data quality, integrity, and security in Microsoft Fabrics environments.
- Azure Data Factory Integration:
- Design and implement Azure Data Factory (ADF) pipelines to orchestrate data workflows, ensuring efficient and reliable data ingestion and transformation.
- Automate data movement and transformation tasks across various data storage and processing services using ADF.
- Monitor and troubleshoot ADF pipelines, ensuring optimal performance and error resolution.
- SQL Development:
- Write and optimize complex SQL queries for data extraction, aggregation, and analysis, ensuring efficient processing of large datasets.
- Develop stored procedures, views, and scripts to support data transformations and reporting.
- Ensure the security and privacy of data by implementing appropriate SQL controls and policies.
- Ability to work on different kinds of databases like My SQL, MariaDB, Microsoft SQL Server, and Dataverse.
- Data Architecture and Modeling:
- Design and maintain data architecture, ensuring data storage solutions are aligned with business requirements and technical standards.
- Create and maintain data models, ensuring accurate and efficient storage and retrieval of data.
- Collaborate with stakeholders to understand data requirements and deliver solutions that meet business needs.
- Data Governance and Quality:
- Implement data governance best practices, ensuring data is accurate, consistent, secure, and compliant with organizational standards.
- Develop and enforce data quality controls and validation processes across all stages of data engineering workflows.
- Monitor data pipelines and troubleshoot issues related to data quality, integrity, and performance.
- Collaboration and Stakeholder Communication:
- Work closely with data scientists, analysts, product managers, and other engineers to deliver data solutions that support business objectives.
- Collaborate with cross-functional teams to gather and refine data requirements, ensuring alignment with the organization's goals.
- Provide technical guidance and expertise in data engineering with the best practices and tools to drive innovation and efficiency.
- Performance Tuning and Optimization:
- Continuously monitor and optimize the performance of data pipelines, databases, and data models to ensure they are operating at maximum efficiency.
- Conduct regular performance tuning of Azure, Databricks, PySpark, and SQL environments to handle large-scale data processing.
Job Requirements
Education:
- Bachelor’s degree in computer science, Information Technology, Data Science, or a related field. Advanced degrees or relevant certifications in data engineering or cloud platforms (e.g., Azure) are a plus.
- Minimum of 3+ years of experience as a Data Engineer, with at least 1 year of hands-on experience in Microsoft Fabrics.
- Proven experience in working with Databricks, PySpark, Azure Data Factory, SQL, and other data engineering tools and platforms.
- Strong understanding of ETL processes, data integration, and data transformation techniques.
- Proven experience in building and maintaining data warehouses and applying data governance.
- Expertise in Microsoft Fabrics, including experience with Lakehouse architecture, OneLake, and data fabric integrations.
- Proficiency in Databricks and PySpark for building and optimizing scalable data pipelines.
- In-depth experience with Azure Data Factory for orchestrating and automating data workflows.
- Strong SQL skills, with the ability to write and optimize complex queries for large datasets and analytical functions.
- Knowledge of cloud platforms, particularly Microsoft Azure, including data storage, security, and cloud-native architectures.
- Familiarity with data modeling concepts and experience creating scalable data architectures.
- Understanding data governance, data security, and data privacy with the best practices.
- Knowledge of Microsoft Power Platform, especially Power BI and Power Apps.
- Excellent problem-solving skills with a focus on delivering scalable and efficient solutions.
- Strong communication skills, both verbal and written, with the ability to convey technical concepts to non-technical stakeholders.
- Team-oriented with the ability to collaborate effectively with backend developers, analysts, and other engineers.
- Strong attention to details, with a commitment to delivering high-quality, reliable data solutions.
- Certifications in Microsoft Azure, Databricks, or other relevant technologies.
- Experience with data visualization tools such as Power BI or Tableau.
- Familiarity with other programming languages such as Python, Java, or Scala for data processing and integration tasks.
- Microsoft Fabric Analytics certified.
- Knowledge of Microsoft Dynamics.
Featured Jobs
Similar Jobs
- Senior Data Warehousing & Busi...The Micro, Small & Medium Enterprise Development Agency - Dokki, Giza1 month ago