Mohd Anas

Data Engineer

• Job Details:Designed and implemented end-to-end data pipelines using Confluent Kafka, Python, and PySpark, supporting both real-time and batch data processing across multiple business domains. Built and optimized Kafka producer and consumer applications in Python to efficiently ingest and process large-scale data streams with high throughput and low latency. Developed automated batch processing pipelines using PySpark and SparkSQL, enabling scheduled data ingestion, transformation, and aggregation from diverse data sources. Configured and managed Kafka S3 and SFTP source connectors for automated ingestion of structured and unstructured data from AWS S3 buckets and SFTP servers into Kafka topics. Utilized ksqlDB for real-time stream processing, including data transformations, filtering, joins, and aggregations, enabling real-time insights and analytics. Built and maintained ETL workflows on AWS cloud, leveraging services such as Amazon S3, ensuring secure, scalable, and cost-effective data operations. Implemented data quality validation, error-handling mechanisms, and logging across all batch and real-time pipelines to ensure data consistency and reliability. Managed Kafka infrastructure, including brokers, topics, partitions, and Zookeeper nodes, to support low-latency, high-availability streaming and batch workloads. Actively used Jira software for agile sprint tracking, creating and updating user stories, sub-tasks, and tracking job progress in real-time.

Oracle Data Engineer Trainee & Machine Learning Trainee

• Job Details:Developed Python scripts for web scraping and OCR to extract structured data from websites, images, and PDFs, reducing manual data entry. Built data extraction pipelines to clean, structure, and store data scraped from diverse sources, enabling further analysis and reporting. Utilized PySpark and SparkSQL to perform data transformations and aggregation on large datasets, enabling faster analytics and reporting. Documented data workflows and supported team-wide knowledge sharing on Python-based data engineering techniques.