Job Details
Experience Needed:
Career Level:
Education Level:
Salary:
Job Categories:
Skills And Tools:
Job Description
Data Engineers are the data professionals who prepare the data in data lake and big data infrastructure to be analyzed by Data Science methodologies. They are engineers who design, build, customize, integrate data from various resources, and manage data on big data., they design and write complex data pipeline that, make sure enterprise raw and processed data is easily accessible, works smoothly, and their goal is to finding trends in data sets optimizing the performance of their company’s big data ecosystem.
Work responsibility:
- Architecting and implementing distributed data pipelines systems
- Architecting and implementing distributed real time data streams processing systems.
- Developing data processing using large data sets in data lakes/big data eco systems, to support data services and analytics requirements.
- Responsible for finding trends in data sets and using/building/developing algorithms to help make raw data more useful to the enterprise.
- Identify ways to improve data reliability, efficiency and quality
- Support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture
- Deploying a variety of data mining/data analysis methods.
- Use data to discover tasks that can be automated
- Mine and analyze data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.
- Assess the effectiveness and accuracy of new data sources and data gathering techniques.
- Applying and supporting custom data models and algorithms to apply to data sets.
- Applying and supporting predictive models.
- Develop company data testing framework and test model quality.
- Develop processes and tools to monitor and analyze model performance and data accuracy.
- Coordinate with different functional teams to implement data pipelines and streams and monitor outcomes.
Job Requirements
Qualifications:
- Bachelor's degree in computer engineering.
- 5-7 years of experience manipulating data sets and building statistical models.
- Experience supporting and working with cross-functional teams in a dynamic environment.
- Strong problem-solving skills with an emphasis on product development.
- Excellent written and verbal communication skills for coordinating across teams.
- A drive to learn and master new technologies and techniques.
- Solid understanding and experience with querying any relational databases
- Experience working with and creating data architectures.
- Experience working with statistical computer languages: R.
- Solid understanding and experience with Python, pandas, pyspark, dash, flask
- Solid Knowledge and Experience with Big Data echo systems, distributed data computing, Map/Reduce, Hadoop, Hive, Spark, Kafka, storm, elastic search, solr
- Solid understanding and experience with Data warehouse concepts a practice.
- knowledge and experience with C, Java, JavaScript and APEX.
- Knowledge of a variety of machine learning techniques (clustering, decision tree learning, artificial neural networks, etc.) and their real-world advantages/drawbacks.
- Knowledge in statistical and data mining techniques: GLM/Regression, Random Forest, Boosting, Trees, text mining, social network analysis, etc.
- Knowledge using web services: Redshift, S3, DigitalOcean, etc.
- Knowledge and Experience analyzing data from 3rd party providers: Google Analytics, Site Catalyst, Coremetrics, Adwords, Crimson Hexagon, Facebook Insights, etc.
- Experience visualizing/presenting data using plotly, dash, D3, zeppelin, jupyter
- Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, apache NIFI, etc.
- Experience with object-oriented/function programming/ scripting languages
