AI/ML Python Developer
BrainBox -
Riyadh, Saudi ArabiaJob Details
Experience Needed:
Career Level:
Education Level:
Salary:
Job Categories:
Skills And Tools:
Job Description
Job Description
We are seeking an experienced Senior Software Engineer to lead the development of a scalable AI system. This system. As a key member of our engineering team, you will be responsible for designing, implementing, and optimizing the architecture that supports millions of users.
Job Responsibilities
Vector Database Management:
- Set up and optimize a unified vector store (e.g., Milvus, Weaviate) capable of handling large-scale data with metadata and access control
- Implement efficient similarity search algorithms with metadata filtering based on user permissions
Data Ingestion and Processing Pipeline:
- Build robust pipelines to ingest, preprocess, and embed documents from users' synced folders
- Implement content hashing and duplicate detection mechanisms to identify shared files
Access Control and Security:
- Develop fine-grained access control systems at the document level, including authentication and authorization mechanisms
- Implement and manage Access Control Lists (ACLs) for shared documents, ensuring data privacy and compliance with regulations like GDPR
Retrieval Optimization:
- Optimize query processing to include user context and permissions, ensuring efficient retrieval of authorized documents only
- Implement caching strategies and optimize data structures for quick permission checks without compromising security
Integration with Language Models:
- Integrate retrieved data with Large Language Models (LLMs) to generate contextually relevant responses
- Ensure that LLM outputs are secure and do not leak sensitive information
Performance and Scalability:
- Ensure the system can scale horizontally to handle increasing loads without performance degradation
- Implement monitoring and logging to track system performance, reliability, and security
Collaboration and Leadership:
- Work closely with cross-functional teams, including data scientists, frontend developers, and product managers
- Provide technical leadership, mentoring junior engineers, and driving best practices in code quality and system design
Job Requirements
Requirements
Education:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Experience:
- 5+ years of experience in software engineering with a focus on backend or systems development.
- Proven experience in building large-scale, distributed systems.
- Technical Skills:
- Programming Languages: Proficiency in Python, Java, or Go
- Vector Databases: Hands-on experience with vector databases like Milvus, Weaviate, Pinecone, or Elasticsearch with k-NN plugin
- Machine Learning: Understanding of embedding models and experience with frameworks like Sentence Transformers
- Access Control Systems: Experience implementing authentication and authorization systems; familiarity with OAuth, OpenID Connect, or similar protocols
- Containerization and Orchestration: Proficiency with Docker and Kubernetes for deploying and scaling applications
- Cloud Services: Experience with AWS, Google Cloud Platform, or Azure
- APIs and Microservices: Strong experience in building and consuming RESTful APIs and microservices architecture
- Security Practices: Knowledge of encryption protocols, secure coding practices, and data protection regulations (e.g., GDPR).
Soft Skills:
- Problem-Solving: Strong analytical and critical thinking skills
- Communication: Excellent verbal and written communication skills
- Team Collaboration: Ability to work effectively in a team environment and collaborate with cross-functional teams
- Leadership: Experience leading projects or teams is highly desirable
Preferred Qualifications
- LLM Integration: Experience integrating systems with Large Language Models like GPT-4 or similar
- Access Control Frameworks: Familiarity with Open Policy Agent (OPA) or similar access control frameworks
- Monitoring Tools: Experience with monitoring and logging tools such as Prometheus, Grafana, and the ELK Stack (Elasticsearch, Logstash, Kibana)
- Security Certifications: Certifications such as CISSP, CISM, or related credentials
- Compliance Knowledge: Understanding of data protection laws and compliance standards, including GDPR and CCPA
Job Benefits
What We Offer
- Work Flexibility
- Opportunity to work on cutting-edge technology in the AI and machine learning space and learn on cutting edge tools
- A collaborative, inclusive, and dynamic work environment
- Professional development opportunities and career advancement
