Omar C.

I help the data engineering team improving the data ingestion pipelines across different services

• Conduct daily data cleaning, ingestion, and processing;
• Design, develop, and perform quality control testing of data pipelines;
• Manage a small team of one data engineer and one data scientist;
• Coordinate with the CS team to provide compliant results for our client.

• Conduct daily data cleaning, ingestion, and processing;
• Perform quality checks on Q products to enhance overall performance;
• Design, develop, and perform quality control testing of data pipelines.

• Conducted integrity checks on the data pipeline across various environments using Grafana and Kibana;
• Designed and implemented new features and resolved pipeline bugs;
• Designed, developed, and maintained the data pipeline, enabling researchers to retrieve and create datasets for model training and analyze model performance;
• Developed monitoring tools used by researchers to track pipeline jobs;
• Conducted weekly code reviews with researchers to ensure a consistent coding style;
• Delivered monthly demos during business reviews to present ongoing work to a wider audience.

• Designed and developed a user-friendly Kafka Console with a web interface, providing an efficient monitoring system for Kafka topics. Calculated essential KPIs to indicate system status and estimated time for processing messages in the queues.es;
• Led the backend design of a Cloudera Web Application, designing data structures using HBase and developing methods to store aggregated performance indicators with Spark;
• Designed and developed a Real-Time Vehicles Counting system using the YOLOv3 framework. Trained an object detection model specializing in recognizing vehicles on highways and created a real-time counting system to estimate traffic on specific streets.