Data Engineer

Ingested data from multiple sources to support the new data platform;

Built data pipelines and developed Databricks notebooks and Apache Spark jobs using Python to efficiently transform data;

Orchestrated Databricks jobs with a lightweight alternative to Apache Airflow, optimizing resource usage;

Implemented real-time Apache Spark streaming jobs for continuous data processing;

Provided coaching and mentorship to team members, improving overall team capabilities and project success;

Achieved the successful deployment of a new data platform, which is currently ingesting and transforming millions of records to support critical business functions such as charging sessions and invoicing.

Led the software engineering team in developing a new solution based on master reference data, ensuring team alignment and project success;

Designed and engineered a backend system from the ground up using Rust to efficiently process data from any triple store;

Integrated machine learning models into the backend, enabling advanced data processing and analysis capabilities;

Delivered a state-of-the-art system that empowers users to explore and interact with their data universe in a safe, compliant, and ethical manner.

Built pipelines to gather and manipulate data from various sources, ensuring smooth data flow and integration;

Developed and managed Python scripts to preprocess data from specific sources for optimal processing and storage;

Developed and maintained Apache Spark applications using Scala for large-scale data processing;

Orchestrated Spark applications using Apache Airflow, automating workflows for efficient data management;

Architected and built a backend engine in Golang to integrate with DataSingularity, reconciling, arbitrating, linking, and ingesting data into an RDF datastore.

Prioritized requirements and requests to ensure alignment with the needs of criminal investigations;

Installed and managed Elasticsearch, Cassandra, Hadoop, and OpenNebula clusters in a private data center;

Managed the deployment and maintenance of investigative software on these clusters, ensuring smooth operations;

Implemented monitoring systems using Prometheus, Alert Manager, and Grafana to ensure platform reliability;

Developed Grafana dashboards for real-time insights and system health monitoring;

Created Terraform code and Ansible playbooks for automating cluster and software deployments.

Delivered a high-performing data platform that efficiently processes and investigates seized data such as emails, chat messages, and photos, enabling swift analysis during criminal investigations, including cases of fraud, and murder.

Designed and developed a scalable system to automatically extract key measurements from financial statements, including those from scanned PDF pages;

Architected both the technical and application aspects of the system to ensure flexibility and scalability;

Developed crawlers to download and process market figures from various sources;

Created APIs for integration with other internal systems, enabling smooth data exchange;

Developed and optimized REST services to enhance the system's performance and responsiveness;

Tested the software to ensure accuracy and reliability in processing financial data;

Implemented microservices architecture to enhance modularity and scalability;

Deployed the software on Azure using Docker containers for efficient cloud-based operations;

The software went live in January 2019, successfully integrating with other internal systems to automate the download, processing, and storage of financial data.

Designed and developed a big data solution using Scala, Spark, Kafka, and Cassandra to calculate availability, multi-currency rates, and inventory for Landal Greenparks;

Architected both the technical and application aspects of the system to ensure scalability and performance;

Developed a real-time streaming solution to continuously track availability throughout the day;

Built and optimized the data model to handle large-scale processing efficiently;

Developed and fine-tuned REST services for seamless communication with other systems;

Thoroughly tested the software to ensure reliability and performance;

Deployed the software into production, ensuring smooth integration with existing systems.

Successfully launched the software in December 2017, within time and budget, replacing a legacy system and reducing processing time from 15 hours to 15 minutes. This improvement enabled Landal Greenparks to offer additional Christmas promotions, contributing to increased revenue.

Designed and developed Spark and MapReduce jobs to efficiently process millions of bibliographic records;

Created scripts to control and manage data pipelines for seamless data ingestion and processing;

Developed a web service capable of ingesting bibliographic records in various formats, ensuring compatibility and scalability;

Collaborated closely with international colleagues to coordinate development and deployment efforts across multiple regions;

Successfully launched several big data systems during my time at OCLC, enabling the processing of terabytes of bibliographic data.

Designed and developed a point-of-sale (POS) system to streamline sales transactions and improve customer experience;

Architected and developed an invoice administration system to manage billing and payment processes efficiently;

Designed and implemented an inventory tracking system to monitor stock levels and optimize inventory management;

Created a comprehensive data model to support all systems, ensuring data consistency and scalability;

Developed and optimized REST services for seamless communication between various components of the system;

Conducted integration testing to ensure smooth functionality across the POS, invoice administration, and inventory management systems;

Designed the technical and application architecture for the entire solution to ensure robustness and scalability;

Successfully launched the POS, invoice administration, and inventory management systems during my time at Hakim, delivering a fully integrated solution that improved operational efficiency.