Data Engineer

Ingested data from multiple sources to support the new data platform;

Built data pipelines and developed Databricks notebooks and Apache Spark jobs using Python to efficiently transform data;

Orchestrated Databricks jobs with a lightweight alternative to Apache Airflow, optimizing resource usage;

Implemented real-time Apache Spark streaming jobs for continuous data processing;

Provided coaching and mentorship to team members, improving overall team capabilities and project success;

Achieved the successful deployment of a new data platform, which is currently ingesting and transforming millions of records to support critical business functions such as charging sessions and invoicing.

Led the software engineering team in developing a new solution based on master reference data, ensuring team alignment and project success;

Designed and engineered a backend system from the ground up using Rust to efficiently process data from any triple store;

Integrated machine learning models into the backend, enabling advanced data processing and analysis capabilities;

Delivered a state-of-the-art system that empowers users to explore and interact with their data universe in a safe, compliant, and ethical manner.

Built pipelines to gather and manipulate data from various sources, ensuring smooth data flow and integration;

Developed and managed Python scripts to preprocess data from specific sources for optimal processing and storage;

Developed and maintained Apache Spark applications using Scala for large-scale data processing;

Orchestrated Spark applications using Apache Airflow, automating workflows for efficient data management;

Architected and built a backend engine in Golang to integrate with DataSingularity, reconciling, arbitrating, linking, and ingesting data into an RDF datastore.

Prioritized requirements and requests to ensure alignment with the needs of criminal investigations;

Installed and managed Elasticsearch, Cassandra, Hadoop, and OpenNebula clusters in a private data center;

Managed the deployment and maintenance of investigative software on these clusters, ensuring smooth operations;

Implemented monitoring systems using Prometheus, Alert Manager, and Grafana to ensure platform reliability;

Developed Grafana dashboards for real-time insights and system health monitoring;

Created Terraform code and Ansible playbooks for automating cluster and software deployments.

Delivered a high-performing data platform that efficiently processes and investigates seized data such as emails, chat messages, and photos, enabling swift analysis during criminal investigations, including cases of fraud, and murder.

Designed and developed a scalable system to automatically extract key measurements from financial statements, including those from scanned PDF pages;

Architected both the technical and application aspects of the system to ensure flexibility and scalability;

Developed crawlers to download and process market figures from various sources;

Created APIs for integration with other internal systems, enabling smooth data exchange;

Developed and optimized REST services to enhance the system's performance and responsiveness;

Tested the software to ensure accuracy and reliability in processing financial data;

Implemented microservices architecture to enhance modularity and scalability;

Deployed the software on Azure using Docker containers for efficient cloud-based operations;

The software went live in January 2019, successfully integrating with other internal systems to automate the download, processing, and storage of financial data.