Data Engineer

Developed and maintained real-time messaging services using Kafka in Java (Micronaut);

Optimized data storage and processing pipelines with Protobufs and Clickhouse, ensuring efficient performance monitoring through Datadog;

Managed AWS infrastructure, including Kubernetes and ECR, and implemented CI/CD pipelines utilizing GitHub Actions and ArgoCD;

Enhanced team workflows and project management by applying Agile methodologies, including Sprints and Kanban, using tools like Jira and Confluence.

Engineered real-time event pipelines in AWS (Firehose, S3, Athena, Quicksight, RDS, Redshift) and Azure (PowerBI), achieving a 30% improvement in data processing efficiency;

Maintained a codebase written in Python (with code quality ensured via Black, Flake8, Xenon), TypeScript, Shell, SQL, and DAX;

Developed an AI-driven "Experience Mining" model using Python Petri Nets, which generated user journeys from browsing data, including clicks on screens and page navigation;

Migrated the CI/CD process from Jenkins to Airflow, including the development of custom operators, alerting mechanisms, and AWS integration, resulting in a 25% reduction in execution time;

Implemented a data firewall for data validation using the Python jsconschema library to filter out events that did not meet predefined rules;

Achieved £5k in monthly savings on cloud infrastructure costs through optimized container management;

Led the migration from deploying directly to production by building infrastructure-as-code (via CDK) and implementing gitflow processes for multiple non-prod environments, reducing service outages from over 10 per quarter to zero, enabling 24/7 uptime.

Led a team of seven in the development and maintenance of data pipelines, ensuring efficient data flow and processing across the institution;

Oversaw the implementation and optimization of digital systems, enhancing the operational efficiency of various departments within the university;

Collaborated with cross-functional teams to identify and address data-related challenges, ensuring data accuracy and reliability;

Provided strategic guidance on data management practices, contributing to the university's digital transformation efforts.

Awarded the 2020 Higher Education Strategic Planners Association (HESPA) Data Award for groundbreaking research and insights into the behavior of commuter students;

Led the Data Analytics and Business Intelligence function for an organization with approximately 4,000 staff, driving data-driven decision-making across the institution.