Mani K.

Data Engineer

Mani is a highly experienced Data Engineer with over nine years of commercial expertise, including seven years dedicated to data pipeline design and ETL development. Proficient in Python, SQL, AWS, and big data technologies, he specializes in building enterprise data warehouses, automating data validation processes, and implementing robust data governance strategies.

His key achievements include developing a Change Data Capture (CDC) solution in AWS Glue for seamless Oracle-to-S3 synchronization, migrating GCP Data Prep workflows to BigQuery using stored procedures, and optimizing costly Alteryx workflows by implementing scalable PySpark solutions. These contributions have significantly improved efficiency and reduced operational costs.

With a strong technical background and a strategic approach to data engineering, Mani consistently delivers high-impact solutions that enhance data reliability, scalability, and cost-effectiveness.

Huvudsaklig expertis

Apache Airflow 5 år
CSV 7 år
Redshift 5 år

Andra kompetenser

BigQuery 8 år
QA 3 år
Tableau 2 år

Mani K.

United States

Hitta en utvecklare

Utvald erfarenhet

Anställningar

Sr Data Engineer

Samach Innovations LLC - 9 år 6 månader

● Designed, implemented, and maintained data integration and ETL (Extract, Transform, Load) pipelines to move and transform data between various systems and databases.

● Ensured data quality, consistency, and reliability by performing data validation and error handling within integration workflows.

● Created and managed workflows using Apache Airflow, scheduling and orchestrating data integration tasks, ensuring timely execution, and monitoring job statuses.

● Customized and optimized Airflow DAGs (Directed Acyclic Graphs), Apache Ni to meet specific data pipeline requirements..

● Utilized Snowflake data warehousing platform for data storage and processing, including creating and managing databases, schemas, and tables.

● Worked on Snowflake stages for data ingestion and developed efficient data loading strategies.

● Developed and optimized complex SQL queries to extract, transform, and analyze data from databases

● Implemented performance tuning techniques to improve query execution times and optimize database performance.

● Wrote Python scripts and code to automate data extraction, transformation, and loading processes.

● Developed custom Python functions, Scala and modules to perform data manipulations and transformations.

● Managed data transfers between systems via SFTP and API integrations.

● Dealt with various data file formats, including Parquet, CSV, SAV, and GZ, ensuring compatibility and efficient processing.

● Conducted data type conversions and transformations to prepare data for analysis and reporting.

● Ensured compatibility between data types used in source and target systems
Teknologier:
- Teknologier:
- Apache Airflow
- BigQuery
- CSV
- Redshift
- AWS Athena
- AWS Glue
- Python
- SQL
- Snowflake
- Data Engineering
- Database testing
- VSCode