English
Svenska
Dansk
Deutsch
Français
Norsk
Suomi
Mani K.
Data Engineer
Mani is a highly experienced Data Engineer with over nine years of commercial expertise, including seven years dedicated to data pipeline design and ETL development. Proficient in Python, SQL, AWS, and big data technologies, he specializes in building enterprise data warehouses, automating data validation processes, and implementing robust data governance strategies.
His key achievements include developing a Change Data Capture (CDC) solution in AWS Glue for seamless Oracle-to-S3 synchronization, migrating GCP Data Prep workflows to BigQuery using stored procedures, and optimizing costly Alteryx workflows by implementing scalable PySpark solutions. These contributions have significantly improved efficiency and reduced operational costs.
With a strong technical background and a strategic approach to data engineering, Mani consistently delivers high-impact solutions that enhance data reliability, scalability, and cost-effectiveness.
Hoofd expertise
- Apache Airflow 5 jaar
- CSV 7 jaar
- Redshift 5 jaar
Andere vaardigheden
- BigQuery 8 jaar
- QA 3 jaar
- Tableau 2 jaar
Geselecteerde ervaring
Dienstverband
Sr Data Engineer
Samach Innovations LLC - 9 jaar 6 maanden
● Designed, implemented, and maintained data integration and ETL (Extract, Transform, Load) pipelines to move and transform data between various systems and databases.
● Ensured data quality, consistency, and reliability by performing data validation and error handling within integration workflows.
● Created and managed workflows using Apache Airflow, scheduling and orchestrating data integration tasks, ensuring timely execution, and monitoring job statuses.
● Customized and optimized Airflow DAGs (Directed Acyclic Graphs), Apache Ni to meet specific data pipeline requirements..
● Utilized Snowflake data warehousing platform for data storage and processing, including creating and managing databases, schemas, and tables.
● Worked on Snowflake stages for data ingestion and developed efficient data loading strategies.
● Developed and optimized complex SQL queries to extract, transform, and analyze data from databases
● Implemented performance tuning techniques to improve query execution times and optimize database performance.
● Wrote Python scripts and code to automate data extraction, transformation, and loading processes.
● Developed custom Python functions, Scala and modules to perform data manipulations and transformations.
● Managed data transfers between systems via SFTP and API integrations.
● Dealt with various data file formats, including Parquet, CSV, SAV, and GZ, ensuring compatibility and efficient processing.
● Conducted data type conversions and transformations to prepare data for analysis and reporting.
● Ensured compatibility between data types used in source and target systems
Technologieën:
- Technologieën:
Apache Airflow
BigQuery
- CSV
Redshift
- AWS Athena
AWS Glue
Python
SQL
Snowflake
- Data Engineering
- Database testing
VSCode
Educatie
MSc.Master of Computer Application
AKT(UP Technical University) Noida, India · 2007 - 2010
Vind jouw volgende ontwikkelaar binnen enkele dagen, niet maanden
In een kort gesprek van 25 minuten:
- gaan we in op wat je nodig hebt om je product te ontwikkelen;
- Ons proces uitleggen om u te matchen met gekwalificeerde, doorgelichte ontwikkelaars uit ons netwerk
- delen we de stappen met je om de juiste match te vinden, vaak al binnen een week.