English
Svenska
Dansk
Dutch
Français
Norsk
Suomi
Mani K.
Data Engineer
Mani is a highly experienced Data Engineer with over nine years of commercial expertise, including seven years dedicated to data pipeline design and ETL development. Proficient in Python, SQL, AWS, and big data technologies, he specializes in building enterprise data warehouses, automating data validation processes, and implementing robust data governance strategies.
His key achievements include developing a Change Data Capture (CDC) solution in AWS Glue for seamless Oracle-to-S3 synchronization, migrating GCP Data Prep workflows to BigQuery using stored procedures, and optimizing costly Alteryx workflows by implementing scalable PySpark solutions. These contributions have significantly improved efficiency and reduced operational costs.
With a strong technical background and a strategic approach to data engineering, Mani consistently delivers high-impact solutions that enhance data reliability, scalability, and cost-effectiveness.
Hauptkompetenz
- Apache Airflow 5 Jahre
- CSV 7 Jahre
- Redshift 5 Jahre
Andere Fähigkeiten
- BigQuery 8 Jahre
- QA 3 Jahre
- Tableau 2 Jahre
Ausgewählte Erfahrung
Beschäftigung
Sr Data Engineer
Samach Innovations LLC - 9 jahre 6 monate
● Designed, implemented, and maintained data integration and ETL (Extract, Transform, Load) pipelines to move and transform data between various systems and databases.
● Ensured data quality, consistency, and reliability by performing data validation and error handling within integration workflows.
● Created and managed workflows using Apache Airflow, scheduling and orchestrating data integration tasks, ensuring timely execution, and monitoring job statuses.
● Customized and optimized Airflow DAGs (Directed Acyclic Graphs), Apache Ni to meet specific data pipeline requirements..
● Utilized Snowflake data warehousing platform for data storage and processing, including creating and managing databases, schemas, and tables.
● Worked on Snowflake stages for data ingestion and developed efficient data loading strategies.
● Developed and optimized complex SQL queries to extract, transform, and analyze data from databases
● Implemented performance tuning techniques to improve query execution times and optimize database performance.
● Wrote Python scripts and code to automate data extraction, transformation, and loading processes.
● Developed custom Python functions, Scala and modules to perform data manipulations and transformations.
● Managed data transfers between systems via SFTP and API integrations.
● Dealt with various data file formats, including Parquet, CSV, SAV, and GZ, ensuring compatibility and efficient processing.
● Conducted data type conversions and transformations to prepare data for analysis and reporting.
● Ensured compatibility between data types used in source and target systems
Technologien:
- Technologien:
Apache Airflow
BigQuery
- CSV
Redshift
- AWS Athena
AWS Glue
Python
SQL
Snowflake
- Data Engineering
- Database testing
VSCode
Ausbildung
MSc.Master of Computer Application
AKT(UP Technical University) Noida, India · 2007 - 2010
Finden Sie Ihren nächsten Entwickler innerhalb von Tagen, nicht Monaten
In einem kurzen 25-minütigen Gespräch würden wir gerne:
- Auf Ihren Bedarf bezüglich des Recruitments von Software-Entwicklern eingehen
- Unseren Prozess vorstellen und somit wie wir Sie mit talentierten und geprüften Kandidaten aus unserem Netzwerk zusammenbringen können
- Die nächsten Schritte besprechen, um den richtigen Kandidaten zu finden - oft in weniger als einer Woche