Mani K.

● Designed, implemented, and maintained data integration and ETL (Extract, Transform, Load) pipelines to move and transform data between various systems and databases.

● Ensured data quality, consistency, and reliability by performing data validation and error handling within integration workflows.

● Created and managed workflows using Apache Airflow, scheduling and orchestrating data integration tasks, ensuring timely execution, and monitoring job statuses.

● Customized and optimized Airflow DAGs (Directed Acyclic Graphs), Apache Ni to meet specific data pipeline requirements..

● Utilized Snowflake data warehousing platform for data storage and processing, including creating and managing databases, schemas, and tables.

● Worked on Snowflake stages for data ingestion and developed efficient data loading strategies.

● Developed and optimized complex SQL queries to extract, transform, and analyze data from databases

● Implemented performance tuning techniques to improve query execution times and optimize database performance.

● Wrote Python scripts and code to automate data extraction, transformation, and loading processes.

● Developed custom Python functions, Scala and modules to perform data manipulations and transformations.

● Managed data transfers between systems via SFTP and API integrations.

● Dealt with various data file formats, including Parquet, CSV, SAV, and GZ, ensuring compatibility and efficient processing.

● Conducted data type conversions and transformations to prepare data for analysis and reporting.

● Ensured compatibility between data types used in source and target systems