Data Engineer

Successfully led the migration of ETL pipelines from SSIS to Azure Data Factory (ADF), ensuring a smooth transition and improved data integration processes;

Identified and implemented optimizations for existing queries, enhancing performance and efficiency of data retrieval and processing;

Utilized Databricks notebooks with PySpark and SparkSQL to analyze data effectively, enabling insightful data-driven decision-making;

Developed Azure Databricks notebooks to retrieve data from various social APIs, including FB, Twitter, YouTube, and Google Analytics;

Automated the ETL process for loading historical data from FB and Instagram APIs, resulting in significant time savings of 500 hours by eliminating manual effort;

Built ADF pipelines to execute Databricks pipelines on a scheduled basis, ensuring timely and accurate data processing;

Created dataflows using ADF V2 pipelines, facilitating seamless data transformation and integration;

Collaborated with clients to understand and analyze requirements, ensuring alignment with project objectives and delivering tailored solutions;

Identified and resolved issues reported by downstream applications, ensuring data integrity and smooth data flow;

Automated the data load process, reducing manual intervention and improving efficiency;

Enhanced data load performance from APIs, implementing strategies to optimize data retrieval and processing for downstream applications;

Provided suggestions and approaches to load data faster for downstream applications, optimizing overall data processing and delivery;

Leveraged Delta tables in Databricks for performing upsert operations, ensuring data consistency and integrity;

Managed offshore teams, providing BAU support, overseeing stakeholder management, and producing process and technical documentation.

Demonstrated solid analytical skills by understanding requirement specification documents and developing SQL queries accordingly;

Developed efficient and optimized stored procedures based on the requirements, ensuring adequate data processing and retrieval;

Conducted proof of concept (POC) to assess the feasibility and benefits of migrating the existing solution to Azure, providing valuable insights for decision-making;

Identified and implemented performance improvements in existing stored procedures and SQL functions, enhancing query execution speed and overall system efficiency;

Designed and created a reusable SSIS (SQL Server Integration Services) package to load data iteratively, streamlining the data loading process and improving data integration capabilities.

Building an Azure-based data warehouse and database platform for fast analysis and decision-making;

Responsible for ELT processes, extracting data from SAP HANA, SAP BW, and SAP ERP to load into the database;

Utilize Power BI for data visualization, providing valuable insights from the extracted data;

Design and optimize data models and structures for efficient storage and retrieval;

Ensure data quality through cleansing, transformation, and enrichment activities;

Monitor and troubleshoot database performance, resolving issues as needed;

Collaborate with stakeholders to understand reporting and analytical needs;

Stay updated with industry trends and incorporate new technologies for improved data analysis.

Developed ETL packages in SSIS for efficient data extraction, transformation, and loading;

Gathered requirements from stakeholders to ensure a clear understanding of business needs;

Designed and built Tabular and Multi-Dimensional SSAS cubes for comprehensive data analysis;

Created visually appealing Power BI reports and dashboards to provide valuable insights;

Conducted data profiling and cleansing to maintain data quality and integrity;

Optimized performance of ETL packages, SSAS cubes, and Power BI reports;

Stayed updated with the latest ETL, SSAS, and Power BI technologies;

Provided guidance and support to junior team members;

Collaborated with stakeholders to understand evolving business needs and make necessary adjustments to ETL processes, SSAS cubes, and Power BI reports.