Bachelor's degree

Designed, developed, and maintained data pipelines using GCP data services (BigQuery, Cloud Storage, Dataflow) to facilitate efficient ETL processes.
Developed and optimized SQL queries to enhance data retrieval and analysis within the GCP environment.
Utilized Python, PySpark, and Spark to process and transform large datasets, ensuring data integrity and accuracy.
Collaborated with client stakeholders to gather requirements, analyze data, and implement effective data engineering solutions.
Analyzed and improved the efficiency, scalability, and stability of system resources, addressing complex technical challenges.
Provided mentorship to junior engineers, fostering a culture of technical excellence and knowledge sharing.
Contributed to the development and implementation of CI/CD pipelines to streamline deployment and improve software delivery.

Cloud Data Engineer

Leading cloud migration projects from on-prem Oracle environments to Google Cloud Platform (GCP).

Designed and implemented real-time & batch data pipelines using Oracle GoldenGate, Fivetran, and native GCP services (BigQuery, Cloud Storage, Pub/Sub, Dataflow, Composer).

Automated ingestion of enterprise datasets into BigQuery with Fivetran connectors for SaaS sources, ensuring schema evolution handling and data quality validations.

Orchestrated data workflows via Cloud Composer (Airflow) with monitoring, retries, and SLA alerts.

Collaborated with cross-functional teams to migrate critical reporting and analytics workloads into GCP, reducing query latency and enabling scalable, cost-efficient analytics.

Implemented best practices for security, governance, and partitioning strategies in BigQuery to optimize performance and cost.

Cloud Technical Lead

Middle Management

Designed and implemented end-to-end ETL/ELT data pipelines on GCP services and AWS EMR clusters, processing multi-terabyte datasets to ingest, transform, and load critical healthcare data from diverse sources into Snowflake.

Developed and optimized PySpark applications for complex data transformations, aggregations, and quality checks, ensuring high performance and data integrity for analytical and reporting needs.

Managed and orchestrated Snowflake data warehouse environments, including schema design, virtual warehouse sizing, performance tuning of queries, and implementing robust data governance practices.

Automated infrastructure provisioning and management across GCP and AWS using Terraform, ensuring consistency, scalability, and disaster recovery capabilities for data platforms (e.g., EMR clusters, S3 buckets, network configurations).

Collaborated closely with data scientists, analysts, and business stakeholders to understand data requirements, define data models, and deliver 

Data Engineer

Data engineering enthusiasts

Know about Shankar Arjunan's work experience, education, bootcamps, certifications, and much more. Shankar Arjunan