Role & responsibilities

• Administer and manage the AWS Data Lake infrastructure, ensuring high availability, security, and performance.
• Configure and manage AWS S3, Lake Formation, and Glue Data Catalog to organize, secure, and catalog data within the data lake.
• Set up and manage Redshift Spectrum for querying and analyzing data stored in S3 using SQL and Redshift.
• Implement and manage data ingestion pipelines to ingest structured and unstructured data from various sources into the data lake using AWS services like Glue ETL, Lambda, and other orchestration tools.
• Define and enforce data governance policies, access control, and security measures using AWS Lake Formation, ensuring compliance with organizational and regulatory requirements.
• Optimize data storage in S3 through partitioning, compression, and appropriate data formats like Parquet, Avro, or ORC to improve query performance.
• Monitor and manage the Glue Catalog for maintaining metadata, table definitions, and data lineage within the data lake.