We are seeking a highly skilled and motivated Big Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in designing, developing, and optimizing scalable data solutions using the Hadoop ecosystem, with a strong focus on PySpark and Hive.
Responsibilities
- Design, develop, and maintain efficient and scalable Big Data solutions using PySpark, Apache Hive, and Hadoop ecosystem tools (e.g., Sqoop)
- Should have strong Python knowledge
- Implement and optimize ETL (Extract, Transform, Load) processes and data warehousing solutions, including Fact, Dimension, and Slowly Changing Dimensions (SCD-2)
- Conduct in-depth data analysis, troubleshoot complex data issues, and ensure the accuracy, reliability, and integrity of data
- Optimize Big Data workflows, including Spark job tuning and Hive query optimization, leveraging partitioning strategies and indexing techniques in distributed storage systems
- Perform rigorous unit testing and validation of data pipelines and transformations
- Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver robust data solutions
Skills & Qualifications
- Big Data Technologies: Demonstrated proficiency with Apache Hadoop, Apache Hive, and PySpark for data processing and analysis
- Data Warehousing & Modeling: Strong understanding and practical experience with data warehousing concepts, dimensional modeling, and SCD-2 implementation
- ETL Development: Proven experience in designing and developing ETL pipelines; familiarity with various ETL tools is an advantage
- Database & SQL: Advanced SQL knowledge, including complex joins, subqueries, and performance tuning of SQL queries
- Scripting: Proficient in shell scripting for automation of batch processes
- DevOps & CI/CD: Experience with CI/CD tools such as Bitbucket and Jenkins
- BI Tools: Familiarity with business intelligence (BI) reporting tools like Tableau
Experience
- Extensive experience in designing, developing, and optimizing scalable data solutions using the Hadoop ecosystem
- Strong focus on PySpark and Hive
Education
- Bachelor’s degree/University degree or equivalent experience