AI HUB Data Engineer

Data engineering is the practice of making the appropriate data available to various data consumers (including data scientists, data and business analysts, citizen integrators, and line-of-business users). It is a discipline that involves collaboration across business and IT units.

Role at a glance:

• Main focus: Data pipelines, data architecture, integration, quality, reliability, and scalable data products.

• Typical outputs: ETL/ELT pipelines, curated datasets, data models, reusable data products, monitoring logic, and technical documentation.

• Key interfaces: Data scientists, Palantir/Foundry experts, IT, cybersecurity, data governance, business analysts, and Quality stakeholders.

In addition to creating and maintaining an optimal pipeline architecture, typical duties and responsibilities for a Data Engineer position may include:

Key Responsibilities:

· Assembling large, complex sets of data that meet non-functional and functional business requirements

· Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.

· Develop robust ETL (Extract, Transform, Load) process to integrate data from various sources.

· Identifying, designing and implementing internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes

· Building required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS, Azure, Databricks, Palantir and SQL technologies

· Building scalable tables to provide actionable insight into key business performance metrics including operational efficiency and customer acquisition

· Working with stakeholders including the Data Product teams to support their data infrastructure needs while assisting with data-related technical issues

· Design and maintain data models, schemas, and database structures to support analytical and operational use cases.

· Optimize data storage and retrieval mechanisms for performance and scalability.

Basic Qualifications:

· Bachelor's degree or higher in the fields of Mechanical Engineering, Electrical Engineering or Mechanical Engineering Technology from an accredited university (other Engineering degrees, in combination with relevant experience may be considered)

· 5+ years of related experience

· Strong experience designing, building, and maintaining scalable data pipelines for structured, semi-structured, and unstructured data

· Advanced SQL skills and strong proficiency in at least one programming language used for data engineering, such as Python, PySpark, Scala, or Java

· Hands-on experience with ETL/ELT patterns, data integration, orchestration, data transformation, and workflow automation

· Knowledge of cloud and enterprise data platforms such as Azure, AWS, Databricks, Snowflake, Palantir Foundry, DB2, or equivalent technologies

· Solid understanding of data modeling, database design, data quality controls, metadata management, lineage, and governance principles

· Ability to optimize data storage, processing performance, reliability, monitoring, and cost efficiency for production-grade data products

· Experience collaborating with data scientists, AI engineers, business analysts, IT, cybersecurity, and data governance teams to enable analytics and AI use cases

· Capacity to translate business and analytical requirements into robust, reusable, documented, and maintainable data assets