Data Engineer SQL - Remote (m/w/d)

Job Title: Data Engineer Job Type: Full-time Scrumconnect is a leading force in technology consultancy, proudly contributing to over 20% of the UK’s most significant citizen-facing public services. Our award-winning team has made a substantial impact, delivering more than 64 services in the past two years alone. At Scrumconnect, we foster a community of talented consultants who thrive on collaboration, sharing knowledge, and continuous learning to address and solve complex challenges. Our mission is to combine advanced software engineering, human-focused design, and data-driven insights to deliver unparalleled service to our clients. We are looking to fill the position of Data Engineer to help develop and maintain data products. Data Engineering teams are responsible for the delivery and operational stability of the data products built and provide ongoing support for those products. Data Engineers work within, and contribute to, the overall data development lifecycle process as part of multi-functional Agile delivery teams focused on one or more products.  Design, build, and operate simple, repeatable ETL data pipelines within distributed processing environments and cloud platforms, as well as localized single-node processing environments. Develop and produce prototyped then productionized code that can be deployed across a range of ETL, data validation, and other data production processes. Develop understanding of the native tooling of one or more of: GCP, Azure, AWS. An intermediate or better level coding in one or more mainstream coding languages (i.e., Python, SQL, Java, Scala, and R), and critically review the code of other data engineers. Develop code for a range of data products including data matching, rule development, scans, operational outputs. Undertake unit testing to support common code development. Review business requirements to ensure they are clear and robust, and transform requirements into reusable production-ready code and/or effective data models. Understand the key principles of database design and be able to resolve technical problems in databases, data processes, data products, and services as they occur. Apply correct techniques in normalizing data and building robust relational structures in database products. Undertake basic data analysis, for example, for data profiling, QA, or problem resolution. Support queries from end users on data quality issues affecting data production; Undertake source system analysis and data profiling to confirm data quality and ensure accurate metadata. Understand relevant data sources, tools, and systems. Work with experts to develop validation frameworks for both simple and complex data sources. Using a working knowledge of cloud data engineering tools in assisting with the development of data pipelines, products, and automation processes. Work with other team members in development of tool performance logging and monitoring. Work with other team members in applying engineering team and community best practice in data products and pipelines. Work with the team in implementing the transition to modern data platforms including Data Warehouse, Lakehouse, across the data products, pipelines, and processes you are responsible for. Engage with other professional communities (Data Science, Architecture, etc.) to identify emerging and cross-community issues affecting your role and escalate as necessary. Describe technical, data, pipeline, and production issues to colleagues of different specialisms. Communicate within the team and across teams to monitor expectations around delivery of data engineering, products, and pipelines, blockers, priorities, and issues. Familiar with developing data pipelines and products for very large volume 'big data' series using a range of native engineering tools and practices and coding approaches. A working knowledge of engineering standards across a platform and native toolset. Typical Data Engineering Experience required (5+ yrs): Knowledge and experience of Azure/AWS Cloud data solution provision. Proficient in SQL . Ability to develop and deliver complex visualisation, reporting and dashboard solutions using tools like Power BI . Experience of data modelling and transforming raw data into datasets and extracts. Experience of working in a large project /scale complex organisation and knowledge of migrating legacy capabilities. Experience in Agile. Experience of building team capability through role modelling, mentoring, and coaching. Ability to design, write, and operate ETL pipelines, in the context of distributed processing, applying coding, data, and documentation standards, in the language required by the business area. Understanding of the principles of data processing in a distributed and or cloud platform, and ability to use this understanding to ensure robust coding in a distributed or cloud environment. Able to write clean, efficient and well documented code for data processing tasks to a specification. Ability to undertake simple data and code analysis for effective quality assurance and to resolve processing issues. Ability to undertake simple data and code analysis for effective quality assurance and to resolve processing issues. Experience of one or more programming/coding languages listed: Python/PySpark, SQL, Proc SQL, NoSQL, MySQL, SQLite, Spark SQL, Hive SQL, PostgreSQL, SAS, SAS E-guide, Scala, RegEx, Java, R Undertake data profiling and source system analysis for data evaluation, issue resolution or data standardisation. Use metadata repositories to complete complex tasks such as data and systems integration impact analysis. Good knowledge of database structures, practices, principles of database integrity etc. Basic knowledge of applying database principles and SQL coding across a range of platform database and data querying tools (i.e. SQL Server, Cloud SQL, Big Query, Hive, Athena etc.) Show an awareness of opportunities for innovation with new tools and uses of data. Experience in more than one of the following tools is required for Engineers engaging in BI development: Plotly, R Shiny, Tableau, QlikView/Qlik sense, Power BI, SAP, Business Objects, MicroStrategy NiFi, Hbase, Bash, Assist, Putty, Neo4J, Spark , Kafka, HDFS, Oozie, Git Hub, Unix, Hadoop, Impala, DoJo, Flume, Elastic, Logstash, Kibana, Airflow, Glue, Big Query, Athena, CML, Hive, Informatica, CuteFTP Ability to explain and communicate technical concepts in non-technical language. Design, build and test data products based on feeds from multiple systems, using a range of different storage technologies, access methods or both. Able to explain and implement the concepts and principles of data modelling. Ability to create and run simple unit tests. Certifications in AWS, Azure, Databricks, or related technologies. Experience with public sector data initiatives and compliance requirements. Knowledge of machine learning and artificial intelligence concepts. Your working time at Scrumconnect will be split between multiple locations, including from our HQ and hub locations, client site or home. Travel requirements vary in frequency and take into account requirements of your work, our clients and the team. The diversity of our people should be reflected in the impact we deliver.