Our client in London a brokerage firm are seeking a Data Engineer to work on the firm's Big Data Analytics platform with a specialism in Java.
The successful candidate will work with the data team to build out the existing platform to support additional technology and business use-cases involving both on-premise and Cloud implementations. The platform delivers data solutions across the group including use-cases for: market data analytics, regulatory reporting, surveillance and revenue analysis.
- To work in partnership with the business/business analysts to identify key requirements for implementation
- To identify any technical requirements for new products
- To work with all the business and technology departments to ensure all business and technical requirements are met
- To identify and manage any integration work
- To analyse, design and build any such projects
- To provide input into the current and on-going system architecture
- To liaise with other development teams as necessary to implement cross-team projects
- Be alert to Conduct Risk issues, specifically the risk of harm to client interests, market integrity and/or competition in financial markets due to inappropriate practices or behaviours across the firm
To undertake and manage:
- Systems analysis and design
- Systems development
- Systems documentation
- Production support and out-of-hours system maintenance
Skills / experience required:
- Strong Data background. Ideally worked previously on clustered data lakes with distributed computing.
- Expertise with Java. Including multithreading, profiling, performance tuning and monitoring.
- Solid database experience. Relational and NoSQL, including logical and physical schema modelling, performance optimisation and management.
- Experience with streaming data - demonstrable in-depth understanding of at least one messaging broker - including performance and monitoring.
- Experience with ETL approach including automation/scheduling - prove in-depth knowledge of at least one frameworks or tool.
- Linux/Unix (5 years) - including some shell scripting and understanding of repositories and resources (cpu, mem, disk, network)
- Strong CI/CD focus
Additional Skills / experience:
- AWS: Data Pipeline, S3, EMR, Lamda, DynamoDB, RedShift
- Hadoop and BigData stack - including Yarn, HDFS, Kafka and Spark.
- Some DevOps experience - deploying and automating. Possibly used one scripting language, puppet, chef, ansible, terraform.
- Containerization technologies - e.g. Docker, K8s
- Monitoring frameworks/tools
- Knowledge of Python is beneficial
- Experience of software development in a financial services environment advantageous
- Willingness to keep up to date with latest technology trends and proactively identify appropriate areas into which they can be applied.
- Solid Computing Degree
- Team player with excellent inter-personal skills and confident communication skills.
- Able to effectively disseminate knowledge and experience to less experienced team members.
- Must be able to deal with our customers effectively, ie development teams working across the globe using Messaging and Data products, alongside external vendors who use the products.
- Must be able to deal with and adapt to change extremely effectively.
- Must be proactive in generating ideas and effective at developing solutions that are balanced, proportionate and effective. It is critical that this is achieved in collaboration with the global team.
- Ability to work and function under pressure, handle multiple tasks, and shifting priorities
- “Self-starter” always looking to improve quality of process and deliverables and keen to take a lead role in that process