Data Engineer

hackajob on-Demand is currently partnering with an AI startup company to help them hire the best talent. At on-demand, we match and speak with exceptional talent like you and provide insights into the problem they are looking to solve and the interview process.

Role: Data Engineer

Opportunity: Perm or Contract

Based: London or New York (remote possible but ideally onsite)

About Us

We are a stealth-mode startup developing cutting-edge AI tools for the hedge fund industry. Our mission is to empower financial institutions with advanced data processing and analysis capabilities. We're assembling a diverse team of talented individuals to create innovative solutions that will transform how hedge funds leverage data for decision-making.

The Role

We're seeking an exceptional Data Engineer to join our core team. You'll be responsible for designing, implementing, and maintaining the robust data infrastructure that powers our AI models and financial analysis tools. In this role, you'll work closely with data scientists, machine learning engineers, and software developers to create an integrated data ecosystem that drives our innovative products.

Key Responsibilities

• Design, build, and optimize data pipelines to ingest, process, and store large volumes of financial and alternative data

• Ensure data quality, reliability, and accessibility for our ML models and analysis tools

• Implement and manage scalable data storage solutions, including time series and relational databases

• Develop data integration processes to combine data from various sources (market data, economic indicators, news feeds, etc.)

• Collaborate with ML engineers and data scientists to support data needs for model development and testing

• Implement data versioning and lineage tracking to support reproducibility and auditing

• Optimize data retrieval and querying for low-latency operations

• Ensure data security and compliance with financial industry regulations

• Participate in the full software development lifecycle, from planning to deployment and monitoring

• Contribute to the design and implementation of APIs for data access and integration

Requirements

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field

• 5+ years of experience in data engineering

• Strong programming skills in Python; experience with Java or Scala is a plus

• Expertise in designing and managing ETL processes

• Proficiency with big data technologies (e.g., Spark, Hadoop, Kafka)

• Experience with both SQL and NoSQL databases (e.g., PostgreSQL, MongoDB, InfluxDB)

• Familiarity with both cloud platforms (AWS, GCP, or Azure) and on-prem data warehousing solutions

• Knowledge of data modeling, data architecture, and data warehouse design

• Experience with version control systems (Git) and CI/CD pipelines

• Strong understanding of data security and privacy best practices

• Excellent problem-solving skills and attention to detail

• Ability to work effectively in a fast-paced, collaborative environment

• Basic understanding of machine learning concepts and data requirements for ML models

• Familiarity with software engineering best practices, including code reviews, testing, and documentation

• Experience with containerization technologies (e.g., Docker) and orchestration tools (e.g., Kubernetes)

Preferred Qualifications

• Experience in data engineering in data-intensive industries such as e-commerce, healthcare, IoT, finance, or scientific research

• Familiarity with real-time data processing and streaming technologies

• Experience with data science tools and libraries (e.g., pandas, NumPy, scikit-learn)

• Knowledge of DevOps practices and tools (e.g., Jenkins, Ansible, Terraform)

• Experience with data visualization tools (e.g., Tableau, PowerBI) and creating dashboards

• Familiarity with agile development methodologies

• Contributions to open-source data engineering or data science projects

• Basic understanding of financial markets and quantitative finance concepts (a plus, but not required)