Founding Engineer, Data Platform

• One of our main initiatives is to broaden the sorts of tasks you can complete in Elicit. We need a data engineer to figure out the best way to ingest massive amounts of heterogeneous data in such a way as to make it usable by LLMs. We need your help to integrate into our customers' custom data providers to that they can create task-specific workflows over them. • In general, we're looking for someone who can architect and implement robust, scalable solutions to handle our growing data needs while maintaining high performance and data quality. • Data pipeline: Python, Flyte, Spark • Probably less relevant to you, but ICOI: • Backend: Node and Python, event sourcing • Frontend: Next.js, TypeScript, and Tailwind • We like static type checking in Python and TypeScript! • All infrastructure runs in Kubernetes across a couple of clouds • We use GitHub for code reviews and CI • We deploy using the gitops pattern (i.e. deploys are defined and tracked by diffs in our k8s manifests) • Am I a good fit? • Consider the questions: • How would you optimize a Spark job that's processing a large amount of data but running slowly? • What are the differences between RDD, DataFrame, and Dataset in Spark? When would you use each? • How does data partitioning work in distributed systems, and why is it important? • How would you implement a data pipeline to handle regular updates from multiple academic paper sources, ensuring efficient deduplication? • If you have a solid answer for these—without reference to documentation—then we should chat! • Location and travel • We have a lovely office in Oakland, CA; there are people there every day but we don't all work from there all the time. It's important to us to spend time with our teammates, however, so we ask that all Elicians spend about 1 week out of every quarter with teammates. • We wrote up more details on this page. • Building and optimizing our academic research paper pipeline • You'll architect and implement robust, scalable systems to handle data ingestion while maintaining high performance and quality. • You'll work on efficiently deduplicating hundreds of millions of research papers, and calculating embeddings. • Your goal will be to make Elicit the most complete and up-to-date database of scholarly sources. • Expanding the datasets Elicit works over • Our users want Elicit to work over court documents, SEC filings, … your job will be to figure out how to ingest and index a rapidly increasing ontology of documents. • We also want to support less structured documents, spreadsheets, presentations, all the way up to rich media like audio and video. • Larger customers often want for us to integrate private data into Elicit for their organisation to use. We'll look to you to define and build a secure, reliable, fast, and auditable approach to these data connectors. • Data for our ML systems • You'll figure out the best way to preprocess all these data mentioned above to make them useful to models. • We often need datasets for our model fine-tuning. You'll work with our ML engineers and evaluation experts to find, gather, version, and apply these datasets in training runs. • Your first week: • Start building foundational context • Get to know your team, our stack (including Python, Flyte, and Spark), and the product roadmap. • Familiarize yourself with our current data pipeline architecture and identify areas for potential improvement. • Make your first contribution to Elicit • Complete your first Linear issue related to our data pipeline or academic paper processing. • Have a PR merged into our monorepo, demonstrating your understanding of our development workflow. • Gain understanding of our CI/CD pipeline, monitoring, and logging tools specific to our data infrastructure. • Your first month: • You'll complete your first multi-issue project • Tackle a significant data pipeline optimization or enhancement project. • Collaborate with the team to implement improvements in our academic paper processing workflow. • You're actively improving the team • Contribute to regular team meetings and hack days, sharing insights from your data engineering expertise. • Add documentation or diagrams explaining our data pipeline architecture and best practices. • Suggest improvements to our data processing and storage methodologies. • Your first quarter: • You're flying solo • Independently implement significant enhancements to our data pipeline, improving efficiency and scalability. • Make impactful decisions regarding our data architecture and processing strategies. • You've developed an area of expertise • Become the go-to resource for questions related to our academic paper processing pipeline and data infrastructure. • Lead discussions on optimizing our data storage and retrieval processes for academic literature. • You actively research and improve the product • Propose and scope improvements to make Elicit more comprehensive and up-to-date in terms of scholarly sources. • Identify and implement technical improvements to surpass competitors like Google Scholar in terms of coverage and data quality.