AI Data Engineer
- Location
- Bengaluru, KA, India
- Work mode
- On-site
- Employment type
- Full-time
About the job
Experience: 2 - 5 years
About ZenteiQ ZenteiQ is a deep-tech company born out of IISc Bangalore, building Scientific Intelligence Infrastructure: physics-native AI for engineering, manufacturing, energy, mobility and national systems. Rather than wrapping general-purpose language models, we train foundation models (BrahmAI) from scratch to reason over thermal, electromagnetic, structural and materials domains, and put them to work through industrial platforms (KogneX) and a talent OS for engineers and researchers (AhamX). We're backed by the IndiaAI Mission (MeitY) and work closely with IISc, ARTPARK and a national AI Hub Network — building the sovereign AI infrastructure that India's engineering and industrial systems will run on.
About the Role At ZenteiQ we're building the data foundation that powers next-generation Large Language Models. As a Data Engineer on the corpus team, you'll build scalable pipelines to source, process, curate, filter and prepare large-scale corpora for training and evaluating foundation models. You'll work closely with AI Researchers, ML Engineers and Platform Engineers to solve challenging data problems across web-scale datasets, unstructured data, multilingual corpora and benchmark datasets.
What You'll Do
- Source and ingest large-scale datasets from Common Crawl, public web sources, open-source repositories and other data sources.
- Build scalable data processing pipelines for large unstructured corpora.
- Develop and optimise distributed data pipelines using tools such as Apache Spark, Ray and Dask.
- Extract and process content from web pages and documents using Trafilatura and related web/content extraction libraries.
- Design data cleaning, filtering, quality assessment and curation pipelines for LLM pretraining corpora.
- Develop heuristic-based filters to remove low-quality, boilerplate, spam, duplicated or otherwise unsuitable content.
- Implement document-level and corpus-level deduplication strategies.
- Identify and mitigate benchmark contamination and evaluation-data leakage in training corpora.
- Prepare and curate benchmark and evaluation datasets.
- Work with large-scale datasets stored and processed through GCS/GCP.
- Monitor data quality, pipeline performance, storage and processing efficiency.
- Collaborate with AI Researchers and ML Engineers to continuously improve corpus quality and training-data pipelines.
What We're Looking For Must Have
• 2–5+ years of experience in Data Engineering, AI Data Engineering, ML Data Engineering or a related field.
- 2–5+ years of experience in Data Engineering, AI Data Engineering, ML Data Engineering or a related field.
- Strong Python programming skills.
- Experience working with large-scale unstructured datasets.
- Experience building data processing / ETL pipelines.
- Strong understanding of data cleaning, filtering, validation and deduplication.
- Hands-on experience with exact and near-duplicate detection and deduplication.
- Familiarity with at least one distributed processing framework: Apache Spark, Ray or Dask.
- Understanding of web-scale data ingestion, such as Common Crawl or other large public datasets.
- Experience with HTML parsing and web/document extraction libraries such as Trafilatura, BeautifulSoup, lxml or Resiliparse.
- Familiarity with GCS/GCP and cloud-based data processing.
- Experience working with large-scale formats such as JSONL, Parquet or Arrow.
- Strong problem-solving and ownership mindset.
- Experience with LLMs, Foundation Models or Generative AI.
- Experience building or maintaining LLM pretraining corpora.
- Familiarity with Hugging Face Datasets, DataTrove or NVIDIA NeMo Curator.
- Experience with WARC/WET/WAT and large-scale Common Crawl processing.
- Knowledge of MinHash, fuzzy or semantic deduplication.
- Research-level understanding of synthetic data generation techniques for model training and evaluation.
- Familiarity with Rust for high-performance data processing.
- Experience with multilingual or domain-specific corpora.
Why ZenteiQ
- Build real physics-native foundation models from scratch, not another LLM wrapper — deep, defensible technical work.
- Be part of a nationally recognised mission: one of 8 startups selected under the government's IndiaAI Mission to build a sovereign foundation model.
- Work alongside IISc-trained scientists and researchers, in a company founded by an IISc professor.
- See your work land in real industry pilots across automotive, mobility, defence and industrial R&D, with measurable impact.
- Join a lean, high-calibre team at an early, high-ownership stage.
From Zenteiq Aitech's careers page (Keka), checked 30 Oct 2026.
How to apply
- Select Apply on zenteiq.ai. It opens Zenteiq Aitech's own application form in a new tab, run on Keka.
- Fill in the form and upload your résumé there. Jobiwork never sees your application.
- Zenteiq Aitech will reply to you directly. No real employer asks you for money to apply.
Posted 4 Sept 2026 · checked 30 Oct 2026 · Report this job