Nvidia and AI4Bharat announce multilingual Indic pretraining corpus
Nvidia and research lab AI4Bharat will release a 2.8 trillion token pretraining corpus covering 22 official Indian languages.
FTMQ SI, written by our newsroom0 views
Nvidia and research lab AI4Bharat have announced a new pretraining dataset called Nemotron pretraining Indic v1, which contains approximately 2.8 trillion tokens across 22 Indian languages. The Hugging Face Blog and the Chenab Times both report that the dataset includes native script data for 22 languages alongside romanized data for 12 languages. [1][2]
The Hugging Face Blog reported that the dataset will be released soon alongside an open data generation pipeline. The GPU accelerated pipeline is designed for translation, quality assessment, and text repair to process multilingual datasets. [1]
According to the Hugging Face Blog, the project targets the 22 languages listed in the Constitution of India. These languages are spoken as a first language by nearly 1.2 billion people, representing 97 percent of the population of India. [1]
Large language models rely on vast self supervised text datasets for natural language processing tasks such as text generation. The authors from AI4Bharat and Nvidia plan to open source the data pipeline to assist future multilingual model pretraining. [1][4]
In short
- The Nemotron pretraining Indic v1 dataset contains 2.8 trillion tokens across 22 Indian languages.
- The project includes native script data for 22 languages and romanized data for 12 languages.
- Nvidia and AI4Bharat developed an open GPU accelerated pipeline for translation and quality assessment.
- The targeted languages are spoken as a first language by 97 percent of the population of India.
Sources
Every paragraph above points to the numbered items it rests on. Read the originals here.
- [1]Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation PipelineHugging Face Blog, 1d ago (the report this story comes from)
- [2]NVIDIA Releases 2.8 Trillion Token Indic Pretraining CorpusThe Chenab Times, 18h ago
Background
- [3]Open Systems International on Wikipedia
- [4]List of large language models on Wikipedia
- [5]Llama (language model) on Wikipedia
Our newsroom writes these reports with the help of software, from the 5 sources listed and nothing else, and checks them against those sources. Facts can still be wrong or move on; the originals are the record. Spotted a mistake? Write to daniel@monsterkong.com.
More in Models & Products
- Munich returns to open source infrastructure but skips Linux desktop1h ago
- Meta updates business terms to allow automated web scraping1h ago
- Reflection AI releases open weights and post training recipes for Beam model1h ago
- Apple plans product launch event in Los Angeles on October 281d ago
- Nvidia releases open Nemotron 3.5 Super VL and Bolt models1d ago
- Alibaba Qwen Team Releases QwenGyre Online Reinforcement Learning System4d ago
Get the day in one email
Reports like this one, the top news of the last 24 hours, every morning. Free, one email a day; readers can comment under every report.
By signing up you agree to our terms and privacy policy. Unsubscribe any time.
Comments
Loading
Join the conversation
Comments are open to readers of our daily email: the top news of the last 24 hours, every morning, free. Sign up and the comment box opens.
Already on the list? Enter the same address and we will send a sign-in link.
By signing up you agree to our terms and privacy policy. Unsubscribe any time.