NASA’s Science Mission Directorate (SMD) Artificial Intelligence and Machine Learning (AIML) Working Group recently launched a Large Language Model (LLM) specially created to help the SMD manage data more efficiently. The model, developed in collaboration with IBM Research, will improve tasks such as assigning metadata, managing documentation, and intelligent search.

To train the model, the researchers used material from a variety of scientific sources related to the SMD’s subject matter areas. The largest percentage of the training data came from the NASA Astrophysics Data System (ADS), with the American Geophysical Union (AGU), the American Meteorological Society (AMS), and PubMed.

The new LLM was tested using multiple benchmarks that assess a model’s ability to reason and identify relevant information. In particular, the researchers used BLURB (Biomedical Language Understanding and Reasoning Benchmark) to measure the model’s ability to answer questions and classify text related to the biomedical field. The model also underwent testing on a variety of NASA-relevant scientific questions with SQUAD2 (Stanford Question Answering Dataset), which grades a model’s ability to answer reading comprehension questions or abstain when a question is impossible to answer.

The team also tested the model with a NASA SMD-specific benchmark. On every test, the new model showed a marked improvement in a variety of information-related tasks over the pre-trained model it was based on. The SMD encoder-only transformer model and the subsequently-refined SMD bi-encoder sentence transformer model are available on GitHub.

SMD is currently leveraging the new model to create a more robust search feature for NASA’s Science Discovery Engine. In the future, the SMD hopes to use LLMs to assist with a variety of data management tasks.