Research platform

Data Sherpa

LLM-based chatbots that help scientists analyze the data and the scientific knowledge produced by large research projects.

One idea, many Sherpas

Each Sherpa is a conversational assistant dedicated to a single scientific project. They all share the same foundation, so a new one can be launched on its own subdomain.

1

Ask

Scientists pose questions about a project in plain language.

2

Retrieve

The Sherpa looks up relevant material from the project's data and knowledge, indexed in a vector database.

3

Answer

A language model composes the response using what was retrieved.

The Sherpas

Every project gets its own address under sherpa.ncsa.illinois.edu.

Dark Energy Survey

SherpaDES

An assistant for the data and scientific knowledge produced by the Dark Energy Survey in its 10 years of operation.

des.sherpa.ncsa.illinois.edu In development
Terahertz Intensity Mapping

SherpaTIM

An assistant for the Terahertz Intensity Mapping (TIM) project.

tim.sherpa.ncsa.illinois.edu Coming soon
Proof of concept

SherpaPOC

The original prototype where the approach was first tested, before the move to Open WebUI.

poc.sherpa.ncsa.illinois.edu Coming soon
Next

Your project here

New Sherpas are added as new projects join, each with its own subdomain.

<project>.sherpa.ncsa.illinois.edu

How it is built

A shared base serves every Sherpa, so adding a project does not mean rebuilding the platform.

Traefik HTTPS with automatic Let's Encrypt certificates, routing each subdomain to the right Sherpa
Sherpa apps on Open WebUI One containerized app per project, each with its own database and data folder
PostgreSQL with pgvector One database service for all Sherpas: application data, plus vectors and embeddings for retrieval
Jetstream2 virtual machine with a persistent volume Everything runs in Docker on one cloud VM; apps and databases live on a volume that outlasts the VM

The base configuration is scripted, so the whole environment can be recreated on a fresh virtual machine. App images are built separately and published to a container registry.