Local Development Setup

This guide walks through the sequence required to start the local infrastructure, initialize OpenSearch indexes, and run both the FastAPI backend and React frontend.

---

📋 System Prerequisites

Before starting, ensure your local development machine has the following dependencies installed:

• Python 3.12+ (Backend runtime)
• Node.js 18+ & npm (Frontend runtime)
• Docker & Docker Compose (For orchestrating OpenSearch, MinIO, and Redis)
• Git

AutoDDG (Dataset Description Generation)

The arq-worker container installs AutoDDG from PyPI automatically, so no separate clone is needed. It only requires an NYU Portkey API key to call the LLM gateway during ingestion:

cp backend/.env_sample backend/.env
# then edit backend/.env and set PORTKEY_API_KEY=<your NYU Portkey key>

backend/.env is git-ignored and must never be committed. Without the key, ingestion still runs — the description step is simply skipped.

---

🐳 Step 1: Start Core Infrastructure

Auctus v2 depends on OpenSearch, MinIO, and Redis.

From the repository root:

docker compose up -d

Port Conflict Check

Stop the placeholder backend container if it conflicts with port8000:

docker stop auctus-backend

---

🐍 Step 2: Configure and Start the Backend

Navigate to the backend directory:

cd backend

Create and activate a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

Install dependencies:

pip3 install -r requirements.txt

Initialize OpenSearch Indexes

With OpenSearch running on http://localhost:9200, create the required index mappings:

python3 -m storage.initialize_os

Seed Synthetic Data (Optional)

Populate the catalog with sample data for local testing:

python3 seed_synthetic.py

This loads datasets defined in:

backend/data/synthetic_datasets.json

Start the Backend

python3 main.py

The application performs index and schema health checks during startup.

---

⚛️ Step 3: Configure and Start the Frontend

Open a second terminal window and navigate to the frontend directory:

cd frontend

Install dependencies:

npm install

Start the Vite development server:

npm run dev

---

🔄 Step 4: Run the Ingestion Pipeline (Optional)

To ingest real data instead of synthetic datasets, run the ingestion driver from the backend/ directory:

python3 run_pipeline_ingest.py [LIMIT]

Where LIMIT is an optional argument to constrain the number of sources processed.

---

🧭 Service Endpoints

Once all services are running, the following endpoints are available:

Service	URL	Purpose
Frontend UI (Vite)	http://localhost:5173	Interactive web application
Backend API (FastAPI)	http://localhost:8000	REST API and OpenAPI documentation
OpenSearch Cluster	http://localhost:9200	Search and indexing backend
OpenSearch Dashboards	http://localhost:5601	Cluster administration and visualization
MinIO Console	http://localhost:9001	Object storage administration
MinIO API	http://localhost:9000	S3-compatible object storage endpoint
---

🚀 Quick Start Summarys

# Terminal 1 (repo root)
docker compose up -d
docker stop auctus-backend

# Terminal 2
cd backend
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt
python3 -m storage.initialize_os
python3 seed_synthetic.py
python3 main.py

# Terminal 3
cd frontend
npm install
npm run dev

🔄 Resetting the Environment (Optional)

When testing end-to-end ingestion pipelines, search indexing behavior, or storage workflows, you may want to start from a completely clean state.

⚠️ Warning

This operation permanently removes all persisted local development data, including:

• OpenSearch indexes
• MinIO objects
• Redis state

Use this reset only when you intentionally want a fresh Auctus v2 installation for local development or integration testing.

Complete Data Wipe

# Stop all containers and remove all associated volumes
docker compose down -v

# Recreate and start infrastructure from scratch
docker compose up -d

# Stop the placeholder backend container if running
docker stop auctus-backend

After the infrastructure has been recreated, rerun the backend initialization steps:

cd backend

source .venv/bin/activate

# Recreate OpenSearch indexes and mappings
python3 -m storage.initialize_os

# Optional: Seed synthetic test data
python3 seed_synthetic.py

# Start the backend
python3 main.py