llmops

A simple llmops

• Components

  • Chat front end
  • Guardrails middleware
  • Ollama LLM backend
  • Custom Arize Logging

• Demo

  d2.mp4

• Flow

  • Guardrails Disabled

    

  • Guardrails Enabled

    

Prerequisites

• Install Docker from here
• Get Guardrails token:
  • Visit Guardrails AI Hub
  • Sign up or log in to your account
  • Navigate to your account settings or API keys section
  • Copy your token and save it somewhere.

Running the Code

1. Run below commands:

   git clone https://github.com/abhishek9sharma/llmops.git
   cd llmops

2. Create a .env file in the root directory. Your structure should look like:

   llmops/
   ├── .env           ← Create this file here
   ├── docker-compose.yml
   ├── Makefile
   ├── grserver/
   ├── ollama-backend/
   ├── streamlit-chat/
   └── README.md
   

   touch .env

3. Add your Guardrails token to the .env file:

   GR_TOKEN=your_guardrails_token_here

4. Optional: Add other environment variables:

   GR_TOKEN=your_guardrails_token_here
   OPENAI_API_KEY=your_openai_api_key
   ANOTHER_VARIABLE=value

5. Run the application:

   make up_with_build

   This will take a while do build.

6. Navigate to http://localhost:8501/