F0ckreddit

avatar
codepasta
markdown
2 months ago
14 kB
38
No Index

Okay so I was replying from the iphone and couldn't read your codeblocks well, now on the computer I notice a few things that are probably wrong. But don't worry: litellm's documentation is poorly organized and some of the information there is outdated and/or misleading. For example, the docker-compose.yaml file still contains errors. In my opinion, these lines should be removed:

      - ./.env:/app/.env
    env_file:
      - ./.env

Docker automatically reads from the .env file, if it exists, and passes on the values. Therefore, env_file: is only necessary if the file would have a different name than .env. And since the ENVs are passed automatically, as already mentioned, the file does not need to be bound, which is why we do not need - ./.env:/app/.env.

Under environment: you should also do some things differently:

The line

CONFIG: "/app/config.yaml"

does not belong there, but should be passed as a command:

# ...
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    command:
      - "--config=/app/config.yaml"
    environment:
# ...

My tip is also to remove all environment variables from the docker-compose.yaml file that are particularly ... variable :D (but also particularly sensitive)

Then firstly you won't have a mess, but everything that belongs together in one place and secondly you can make the docker-compose.yaml more portable and your ENVs more secure.

At the moment you can see the password of your database, for example: postgresql://postgres:--> postgres <-- YOUR PASSWORD You should therefore change it immediately after it has been seen publicly here. It's easy to overlook something like that. That's why it's better to put it in .env.

You can also delete these lines completely for the time being:

      EXPOSE_MODELS: "true"
      ALLOW_MODEL_LIST_UPDATES: "true"
      LOAD_FROM_CONFIG: "true"
      MODEL_LIST_FROM_DB: "true"
      DEBUG: "true"

If it should help you, here you can see my complete docker-compose.yaml as I have structured it:

volumes:
  litellm-db:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: litellm-app
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    command:
      - "--config=/app/config.yaml"
    ports:
      - "${LITELLM_APP_HOST_IP}:${LITELLM_APP_HOST_PORT}:4000"
    environment:
      STORE_MODEL_IN_DB: "True"
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
      DATABASE_URL: ${DATABASE_URL}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      CODESTRAL_API_KEY: ${CODESTRAL_API_KEY}
      COHERE_API_KEY: ${COHERE_API_KEY}
      DEEPSEEK_API_KEY: ${DEEPSEEK_API_KEY}
      FIREWORKS_API_KEY: ${FIREWORKS_API_KEY}
      GEMINI_API_KEY: ${GEMINI_API_KEY}
      GITHUB_API_KEY: ${GITHUB_API_KEY}
      GROQ_API_KEY: ${GROQ_API_KEY}
      HUGGINGFACE_API_KEY: ${HUGGINGFACE_API_KEY}
      LLAMA_CPP_API_KEY: ${LLAMA_CPP_API_KEY}
      MISTRAL_API_KEY: ${MISTRAL_API_KEY}
      OLLAMA_API_KEY: ${OLLAMA_API_KEY}
      OPENROUTER_API_KEY: ${OPENROUTER_API_KEY}
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      REPLICATE_API_KEY: ${REPLICATE_API_KEY}
    restart: always

 
  #db:
  postgres:
    image: postgres
    container_name: litellm-postgres
    volumes:
      - litellm-db:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: llmproxy
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    ports:
      - "${LITELLM_DB_HOST_IP}:${LITELLM_DB_HOST_PORT}:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
      interval: 2s
      timeout: 3s
      retries: 5
    restart: always

Do you see what I mean? In this way you have made this file portable and you can simply copy the entire content and pass it on without having to worry or make anything sensitive unrecognizable beforehand.

You can therefore copy my code 1 to 1 and then add your own configuration:

# ...
depends_on:
  redis:
    condition: service_healthy
  postgres:
    condition: service_healthy
restart: unless-stopped
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:4000/health"]
  interval: 30s
  timeout: 10s
  retries: 3
deploy:
  resources:
    limits:
      cpus: "0.75"
      memory: "8G"
networks:
  - ai-network
# ...

The same actually also applies to the litellm-config.json file. I don't think you should specify too many configs here, but only the most necessary and you can add the rest in the UI. Necessary would be e.g. cost per token if it is missing somewhere, claude-3.7 still needs drop_params, some models have no system prompt etc. Otherwise, no sensitive data should be entered here either. I also noticed that you write api_key: env/OPENAI_API_KEY, but it should actually be api_key: os.environ/OPENAI_API_KEY. But you can also just look at my config here and copy what you need:

model_list:

  - model_name: claude-3.5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      supports_system_message: True
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: claude-3.7-sonnet
    litellm_params:
      model: anthropic/claude-3-7-sonnet-20250219
      supports_system_message: True
      api_key: os.environ/ANTHROPIC_API_KEY
      drop_params: true

  - model_name: claude-3.5-haiku
    litellm_params:
      model: anthropic/claude-3-5-haiku-20241022
      supports_system_message: True
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: claude-3-opus
    litellm_params:
      model: anthropic/claude-3-opus-20240229
      supports_system_message: True
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: codestral-chat
    litellm_params:
      model: codestral/codestral-latest
      supports_system_message: False
      api_key: os.environ/MISTRAL_API_KEY
      input_cost_per_token: 0.00000001
      output_cost_per_token: 0.00000001

  - model_name: codestral-code
    litellm_params:
      model: text-completion-codestral/codestral-2405
      supports_system_message: False
      api_key: os.environ/MISTRAL_API_KEY
      input_cost_per_token: 0.00000001
      output_cost_per_token: 0.00000001

  - model_name: command-r
    litellm_params:
      model: cohere/command-r
      supports_system_message: True
      api_key: os.environ/COHERE_API_KEY

  - model_name: command-r-plus
    litellm_params:
      model: cohere/command-r-plus
      supports_system_message: True
      api_key: os.environ/COHERE_API_KEY

  - model_name: deepseek-chat
    litellm_params:
      model: deepseek/deepseek-chat
      supports_system_message: True
      api_key: os.environ/DEEPSEEK_API_KEY

  - model_name: deepseek-reasoner
    litellm_params:
      model: deepseek/deepseek-reasoner
      supports_system_message: True
      api_key: os.environ/DEEPSEEK_API_KEY

  - model_name: fireworks-deepseek-r1
    litellm_params:
      model: fireworks_ai/accounts/fireworks/models/deepseek-r1
      supports_system_message: True
      api_key: os.environ/FIREWORKS_API_KEY
      input_cost_per_token: 0.000003
      output_cost_per_token: 0.000008

  - model_name: fireworks-deepseek-v3
    litellm_params:
      model: fireworks_ai/accounts/fireworks/models/deepseek-v3
      supports_system_message: True
      api_key: os.environ/FIREWORKS_API_KEY

  - model_name: fireworks-llama-70b
    litellm_params:
      model: fireworks_ai/accounts/fireworks/models/llama-v3p3-70b-instruct
      supports_system_message: True
      api_key: os.environ/FIREWORKS_API_KEY
      input_cost_per_token: 0.0000009
      output_cost_per_token: 0.0000009

  - model_name: gemini-1.5-flash-8b
    litellm_params:
      model: gemini/gemini-1.5-flash-8b
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY

  - model_name: gemini-1.5-pro
    litellm_params:
      model: gemini/gemini-1.5-pro
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY

  - model_name: gemini-2.0-flash
    litellm_params:
      model: gemini/gemini-2.0-flash
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY

  - model_name: gemini-2.0-flash-001
    litellm_params:
      model: gemini/gemini-2.0-flash-001
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY

  - model_name: gemini-2.0-flash-lite
    litellm_params:
      model: gemini/gemini-2.0-flash-lite-preview-02-05
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY

  - model_name: gemini-2.0-pro
    litellm_params:
      model: gemini/gemini-2.0-pro-exp-02-05
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY
      input_cost_per_token: 0.00000001
      output_cost_per_token: 0.00000001

  - model_name: gemini-2.0-thinking
    litellm_params:
      model: gemini/gemini-2.0-flash-thinking-exp
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY
      input_cost_per_token: 0.00000001
      output_cost_per_token: 0.00000001

  - model_name: github-llama-70b
    litellm_params:
      model: github/Llama-3.3-70B-Instruct
      supports_system_message: True
      api_key: os.environ/GEMINI_API_KEY
      input_cost_per_token: 0.00000001
      output_cost_per_token: 0.00000001

  - model_name: groq-llama-70b
    litellm_params:
      model: groq/llama3-70b-8192
      supports_system_message: True
      api_key: os.environ/GROQ_API_KEY

  - model_name: llama-cpp
    litellm_params:
      model: openai/custom-model
      supports_system_message: False
      api_key: os.environ/LLAMA_CPP_API_KEY
      api_base: http://localhost:8000
      input_cost_per_token: 0.00000001
      output_cost_per_token: 0.00000001

  - model_name: mistral-small
    litellm_params:
      model: mistral/mistral-small-latest
      supports_system_message: True
      api_key: os.environ/MISTRAL_API_KEY
      input_cost_per_token: 0.0000001
      output_cost_per_token: 0.0000003

  - model_name: mistral-large
    litellm_params:
      model: mistral/mistral-large-latest
      supports_system_message: True
      api_key: os.environ/MISTRAL_API_KEY

  - model_name: ollama
    litellm_params:
      model: ollama/custom-model
      supports_system_message: False
      api_key: os.environ/OLLAMA_API_KEY
      api_base: https://ollama.mountai.co
      input_cost_per_token: 0.00000001
      output_cost_per_token: 0.00000001

  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      supports_system_message: True
      api_key: os.environ/OPENAI_API_KEY

  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      supports_system_message: True
      api_key: os.environ/OPENAI_API_KEY

  - model_name: o3-mini
    litellm_params:
      model: openai/o3-mini
      supports_system_message: False
      api_key: os.environ/OPENAI_API_KEY

  - model_name: text-embedding-3-large
    litellm_params:
      model: openai/text-embedding-3-large
      supports_system_message: False
      api_key: os.environ/OPENAI_API_KEY

  - model_name: text-embedding-3-small
    litellm_params:
      model: openai/text-embedding-3-small
      supports_system_message: False
      api_key: os.environ/OPENAI_API_KEY

  - model_name: openai-moderation
    litellm_params:
      model: openai/text-moderation-latest
      supports_system_message: False
      api_key: os.environ/OPENAI_API_KEY

  - model_name: openai-whisper
    litellm_params:
      model: openai/whisper-1
      supports_system_message: True
      api_key: os.environ/OPENAI_API_KEY

  - model_name: openai-tts
    litellm_params:
      model: openai/tts-1
      supports_system_message: True
      api_key: os.environ/OPENAI_API_KEY

Now comes the .env with the sensitive data. Note that I have introduced additional variables that were also previously included in my docker-compose.yaml (e.g. LITELLM_APP_HOST_IP etc.).

If you are not already doing this, I would strongly recommend using a reverse proxy and not exposing your containers to 0.0.0.0, but only to the internal real local IP (which starts with 192.168... or 172.1... or 10.0...). In my .env I write the Postgres password and the database URL directly below each other, so that I can transfer the password directly from one line to the next and make no mistakes:

# Development Configs
LITELLM_MASTER_KEY=sk-abcABC123123xyzXYZ987987blaBLA

LITELLM_SALT_KEY=sk-abcABC123123xyzXYZ987987blaBLA

POSTGRES_PASSWORD=0000-YOUR-POSTGRES-PASSWORD-0000
DATABASE_URL=postgresql://llmproxy:0000-YOUR-POSTGRES-PASSWORD-0000@postgres:5432/litellm

LITELLM_APP_HOST_IP=172.16.0.4
LITELLM_APP_HOST_PORT=53136

LITELLM_DB_HOST_IP=172.16.0.4
LITELLM_DB_HOST_PORT=53137

# Anthropic
ANTHROPIC_API_KEY=sk-ant-api00-abcABC123123xyzXYZ987987blaBLA

# Codestral
CODESTRAL_API_KEY=abcABC123123xyzXYZ987987blaBLA

# Cohere
COHERE_API_KEY=abcABC123123xyzXYZ987987blaBLA

# Deepseek
DEEPSEEK_API_KEY=sk-abcABC123123xyzXYZ987987blaBLA

# Fireworks
FIREWORKS_API_KEY=fw_abcABC123123xyzXYZ987987blaBLA

# Gemini
GEMINI_API_KEY=AIzaSyA-abcABC123123xyzXYZ987987blaBLA

# Github
GITHUB_API_KEY=github_pat_abcABC123123xyzXYZ987987blaBLA

# Groq
GROQ_API_KEY=gsk_abcABC123123xyzXYZ987987blaBLA

# HuggingFace
HUGGINGFACE_API_KEY=hf_abcABC123123xyzXYZ987987blaBLA

# Llama-Cpp
LLAMA_CPP_API_KEY=sk-abcABC123123xyzXYZ987987blaBLA

# Mistral
MISTRAL_API_KEY=abcABC123123xyzXYZ987987blaBLA

# Ollama
OLLAMA_API_KEY=sk-abcABC123123xyzXYZ987987blaBLA

# OpenAI
OPENAI_API_KEY=sk-proj-abcABC123123xyzXYZ987987blaBLA

# Openrouter
OPENROUTER_API_KEY=sk-or-v1-abcABC123123xyzXYZ987987blaBLA

# Replicate
REPLICATE_API_KEY=abcABC123123xyzXYZ987987blaBLA

So, I hope that helps you.

As for the logs, you are right with 11 containers it gets confusing, but you can also simply enter docker logs litellm, just as you named your container, and then directly with docker instead of docker compose.

Oh one last tip, so that this separation between docker-compose.yaml, config file and .env file really makes sense: Make the .env file more secure and change the permissions of it to sudo chmod 440 .env and the owner to sudo chown $USER:docker .env

And yeah, good luck :)