Generate-and-evaluate endpoint

This endpoint allows you to submit a request to generate images using Replicate models and evaluate them using selected evaluation models.

Endpoint

POST /generate-and-evaluate

Input JSON

{
  "api_key": "your_replicate_api_key",
  "title": "Evaluation Title",
  "rows": [
    {
      "prompt": "A prompt for image generation",
      "seed": 12345,
      "examples": [
        {
          "model": "owner/model_name[:version]",
          "prompt_input": "prompt_input_name",  # optional, defaults to "prompt"
          "seed_input": "seed_input_name",  # optional, defaults to "seed"
          "inputs": {
            "key": "value",
            ...
          }
        }
      ]
    }
  ],
  "eval_models": ["ImageReward", "Aesthetic", ...]
}
    

Example curl command

curl -X POST http://localhost:8000/generate-and-evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "api_key": "your_replicate_api_key",
    "title": "Test Evaluation",
    "rows": [
      {
        "prompt": "A beautiful sunset over the ocean",
        "seed": 42,
        "examples": [
          {
            "model": "stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf",
            "prompt_input": "prompt",
            "seed_input": "seed",
            "inputs": {
              "width": 512,
              "height": 512,
              "num_inference_steps": 50
            }
          }
        ]
      }
    ],
    "eval_models": ["ImageReward", "Aesthetic", "CLIP"]
  }'
    

Response:

Upon successful submission, you will receive a JSON response with the evaluation ID and a URL to view the results:

{
  "evaluation_id": "uuid-of-the-evaluation",
  "results_url": "http://localhost:8000/results/uuid-of-the-evaluation/"
}
    

Notes:

• Replace your_replicate_api_key with your actual Replicate API key.
• You can include multiple rows and examples per row in your request.
• The seed field is optional. If not provided, a random seed will be generated.
• Ensure that the model you specify is available on Replicate and that you have permission to use it.
• The eval_models array should contain one or more of the supported evaluation models.

Evaluate-images endpoint

This endpoint allows you to submit existing images for evaluation using selected evaluation models.

Endpoint

POST /api/evaluate-images

Input JSON

{
  "api_key": "your_replicate_api_key",
  "title": "Evaluation Title",
  "data": [
    {
      "prompt": "A prompt for image evaluation",
      "images": [
        {
          "url": "https://example.com/image1.jpg",
          "labels": {
            "key1": "value1",
            "key2": "value2"
          }
        },
        {
          "url": "https://example.com/image2.jpg",
          "labels": {
            "key1": "value1",
            "key2": "value2"
          }
        }
      ]
    }
  ],
  "models": ["ImageReward", "Aesthetic", ...]
}
    

Example curl command

curl -X POST http://localhost:8000/api/generate-and-evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "api_key": "YOUR_REPLICATE_API_KEY",
    "title": "Flux Model Comparison",
    "rows": [
      {
        "prompt": "the Great Pyramid",
        "examples": [
          {
            "model": "black-forest-labs/flux-schnell",
            "inputs": {
              "go_fast": false
            }
          },
          {
            "model": "black-forest-labs/flux-schnell",
            "inputs": {
              "go_fast": true
            }
          }
        ]
      },
      {
        "prompt": "robots meditating in a vipassana retreat",
        "examples": [
          {
            "model": "black-forest-labs/flux-schnell",
            "inputs": {
              "go_fast": false
            }
          },
          {
            "model": "black-forest-labs/flux-schnell",
            "inputs": {
              "go_fast": true
            }
          }
        ]
      }
    ],
    "eval_models": ["ImageReward", "Aesthetic", "CLIP", "BLIP", "PickScore", "DreamSim"]
  }'
    

Response:

Upon successful submission, you will receive a JSON response with the evaluation ID and a URL to view the results:

{
  "evaluation_id": "uuid-of-the-evaluation",
  "results_url": "http://localhost:8000/results/uuid-of-the-evaluation/"
}
    

Notes:

• Replace your_replicate_api_key with your actual Replicate API key.
• You can include multiple rows and images per row in your request.
• The prompt field is optional but must be consistent across all rows (either all present or all absent).
• The labels field for each image is optional.
• The eval_models array should contain one or more of the supported evaluation models.
• If using DreamSim, the first image in each row will be treated as the reference image.