FastAPI Service

The FastAPI project behind MetalGlot lives here: grctest/fastapi-gemma-translate.

It is the local API layer that loads TranslateGemma GGUF models, exposes translation endpoints, and provides a simple deployment surface for both direct Python runs and Docker-based installs.

What the project does

The service is a REST API for running Google’s TranslateGemma models locally.

From the repository README, the main goals are:

• Translation services for local workloads.
• Support for multiple model variants.
• Automatic interactive API docs through Swagger UI and ReDoc.
• Docker-friendly deployment for both CPU and Nvidia GPU environments.

How it fits into MetalGlot

MetalGlot is the product experience. The FastAPI project is the inference and serving layer.

That separation gives MetalGlot a few advantages:

• The translation engine can run fully on-device.
• Model loading is explicit instead of hidden inside the UI.
• Docker images can package repeatable CPU and CUDA deployments.
• The API can be tested directly outside the desktop app.

Key implementation areas

The repository currently includes:

• Directory app/ for the Python application code.
• File Dockerfile for the CPU build.
• Files LegacyCudaDockerfile, MainstreamCudaDockerfile, and FutureCudaDockerfile for different Nvidia generations.
• File requirements.txt for Python dependencies.

API surface

The README documents a model lifecycle and multiple translation endpoints.

Model lifecycle

Model loading is explicit. Translation requests are rejected unless the requested model is already loaded.

Important endpoints include:

• Endpoint POST /model/load.
• Endpoint GET /model/status.

Example load request:

curl -X POST "http://127.0.0.1:8080/model/load" \
  -H "Content-Type: application/json" \
  -d '{"model":"translategemma-4b-it-Q8_0"}'

For vision-enabled translation, the service can also load a matching mmproj file.

Translation endpoints

• Endpoint POST /translate for stable locale lists.
• Endpoint POST /experimental_translation for stable plus experimental locale lists.
• Endpoint POST /translate_image for image translation when the model has been loaded with vision support.

Built-in interactive docs

When the service is running locally, the repository exposes:

• Swagger UI at http://127.0.0.1:8080/docs.
• ReDoc at http://127.0.0.1:8080/redoc.

Concurrency and stability

The project includes runtime settings for controlling inference concurrency, especially for CUDA setups using llama-cpp-python.

The documented safe default is:

LLAMA_MAX_CONCURRENT_INFERENCES=1
LLAMA_INFERENCE_ACQUIRE_TIMEOUT_SECONDS=45

That keeps heavier 12B and 27B workloads from piling up unstable parallel requests on GPU machines.

Local development and Docker

The service can be started directly with Uvicorn for development, or run through prebuilt Docker images for easier deployment.

See the Docker images page for the container variants exposed for CPU and CUDA environments.