Fine-Tuning TranslateGemma

This page documents a practical workflow for fine-tuning TranslateGemma on a new language pair without turning the model into a narrow translate-only system. It uses the Welsh pilot from grctest/finetuned-gemmatranslate-cy as the reference implementation, but the same method applies to other languages if you adapt the data layer carefully.

Reference implementation

The Welsh repository is the best starting point if you want a working end-to-end example rather than isolated snippets:

• GitHub: grctest/finetuned-gemmatranslate-cy.
• Data preparation: 01_prepare_data.py.
• Token analysis: 01b_analyze_token_lengths.py.
• LoRA fine-tuning: 02_finetune.py.
• Merge: 03_merge.py.
• Inference: 04_inference.py.

This guide does not just restate the repo layout. It explains the design choices that made the Welsh run usable: the translation-instruction balance, the token-length checks, the H200 profile choices, and the export path for local deployment.

Workflow overview

1. Choose the base TranslateGemma model and decide what behavior you want to preserve.
2. Prepare a compatible Python and CUDA environment.
3. Build a dataset recipe that keeps translation supervision and instruction retention separate.
4. Normalize, deduplicate, rebalance, and split the dataset.
5. Measure token lengths before committing to an expensive run.
6. Run a small LoRA pilot first.
7. Scale up only after the pilot looks healthy.
8. Merge the adapters and export a GGUF for local runtimes.
9. Evaluate against the base model before calling the fine-tune production-ready.

1. Pick the right training target

For the Welsh pilot, TranslateGemma-4B was the right starting point because it was small enough to validate quickly on one GPU while still matching the task framing used by the training scripts.

That framing matters:

• Translation rows are rendered through TranslateGemma’s chat template with explicit source and target language codes.
• Instruction rows are rendered as standard user-assistant turns.
• The trainer uses completion-only loss so the optimization pressure stays on the model’s answer rather than the prompt tokens.

The goal should not be “make the model translate more often.” The goal should be “improve translation quality while keeping the model useful as an instruction-following assistant.” If you fine-tune only on bilingual pairs, you can improve direct translation behavior and still make the model worse to use.

Start with 4B unless you have already proven that your data pipeline is sound. Move to 12B or 27B after the recipe is stable and the evaluation plan is clear.

2. Choose a hardware profile that matches the recipe

The reference pilot used a single Nvidia H200 with a profile tuned for packed 2048-token LoRA training.

Welsh reference profile

Setting	Value
per_device_train_batch_size	12
gradient_accumulation_steps	8
Effective batch size	96
max_seq_length	2048
bf16	True
gradient_checkpointing	True
packing	True
optimizer	adamw_torch_fused
dataset_fraction	0.05 for the pilot

The practical reasons to use H200 for this workflow were:

• Enough VRAM headroom for packed 2048-token sequences without collapsing the batch size.
• Strong bf16 performance for this style of fine-tuning.
• Access to a fast Hopper-era attention path, with the repo preferring Flash Attention 3 where available.

Reference timing

Run type	Hardware	Dataset fraction	Approximate time
Pilot validation	1x H200	5%	about 40 minutes
Full run estimate	1x H200	100%	about 33 hours
Scaled run estimate	8x H200	100%	about 4.5 hours

If you are still validating the recipe, one strong GPU is enough. If you need same-day turnaround on a full run, multi-GPU starts to matter. The key decision is not just cost. It is how quickly you want feedback after changing the recipe.

3. Set up the environment

The Welsh run used a CUDA 12.8-oriented environment with a local snapshot of the base model available under ./local_model.

python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install flash_attn_3 --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch2110
hf auth login

Notes:

• On Windows, replace source venv/bin/activate with the platform equivalent for your shell.
• The preparation stage expects ./local_model to contain a complete local TranslateGemma snapshot, including tokenizer files, processor config, and model weights.
• The important packages are not incidental: transformers, torch, datasets, peft, and trl each map to a distinct part of the workflow.

4. Design the dataset before you touch the GPU

The Welsh run is most useful as a data-engineering example. The main target was a deliberate 70:30 translation-to-instruction mix, not maximum row count.

Keep a flat dataset contract

Normalize every source into the same structure before training:

• Field task.
• Field source_text.
• Field target_text.
• Field source_lang_code.
• Field target_lang_code.

That gives you one place to deduplicate, one place to rebalance, and one prompt-building contract for the trainer.

Welsh reference sources

High-value translation and terminology sources:

• Dataset techiaith/legislation-gov-uk_en-cy.
• Dataset techiaith/cofnodycynulliad_en-cy.
• Dataset techiaith/bydtermcymru-tm-en-cy.
• Dataset AndreasThinks/welsh-government-pairs.
• Dataset TermCymru/TermCymru.

Instruction-retention sources:

• Dataset akoksal/muri-it-language-split slices for English, Welsh, and other languages.
• Dataset locailabs/nemotron-chat-welsh.

The important design choice is that not every row needs to be direct translation supervision. The multilingual Muri slices were used to preserve instruction-following behavior and turn structure while the model was being pushed harder toward translation.

Use high-volume corpora conservatively

Two decisions from the Welsh run are worth copying unless you have strong evidence to do otherwise:

• The Cardiff University translation memory was capped at usage=0.1.
• Dataset Helsinki-NLP/opus-100 was left at usage=0.0.

That restraint kept the recipe from being dominated by translation-heavy corpora that would have forced a much larger instruction set just to maintain the same behavior target.

Let domain resources do more than one job

TermCymru was especially valuable because it supported both:

• Direct translation pairs in both directions.
• Synthetic instruction-response pairs built from English and Welsh definitions.

That is a strong pattern to reuse in other languages. If you have a glossary or terminology bank with definitions, do not treat it as translation-only data.

5. Run cleanup as a first-class stage

Data cleanup was not optional in the Welsh workflow. After the sources were normalized, the pipeline:

• Filtered empty or null pairs.
• Deduplicated globally across text, task, and language-code fields.
• Rebalanced toward the target 70:30 translation-instruction mix.
• Balanced translation directions to keep en->cy and cy->en near parity.
• Held out 1000 evaluation rows stratified by task and direction.

Welsh reference output after cleanup

Metric	Value
Raw constructed rows	1,532,911
Duplicates removed	174,735
Rows removed by rebalancing	3,408
Final rows after cleanup	1,354,653
Final translation rows	948,257
Final instruction rows	406,396
Final direction counts	474,108 cy->en, 474,149 en->cy

Those duplicate removals are large enough to matter. Repeated institutional bilingual pairs can distort the distribution very quickly and make the model memorize a narrow phrasing style instead of learning robust translation behavior.

6. Measure token lengths before full training

The Welsh project used 01b_analyze_token_lengths.py to justify max_seq_length=2048 before spending real money on a full run.

python 01b_analyze_token_lengths.py --num-proc 6 --dataset-fraction 0.05 | tee 01b_token_analysis_0.05.txt
python 01b_analyze_token_lengths.py --num-proc 6 --dataset-fraction 1.0 | tee 01b_token_analysis_full.txt

Why 2048 worked in the reference run

Scan	Segment	Mean	95th percentile	99th percentile	Max
5%	Global	308.30	1580	2049	3239
5%	Translation	206.55	not reported	1983	3239
5%	Instruction	546.66	2049	2049	2918
100%	Global	306.98	1587	2049	3294
100%	Translation	206.23	not reported	1983	3294
100%	Instruction	542.08	2049	2049	3164

The main lesson is simple: do not inherit sequence-length settings from another project just because the model family is the same. Measure your own processed data first.

In the Welsh run, the 5% scan was close enough to the full scan to justify using a small exploratory pass before a full analysis. That is a good default pattern when you are iterating on the recipe.

7. Run a pilot LoRA fine-tune

The Welsh pilot used 02_finetune.py with one epoch over a 5% slice of the training set.

python 02_finetune.py --profile H200F --num-proc 20 --sft-num-proc 30 | tee 02_finetune_log.txt

Core training choices from the reference run

• LoRA rank 16.
• LoRA alpha 32.
• LoRA dropout 0.05.
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
• Embeddings frozen.
• Setting completion_only_loss=True.
• Sequence packing enabled.

What a healthy pilot looked like

The logged pilot showed a useful pattern:

• Loss moved from 2.288 down to 1.718.
• Mean token accuracy moved from 0.57 to 0.6413.
• Final aggregate training loss ended at 1.864.

That does not prove production quality, but it does show that the prompt formatting, dataset mix, and training profile were behaving sensibly. A pilot should answer “is this recipe worth scaling up?” before you ask “how do I run it on more GPUs?“

8. Merge the adapters and export a GGUF

Once the pilot or full run is finished, merge the LoRA adapters back into the base model and create a GGUF artifact for local runtimes.

python 03_merge.py --profile cpu
python llama.cpp/convert_hf_to_gguf.py ./final_merged_model --outfile 4B_cy_q8_0.gguf --outtype q8_0

This produces:

• A merged Hugging Face-style model artifact.
• A GGUF build that is easier to use in local inference tools and desktop workflows.

If your goal is practical deployment rather than a one-off experiment, the GGUF export is not optional. It is the bridge between a successful training run and an actually usable model.

9. Test the merged model and evaluate it properly

The reference repo includes a simple inference path:

python 04_inference.py --profile cpu

Use that for smoke testing, but do not stop there. Before shipping a fine-tuned model, compare it against the base model with real translation metrics such as:

• MetricX.
• BLEU.
• COMET.

Eyeballing a few good outputs is not enough.

Adapting this workflow to another language

The reusable parts of the method are the easiest parts:

• The LoRA training pattern.
• The token-length analysis step.
• The merge and GGUF export path.
• The practice of running a pilot before a full run.

The part you will usually need to change first is the data adapter layer in 01_prepare_data.py.

Expect to modify:

• Dataset-loading logic.
• Field mapping and normalization.
• Language-code handling.
• Glossary and terminology enrichment.
• Any synthetic-example generation tied to Welsh-specific resources.

If you are moving to another underrepresented language, keep these rules:

• Preserve a deliberate translation-instruction target instead of blindly maximizing row count.
• Rerun token analysis every time the recipe changes materially.
• Balance translation directions where the product needs both directions.
• Prefer domain-specific corpora and terminology banks over generic bulk data when the two conflict.

Common failure modes

• Flooding the recipe with translation memory data and then wondering why instruction-following quality collapsed.
• Skipping global deduplication and overweighting repeated institutional phrases.
• Reusing the Welsh token-length assumptions without measuring the new dataset.
• Treating GPU rental as the main problem when the real problem is usually weak data curation.
• Declaring success before the fine-tuned model is compared against the base model on a real evaluation set.

Summary

If you want a reliable TranslateGemma fine-tune, spend more effort on the data recipe than on the launch command. The Welsh reference run worked because it combined curated bilingual corpora, instruction-retention data, aggressive cleanup, a measured 2048 token ceiling, and a small pilot run before scaling. Start there, then adapt the data layer to your language instead of copying the surface details blindly.