System Prompt, Skill, RAG
Splitting Languages Inside the Machine
When you build an LLM app in Japanese, the three layers each call for a different language. Instructions in English, output in Japanese, source material left in the original. A record of why I don't force all three into one, measured with tiktoken and multilingual embeddings.
I got curious, so I measured it.
You hear that "Japanese is heavy for LLMs." I wanted to see, with numbers in my own hands, exactly how heavy — and where that weight actually bites. I ran Tatoeba's English↔Japanese sentence pairs through tiktoken, plus two multilingual embedding models, and the right way to operate fell out of the results — let me put the conclusion first.
From here on, it's the order in which I verify each of those three layers with numbers, one at a time. The conclusion is already in Table 1, so a reader in a hurry can read just that and close the tab.
The conclusion: write the three layers in different languages
A large language model reads a sentence only after chopping it into grains called tokens. Japanese and English get chopped differently, and that changes both cost and retrieval quality. With that in mind, when you embed an LLM in a Japanese app, the language you touch isn't one — there are at least three. Each follows its own rule.
| Layer | Language | Why |
|---|---|---|
| Output (the face the user sees) | Japanese | Obviously |
| Instructions (system prompt / skill body) | English | ≈ 1.6× token savings, plus steadier instruction-following |
| Material (RAG documents, sources) | Keep in Japanese | Translation loses information. Same-language matching wins even with multilingual embeddings |
The rest of the piece is the order in which I verify, with numbers, why each of these three layers comes out the way it does. §02–§03 are how heavy Japanese is in tokens (why the instructions go English). §04 is the fine print of operating instructions and output. §05 is that RAG wins on same-language matching (why you leave the material in the original). §06 is exceptions and pitfalls.
For the same meaning, Japanese is about 2× heavier
There are two grounds for writing instructions in English: one is token economy, the other is steadiness of instruction-following. Let me start with the first. I'll open with a concrete example, then move straight to the measurements.
Individual words are one thing. What I want is the average when natural sentences flow through. I ran 2.4 × 105 pairs (n = 237,713) from Tatoeba's translations. For each pair: Japanese token count ÷ English token count. Binned into a 0.0–4.0 histogram, that's Fig.2.
The numbers sit still. The same holds for the total token ratio across the whole corpus: cl100k_base 2.06×, o200k_base 1.57×. Even after averaging out the per-sample scatter, the outer edges land in almost the same place. This isn't the accident of any one sentence — it's an asymmetry burned into the vocabulary itself.
The more kanji, the lighter — and it shows up in the bill
The median came out clean, but the spread is wide too. From a ratio of 0.5 to 4, an order-of-magnitude difference lives inside a single corpus. The key is the kanji ratio — kanji are dense in meaning, and in o200k_base many of them are registered as a single token. Hiragana and katakana, by contrast, get chopped into long subwords.
This weight shows up in the bill. OpenAI's latest input price is $5.00 / 1M tokens for GPT-4o (at the time of writing). To carry the same meaning, Japanese needs 1.57× as many tokens, so a job that costs $5.00 in English costs $7.85 in Japanese. +57%. On the GPT-4-generation cl100k_base it was +106%.
| Language / generation | Tokens needed | Cost ($5/Mt) | vs. English |
|---|---|---|---|
| English | 1.00 M | $5.00 | 1.00× |
| Japanese (cl100k_base) | 2.06 M | $10.30 | 2.06× |
| Japanese (o200k_base) | 1.57 M | $7.85 | 1.57× |
It's a kind of hidden language tax. When a Japanese speaker uses a model, they pay 1.5 to 2 times what an English speaker pays to convey the same content. The same coefficient works against you in burning through the context window and in the felt speed of inference. The skew in the vocabulary passes straight through to a skew in the bill.
From here on it isn't observation but the lines drawn from it. For each layer of the three-layer model (Table 1), one level of detail more.
Instructions in English, output in Japanese
The system prompt and the body of a skill can go boldly in English. Add one line at the end — "Respond in Japanese." — and the output comes back in Japanese. That alone lays the 1.57× saving right on top.
And it isn't only about savings. There's also a felt sense that complex branching, tool selection, and chains of reasoning are steadier in English — that's a rule of thumb separate from the numbers in this piece, but it probably comes from the same skew (the English weighting of the training corpus). Instruction-following ability and token efficiency line up in the same direction.
Two things should stay in Japanese, as exceptions.
- Domain-specific vocabulary (statute names, the nuance of honorifics, in-house proper nouns) breaks in meaning when forced into English. Drop just those words in as Japanese.
- Few-shot examples should be written in the language you want as output. The model learns the output format from them, so writing them in English pulls the output toward English.
RAG is the reverse — same-language matching wins even with multilingual embeddings
Instructions go English, so why is the RAG material left in the original? At first this struck even me as backwards. Wouldn't translating the search keys into English make them short and light?
Multilingual embeddings — like bge-m3 or paraphrase-multilingual-mpnet — map the same meaning to nearby vectors even across different languages. A Japanese query and an English summary can still match on meaning. Cross-lingual retrieval works.
But "works" and "best" turned out to be different. I wasn't convinced until I measured it myself, so I did.
Tatoeba contains multiple Japanese translations of the same English sentence. I collect 3.3 × 104 pairs where an ei has translations ji,1 and ji,2. For a query ji,1, I retrieve (a) another Japanese translation ji,2 from the Japanese pool (same language), or (b) the translation ei from the English pool (cross language). I measure Recall@1 while varying the pool size N.
Now that the trend is clear, let me check a SOTA model too. I ran the same condition (N=10,000) on BAAI/bge-m3 (the current SOTA in multilingual embeddings).
| Model | same R@1 | cross R@1 | gap | pairwise * |
|---|---|---|---|---|
| paraphrase-multilingual-mpnet | 82.86 % | 79.63 % | +3.23 pp | 76.68 % |
| bge-m3 (SOTA) | 83.93 % | 81.33 % | +2.60 pp | 83.68 % |
cos(query, ja_target) exceeds cos(query, en_target)Three conclusions follow.
- The Recall@1 gap is small, 2–3 pp, but it widens as the pool grows. On a 500-item corpus the difference is nearly nil; at 30,000 it grows to a 6 pp gap. A production RAG is far larger.
- The pairwise gap is large, 77–84%. "Same-language has the higher cosine score" in the vast majority of cases. This is invisible in Recall@1, but it bites in threshold decisions and in blending with other signals.
- It doesn't vanish at SOTA. Multilingual embeddings make cross-lingual retrieval work; they don't make it best.
So the right call for RAG material is to leave it in the original — both the search keys and the sources. If you're handling Japanese documents, write the summaries you build for retrieval in Japanese too. This is the basis for the "material stays Japanese" line of the three-layer model (Table 1).
Over-trusting multilingual embeddings — "vectors are language-independent" — and normalizing every search key to English. Run the eval yourself and you lose 3–6 pp on Recall@1 and 20–33 pp on pairwise. The moment you read "works" as "best," retrieval quality drops.
Exceptions and pitfalls: Skill descriptions and katakana words
The three-layer model has two fine-grained exceptions. Those go on the record too.
A skill's description goes in the user's language
The eval in §05 shows that "query language and material language should match." The same logic applies to a skill's description — a skill's trigger decision is made on the semantic similarity between the user's query and the description. If the description is in English and the user writes their question in Japanese, the match is naturally weaker.
Description in the user's language, body in English — that asymmetric hybrid is the practical answer. This is the one place in the implementation where you cross layers.
Katakana words can get worse across a generation
From cl100k_base to o200k_base the whole thing shifted left, but not everything improved. Some examples actually got worse. Those go on the record too.
| Japanese | cl100k | o200k | Δ |
|---|---|---|---|
| 彼女はダイエットをしている。 | 2.60 | 3.67 | +1.07 |
| 私ね、ビルマ語を習ってるのよ。 | 2.57 | 3.50 | +0.93 |
| コアラはユーカリの葉を食べるんだよ。 | 1.92 | 2.83 | +0.91 |
| 今ダイエット中なの。 | 1.80 | 2.67 | +0.87 |
At first I started to write that "as generations advance, the Japanese ratio falls uniformly." The median did. The corpus ratio did. But look at individual sentences and there really were cases that got worse, like the ones above. Not hiding the one loss after fourteen straight wins seems better than bragging about the fourteen. Standing on the side of the observational record means exactly this.
I've slipped in two fine caveats, but the broad shape of the three-layer model doesn't change. Output in Japanese, instructions in English, material in the original — operate with the exceptions in mind, and cost and quality stop being a trade-off.
What I don't know
The numbers came out, but there's more still unanswered than answered.
- Is instruction-following really steadier in English? I wrote "felt sense," but run IFEval in both languages and it becomes a number. It's worth checking whether quality rises too, not just whether cost falls. That's a separate measurement.
- Where's the floor? As the kanji ratio approaches 1.0, does the ratio drop below 1, stop at 1, or asymptote to some positive value? In the under-sampled region (kanji ratio > 0.8) it fell to 1.17×. With a large enough vocabulary, Japanese shorter than the same-meaning English should be possible in principle.
- Does the RAG gap change by domain? Here I used Tatoeba's everyday sentences, but in specialized domains — law, medicine, code — the advantage of same-language matching might widen further. Reproducing it on a real corpus is homework.
- What about other non-English languages? Hangul, the Arabic script, Devanagari — how heavy are they, how far do they tilt in the RAG direction? I don't have the topographic map yet.
From this material, this is as far as I can go. Beyond here, it takes a different measurement.