すべてのチャレンジ
中級 0件解決済み 正答率0%

Byte Pair Encoding (BPE) Merges

Learn subword merges the way GPT tokenizers do: repeatedly merge the most frequent pair of adjacent symbols.

このチャレンジで学べること

  • Count pairs of adjacent symbols
  • Merge the most frequent pair repeatedly
  • Understand how GPT-style tokenizers are built

問題

翻訳を表示

GPT-style tokenizers use Byte Pair Encoding: start from single characters and repeatedly merge the most common pair of neighbors into a new symbol.

  1. Split every word in words into characters. Repeated words count multiple times.
  2. Count every pair of adjacent symbols across all words.
  3. Pick the most frequent pair. If pairs tie, pick the one that is smallest when you compare (first symbol, second symbol) as strings.
  4. Replace every occurrence of that pair, left to right, with the merged symbol, and record the merged symbol.
  5. Repeat num_merges times, or stop early when no word has two symbols left.

Implement bpe_merges(words, num_merges).

Input

words: The training words (lowercase letters).

num_merges: How many merges to learn.

Output

Return the merged symbols in the order they were learned.

制約

1 ≤ number of words ≤ 1,000, 1 ≤ length of each word ≤ 30

0 ≤ num_merges ≤ 200

例

例 1
入力:
words = ["low", "lower", "lowest"]
num_merges = 2

期待される出力:
expected = ["lo", "low"]
入力:
std::vector<std::string> words = {"low", "lower", "lowest"};
long long num_merges = 2;

期待される出力:
std::vector<std::string> expected = {"lo", "low"};
入力:
string[] words = new string[] {"low", "lower", "lowest"};
long num_merges = 2;

期待される出力:
string[] expected = new string[] {"lo", "low"};

解説

Pairs: "l o" and "o w" appear 3 times each; the tie goes to "l o" ("l" < "o"), giving "lo". Now "lo w" appears 3 times, giving "low". Result: ["lo", "low"].

例 2
入力:
words = ["aaa"]
num_merges = 3

期待される出力:
expected = ["aa", "aaa"]
入力:
std::vector<std::string> words = {"aaa"};
long long num_merges = 3;

期待される出力:
std::vector<std::string> expected = {"aa", "aaa"};
入力:
string[] words = new string[] {"aaa"};
long num_merges = 3;

期待される出力:
string[] expected = new string[] {"aa", "aaa"};

解説

"aaa" has the pair "a a" twice. Merging left to right gives "aa" + "a". Then "aa a" merges into "aaa", and the word is a single symbol, so the third merge has nothing to do. Result: ["aa", "aaa"].

解き方のヒント

まずは自分で考えてから、ヒントを 1 つずつ開いてみましょう。

ヒント 1

Track the current symbol sequence for each word and repeatedly count neighboring pairs across all repeated words.

ヒント 2

Use pair counting plus deterministic tie-breaking by comparing the pair as two strings, and then update every word left to right.

ヒント 3

Watch words that shrink to one symbol; when no adjacent pair exists anywhere, stop even if merges remain.

目安の計算量

O(num_merges * total_symbols) time, O(total_symbols) extra space

参考資料

この課題はCodeUtility用に調整されています。上記の説明と制約に従ってください。

ランキング

全体ランキング

コーダーごとのいちばん良い合格解答です。速く、メモリが少なく、評価が高い解答ほど上位になります。

まだ合格した解答はありません。解いて 1 位になりましょう。

他の人の解答から学ぶ

コミュニティの解答

まだ公開された解答がありません。この課題を解いて最初の解答を共有しましょう。