Tokenizer Transfer¶

Tokenizer transfer: learn a tokenizer on one dataset and then apply it to a second dataset¶

In this example, we'll train a tokenizer on mimic and apply it to ucmc.

Set the paths to your raw datasets:

raw_mimic="/path/to/data-raw/mimic-2.1.0"
raw_ucmc="/path/to/data-raw/ucmc-2.1.0"

Run collation on each dataset separately with the same config:

cocoa collate \
    --raw-data-home ${raw_mimic} \
    --processed-data-home ./processed/mimic \
    --verbose

cocoa collate \
    --raw-data-home ${raw_ucmc} \
    --processed-data-home ./processed/ucmc \
    --verbose

Learn a tokenizer on the first dataset:

cocoa tokenize \
  --processed-data-home ./processed/mimic

Supply the --tokenizer-home argument to the tokenize command to load the previously learned tokenizer (with fixed vocabulary and binning cutpoints):
```
cocoa tokenize \
  --tokenizer-home ./processed/mimic/tokenizer.yaml \
  --processed-data-home ./processed/ucmc \
  --verbose
```

Proceed as usual:

cocoa winnow \
  --processed-data-home ./processed/mimic \
  --verbose

cocoa winnow \
  --processed-data-home ./processed/ucmc