= Tests Generated from Unicode Files
:toc:

== Instructions
The `UNIDATA` folder referred to below can be obtained by running
`tools/UCD-download.raku` from MoarVM's repository. The documentation below uses
`$UNIDATA` in paths as a stand-in for wherever the directory actually is, which
you need to substitute in yourself. Angle bracket constructs are used to
indicate "metasyntactic" pathname portions, e.g. `foo-<bar>.txt` means that
`<bar>` is replaced with some other text, and depending on context there may be
multiple files that match the `foo-<bar>.txt` pattern.

CAUTION: Be sure to use a version of Rakudo that's been built to use the updated
MoarVM. Some of the scripts below may produce incorrect results if the copy of
Rakudo being used has outdated Unicode information.

.Please list these data points if possible:
* Script path (should be the heading)
* Steps: list of steps to run for each Unicode update. Sometimes there is just one,
  other times there are multiple things to do. Please list the command and command's
  arguments (including the data file)
* Version: The Unicode version that was used for generating the tests
* Date: The date the file was regenerated (YYYY.MM.DD)
* Generated File(s): a list of files which this script generates

== Todo

Nice things to add: ensure all tests generate files which have a heading that
shows which Unicode version was used to generate them. Some tests do this already,
but many do not.

== S15-nfg/emoji-test-gen.raku

.Steps
* Run `S15-nfg/emoji-test-gen.raku $UNIDATA/emoji-<version>/emoji-test.txt`,
  where `<version>` indicates what Emoji version you're updating for. *Do not*
  use the unversioned `$UNIDATA/emoji/` directory, this does not contain the
  necessary file.

.Version
* Emoji v17.0

.Date
* Regenerated 2025.12.03

.Generated File(s)
* `S15-nfg/emoji-test.t`

== S15-nfg/mass-roundtrip-test-gen.raku

.Steps
* Run `S15-nfg/mass-roundtrip-test-gen.raku --test
  $UNIDATA/NormalizationTest.txt` to test the new normalization tests. If
  everything passes, proceed.
* Run `S15-nfg/mass-roundtrip-test-gen.raku $UNIDATA/NormalizationTest.txt` to
  generate the tests.

.Version
* Unicode v17.0.0

.Date
* Regenerated 2025.12.03

.Generated File(s)
* `S15-nfg/mass-roundtrip-nfc.t`
* `S15-nfg/mass-roundtrip-nfd.t`
* `S15-nfg/mass-roundtrip-nfkc.t`
* `S15-nfg/mass-roundtrip-nfkd.t`


== S32-str/CollationTestGen.raku

.Steps
* Enter the `S32-str` directory. (This is necessary to ensure the files are
  written to the right location.)

* Run `raku ./CollationTestGen.raku --folder=$UNIDATA/UCA/CollationTest
  --test`.

* There may be some failures. If and only if all the failures look OK (as in,
  they're not actually an issue), follow the script's instructions to update
  things to mark them as "todo" tests, and proceed.

* Run `raku ./CollationTestGen.raku --folder=$UNIDATA/UCA/CollationTest` (that
  is, _without_ the `--test` this time) to generate the files.

* Check for any new `S32-str/CollationTest_NON_IGNORABLE-<n>.t` files. As
  Unicode updates, there may be enough new collation tests that the generator
  script creates a brand new numbered file, which will have to be added to
  `spectest.data`.

.Version
* Unicode v17.0.0

.Date
* Regenerated 2025.12.03

.Generated File(s)
* `S32-str/CollationTest_NON_IGNORABLE-<n>.t`

== S15-normalization/test-gen.raku

.Steps
* Run `S15-normalization/test-gen.raku $UNIDATA/NormalizationTest.txt`

* Check for possible new test files to be added to `spectest.data` (see the
  generated file section; look out for files with `<n>` in their pattern).

.Version
* Unicode v17.0.0

.Date
* Regenerated 2025.12.03

.Generated File(s)
* `S15-normalization/nfc-<n>.t`
* `S15-normalization/nfc-sanity.t`
* `S15-normalization/nfd-<n>.t`
* `S15-normalization/nfd-sanity.t`
* `S15-normalization/nfkc-<n>.t`
* `S15-normalization/nfkc-sanity.t`
* `S15-normalization/nfkd-<n>.t`
* `S15-normalization/nfc-concat.t`

== tools/GraphemeBreakTestGen.raku

.Steps

* Run `raku tools/GraphemeBreakTestGen.raku $UNIDATA/auxiliary/GraphemeBreakTest.txt`.

* Check for possible new files to be added to `spectest.data` (check the
  generated files list for the patterns to look for).

.Version
* Unicode v17.0.0

.Date
* Regenerated 2025.12.03

.Generated File(s)
* `S15-nfg/GraphemeBreakTest-<n>.t`

== External

=== Also you will have to do stuff in rakudo

Edit `t/09-moar/UnipropCheck.rakumod` folder to be set to the current version.

You will have to now copy:

--------------------------------------------------
mkdir -p t/3rdparty/Unicode/<version>/extracted
cp $UNIDATA/LineBreak.txt $UNIDATA/UnicodeData.txt t/3rdparty/Unicode/<version>
cp $UNIDATA/extracted/DerivedGeneralCategory.txt t/3rdparty/Unicode/<version>/extracted
--------------------------------------------------
