Indexing reference genomes

The above command took ~27 seconds with a single thread. Using multiple threads can give significant speed-ups for large reference sets, but this is optional. Note that memory use increases with more threads, so you may want to reduce the thread count if you run out of memory during indexing. It does not affect memory use at query time, once indexing is finished.

Peak memory usage during index construction can also be controlled by partitioning the index into smaller pieces. This is done by a FracMinHash-based batching, controlled by options -m and -r. krepp partitions the index into -m (more or less) equally sized pieces; these partitions can be built independently but queried together. The -r option determines which partition is constructed: if --no-frac is given, only the -rth partition is built; otherwise all partitions from 0th to -rth are built and saved (see krepp index --help for details). You don't need to construct all partitions; krepp will search in whatever is available, and these partitions can be distributed independently. The default is -m 4 -r 1 --frac, so 50% (0th and 1st out of 4 partitions) of the minimized k-mers will be indexed. The only requirement is keeping the -m value (and of course -i and -t) fixed across all partitions. For instance, one can index 3% of the reference k-mers and construct a lightweight index by running:

krepp index -o /path/to/index -i /path/to/input.tsv -t /path/to/tree.nwk -m 100 -r 2 --no-frackrepp index -o /path/to/index -i /path/to/input.tsv -t /path/to/tree.nwk -m 100 -r 1 --no-frackrepp index -o /path/to/index -i /path/to/input.tsv -t /path/to/tree.nwk -m 100 -r 0 --no-frac

or, alternatively

krepp index -o /path/to/index -i /path/to/input.tsv -t /path/to/tree.nwk -m 100 -r 2 --frac

This is faster but uses more memory during index construction, despite resulting in an index of the same size. If this doesn't work well for your task, you can keep adding batches partially and make index larger:

krepp index -o /path/to/index -i /path/to/input.tsv -t /path/to/tree.nwk -m 100 -r 3 --no-frackrepp index -o /path/to/index -i /path/to/input.tsv -t /path/to/tree.nwk -m 100 -r 4 --no-frac

or, if you have enough memory, you can do this in parallel:

seq 3 99 | xargs -I{} -P4 \    krepp index -o /path/to/index -i /path/to/input.tsv -t /path/to/tree.nwk -m 100 -r {} --no-frac

If you have a phylogeny available (in Newick format), run krepp index with the -t option and the path to the Newick tree (inside data/):

krepp index \    -i input_map.tsv \    -o toy-index \    -t reference_tree.nwk \    --num-threads 4

An index constructed with a backbone can be used for phylogenetic placement without providing a tree later, and distance estimates are unaffected. You can always specify a backbone with the krepp place subcommand, using the -t (--nwk-file) option. Note that the tip labels of the phylogeny must match the reference IDs given in -i /path/to/input.tsv for krepp index.

The available indexes are mostly microbial and the majority come with a reference phylogeny (for placement). Some major instances are listed below; they vary in size to fit the memory available to you:

• Web of Life (WoL): lightweight and a reliable phylogeny (67 GB, 41 GB, 2 GB ← we downloaded this one)

• RefSeq-microbial: denser sampling and an ultra-large uDance phylogeny (180 GB)

• GTDB: the latest release, an up-to-date reference (230 GB, 100 GB)

  

  For any reference database, the rule of thumb is to use the largest index that fits your available memory. The indexes vary in size but are based on the same genome set. The level of compression and the extent of k-mer downsampling differ. Depending on the query data and the task, a small index might perform as accurately as a large one. The downloads can be either through our servers or an AWS S3 bucket.