parakeet.training package

Submodules

parakeet.training.cli module

parakeet.training.cli.default_argument_parser()

A simple yet genral argument parser for experiments with parakeet.

This is used in examples with parakeet. And it is intended to be used by other experiments with parakeet. It requires a minimal set of command line arguments to start a training script.

The --config and --opts are used for overwrite the deault configuration.

The --data and --output specifies the data path and output path. Resuming training from existing progress at the output directory is the intended default behavior.

The --checkpoint_path specifies the checkpoint to load from.

The --device and --nprocs specifies how to run the training.

Returns

argparse.ArgumentParser

the parser

See also

parakeet.training.experiment

parakeet.training.default_config module

parakeet.training.default_config.get_default_training_config()

parakeet.training.experiment module

class parakeet.training.experiment.ExperimentBase(config, args)

Bases: object

An experiment template in order to structure the training code and take care of saving, loading, logging, visualization stuffs. It’s intended to be flexible and simple.

So it only handles output directory (create directory for the output, create a checkpoint directory, dump the config in use and create visualizer and logger) in a standard way without enforcing any input-output protocols to the model and dataloader. It leaves the main part for the user to implement their own (setup the model, criterion, optimizer, define a training step, define a validation function and customize all the text and visual logs).

It does not save too much boilerplate code. The users still have to write the forward/backward/update mannually, but they are free to add non-standard behaviors if needed.

We have some conventions to follow. 1. Experiment should have model, optimizer, train_loader and valid_loader, config and args attributes. 2. The config should have a training field, which has valid_interval, save_interval and max_iteration keys. It is used as the trigger to invoke validation, checkpointing and stop of the experiment. 3. There are four methods, namely train_batch, valid, setup_model and setup_dataloader that should be implemented.

Feel free to add/overwrite other methods and standalone functions if you need.

Parameters

config: yacs.config.CfgNode

The configuration used for the experiment.

args: argparse.Namespace

The parsed command line arguments.

Examples

>>> def main_sp(config, args):
>>>     exp = Experiment(config, args)
>>>     exp.setup()
>>>     exe.resume_or_load()
>>>     exp.run()
>>>
>>> config = get_cfg_defaults()
>>> parser = default_argument_parser()
>>> args = parser.parse_args()
>>> if args.config:
>>>     config.merge_from_file(args.config)
>>> if args.opts:
>>>     config.merge_from_list(args.opts)
>>> config.freeze()
>>>
>>> if args.nprocs > 1 and args.device == "gpu":
>>>     dist.spawn(main_sp, args=(config, args), nprocs=args.nprocs)
>>> else:
>>>     main_sp(config, args)

close()

Close visualizer to avoid hanging after training

dump_config()

Save the configuration used for this experiment.

It is saved in to config.yaml in the output directory at the beginning of the experiment.

init_parallel()

Init environment for multiprocess training.

new_epoch()

Reset the train loader and increment epoch.

property parallel

A flag indicating whether the experiment should run with multiprocessing.

read_batch()

Read a batch from the train_loader.

Returns

List[Tensor]

A batch.

resume_or_load()

Resume from latest checkpoint at checkpoints in the output directory or load a specified checkpoint.

If args.checkpoint_path is not None, load the checkpoint, else resume training.

run()

The routine of the experiment after setup. This method is intended to be used by the user.

save()

Save checkpoint (model parameters and optimizer states).

setup()

Setup the experiment.

setup_checkpointer()

Create a directory used to save checkpoints into.

It is “checkpoints” inside the output directory.

setup_dataloader()

Setup training dataloader and validation dataloader. A subclass should implement this method.

setup_logger()

Initialize a text logger to log the experiment.

Each process has its own text logger. The logging message is write to the standard output and a text file named worker_n.log in the output directory, where n means the rank of the process.

setup_model()

Setup model, criterion and optimizer, etc. A subclass should implement this method.

setup_output_dir()

Create a directory used for output.

setup_visualizer()

Initialize a visualizer to log the experiment.

The visual log is saved in the output directory.

Notes

Only the main process has a visualizer with it. Use multiple visualizers in multiprocess to write to a same log file may cause unexpected behaviors.

train()

The training process.

It includes forward/backward/update and periodical validation and saving.

train_batch()

The training loop. A subclass should implement this method.

valid()

The validation. A subclass should implement this method.

Module contents