.. _with_jobman: =========================== Using jobman with Pylearn2 =========================== `Jobman`_ enables you to store experiments, their original hyper-parameters, and some results, in a database. It is useful, for instance, to schedule experiments to be launched later, launch them (for instance, use a cluster to launch them all in parallel), and finally collect some measurements (reconstruction error, classification error, training time, for instance) and add them in the database. Then, we can use queries on that database to do some analyses of the results. For the moment, what can be inside the database is limited to simple data types, and things that are not too big in memory: integers, floating-point numbers, strings, and dictionaries of the above (if the keys are strings). Bases of Jobman =============== * state dictionary, linked with the DB * experiment function, exp(state, channel) See `Jobman documentation `_ for installation instructions, and information on how to use it. .. _Jobman: http://deeplearning.net/software/jobman pylearn2/jobman interface ========================= Such an ``experiment`` function is in Pylearn2 :file:`pylearn2/scripts/jobman/experiment.py`, and will use a specially-crafted ``state`` dictionary, that specifies: * which experiment (model, trainer, and so on) to instanciate; * which hyper-parameters exist and which values they should take; * and finally which results (or outputs) you want to extract from the trained experiment and store in the database. For the moment, only one such experiment function exists: ``pylearn2.scripts.jobman.train_experiment``, that trains a model the same way ``scripts/train.py`` would. Other functions may be implemented in the future, with the same structure in ``state``. The ``state`` dictionary ------------------------ Here's the structure for the ``state`` dictionary-like object, that represents a single experiment. For your convenience, you can access to ``state``'s properties either as items of a dictionary (``state["some_attr"]``) or as attributes of an object (``state.some_attr``). .. attribute:: state.yaml_template String value This is a string containing a template for the YAML file that will represent your experiment. This template will typically be shared among different experiments, when these experiments differ only by the value of some hyper-parameters. Instead of specifying the values of hyper-parameters in the YAML file, you can specify them using the Python substitution syntax. For instance, if one of your hyper-parameters is a learning rate, you can specify the following in your template: .. code-block:: python "learning_rate": %(learning_rate)d If you want to specify a whole object as a hyper-parameter (for instance, if you want to try different classes), you can take advantage from the fact that Python's serialization for dictionaries is close to YAML. For instance, you can specify: .. code-block:: python "termination_criterion": %(term_crit_builder)s %(term_crit_args)s The identifier inside the parentheses is the *name* you give to your hyper-parameter, the letter afterwards identifies its type (here, "s" for string, "d" for decimal number, every type can usually be converted automatically into a string). These "place-holders" in the template will be replaced by actual values, that are provided in :attr:`state.hyper_parameters`. .. attribute:: state.hyper_parameters Dictionary, all keys must be strings. This dictionary contains the mapping from hyper-parameter names, as used in the yaml_template string, to the actual value you want to give them in that particular experiment. For instance, for a YAML template like the one above, a possible value of ``state.hyper_parameters`` could be: .. code-block:: python { "learning_rate": 1e-4, "term_crit_builder": "!obj:pylearn2.training_algorithms.sgd.EpochCounter", "term_crit_args": { "max_epochs": 2 } } The resulting YAML file that will be used in your experiment will look like: .. code-block:: yaml { [...] "learning_rate": 1e-4, [...] "termination_criterion": !obj:pylearn2.training_algorithms.sgd.EpochCounter { max_epochs: 2, }, } You can achieve the same effect by specifying only one place-holder for a complete YAML object. If your template is: .. code-block:: python { "termination_criterion": %(term_crit)s } and you have, in state.term_crit: .. code-block:: python { '__builder__': 'pylearn2.training_algorithms.sgd.EpochCounter', 'max_epochs': 2 } The special key ``'__builder__'`` will be used as the constructor function of the object to build, which will result in the same final YAML string: .. code-block:: yaml { "termination_criterion": !obj:pylearn2.training_algorithms.sgd.EpochCounter { max_epochs: 2, } } When that YAML string gets generated, it will be parsed to build a ``train_object``, that will be trained (``train_object.main_loop()`` is called), then returned. That object will depend on what is described in the YAML file. Usually, you will want to extract some information from that object, and save that in the database, to be used as some metric. This is done by :attr:`state.extract_results`. .. attribute:: state.extract_results String, containing the name of a function to call on "train_object". The function named in :attr:`state.extract_results` takes one argument, and returns a dictionary. Like :attr:`state.hyper_parameters`, this dictionary's keys are strings, and values can be ints, floats, strings or dictionaries (with integer keys, and they can be nested). The returned dictionary will be used as :attr:`state.results`. This function has to be in some place where it can be importable. For instance, say we want to record the best epoch according to some validation error, and the training and validation errors (classification and NLL) at that training epoch. Those will kept in train_obj.model.monitor.channels, so our function may look like: .. code-block:: python def result_extractor(train_obj): import numpy channels = train_obj.model.monitor.channels best_index = numpy.argmin(channels["valid_nll"].val_record) return dict( best_epoch=best_index, train_nll=channels["train_nll"].val_record[best_index], train_classerr=channels["classerr"].val_record[best_index], valid_nll=channels["valid_nll"].val_record[best_index], valid_classerr=channels["valid_classerr"].val_record[best_index]) .. note:: The channels and their name will depend on the content of your YAML template string. If that function is placed in pylearn2/scripts/examples/extractor.py, we will specify: .. code-block:: python state.extract_results = "pylearn2.scripts.examples.extractor.result_extractor" For a working example, you can have a look at ``pylearn2/scripts/jobman/tester.py`` .. attribute:: state.results Dictionary, all keys are strings. This dictionary will be empty initially, but will be filled after the experiment finishes, by the call to :attr:`state.extract_results`. For instance, the result specification above may give the following result: .. code-block:: python { 'best_epoch': 47, 'train_nll': 0.02, 'train_classerr': 0.0, 'valid_nll': 0.05, 'valid_classerr': 0.03, } Having those informations in the database allows you to efficiently search for the model that has the lowest classification error, for instance. You can see a complete example of a ``state`` dictionary in ``pylearn2/scripts/jobman/tester.py``. pylearn2/scripts/jobman/experiment.py ------------------------------------- train_experiment ++++++++++++++++ Here is an outline of what is really happening inside the train_experiment function. * The ``state.hyper_parameters`` dictionary is converted to a YAML-friendly sub-class of dict. * Place-holders in ``state.yaml_template`` are replaced by the values contained in ``state.hyper_parameters``, to create a YAML string. * That string is passed to ``pylearn2.config.yaml_parse.load``, and a ``train_obj`` object is returned. * ``train_obj.main_loop()`` is called. * The function whose name is specified in ``state.extract_results`` is imported, then called on ``train_obj``, returning a results dictionary. * That dictionary is affected to ``state.results`` * The function exits, ``state`` gets synchronized with the database, and the working directory gets rsync'ed.