2.1. The PHENIX software development environment

2.1.3. A unifying user interface toolkit (Phil)

The third fundamental component behind the PHENIX GUI is the Python-based hierarchical interchange language (Phil; https://cctbx.sourceforge.net/libtbx_phil.html ), which was developed as part of the PHENIX project (Grosse-Kunstleve et al., 2005). Phil is designed to simultaneously enable intuitive command-line user interfaces and aid in largely automating the generation of graphical user interfaces. PHENIX command-line users are presented with a minimal but powerful syntax for defining input parameters; the only two major syntax elements are the equal sign for keyword-value assignments and curly braces for the delineation of a hierarchical structure. For methods developers, Phil provides flexible means for the modular assembly of parameter files. The Phil modules are typically implemented along with the algorithms using the parameters. Phil supports embedded help text for users. Typically the algorithms and the Phil modules including the help text are kept in the same Python source code file. This arrangement ensures that the algorithms, associated parameters and help texts can easily be developed and maintained together and are approachable by new methods developers.

Phil parameter files can be automatically converted to a graphical presentation. This makes use of metadata embedded in the Phil modules, primarily the type for each parameter (for example int, float, choice, path etc.), which is defined by the developer. For each type there is a corresponding default presentation in the GUI. Optionally, the type information is augmented by secondary information defined by the developer, to direct customized presentation in the GUI. A typical example is shown in Figs. 1(a) and 1(b). Nearly all of the information required to create and manage the graphical controls for editing these parameters is contained in the metadata. The hierarchical organization of the Phil parameters helps determine the window layout. Developers may modify most of the options for underlying programs without disrupting the GUI. The user-modified parameters are retrieved from the GUI by forming new internal Phil objects, which may be saved as input files for the command-line programs or used directly to start computational processes. Advanced users may opt to edit the Phil files prior to executing the processes. The need for repetitive input is reduced both by the ability to restore and merge past inputs and by the option of saving default settings for specific contexts.

Figure 1 (a) Example of phenix.refine parameter specifications using the libtbx.phil module in cctbx, with metadata used in the GUI. (b) Controls drawn automatically based on the specifications in (a). (c) A practical example of fully automatic interface generation: PDB file preparation for refinement in phenix.ready_set.

Each Phil parameter definition or hierarchical level is tagged with an `expert level' attribute, which allows the display in the GUI to be limited to basic settings or increasing levels of complexity. Because many of the settings are distracting for novices or used primarily for development purposes, the default interface is kept simple, and a minimal set of controls is displayed in the main window.

The Phil implementation is intended to be reusable for a wide variety of purposes. To this end, the Phil type system is designed to be extensible from Python. For example, the extension mechanism is used to support crystallography-specific types, most notably unit-cell and space-group types. These accommodate common abbreviations and syntactical variations. For example, the space-group symbols `p212121' or `19' are converted to the standardized form `P 21 21 21'.

2.2. Execution of programs

The PHENIX software development framework allows any program to be run within the GUI as a Python module. This enables immediate feedback, such as real-time plots or progress bars. For maximum flexibility, however, several additional approaches to starting processes have been implemented. Programs may be started either directly from the GUI, as separate processes on the same computer, or on a cluster managed by a batch queuing system (for example the Sun Grid Engine or the Portable Batch System). The latter two mechanisms allow the GUI to be closed (and resumed later) without interrupting running processes, at the cost of losing some interactive features that require direct communication between the GUI and running process. In all cases, the printed console output is automatically propagated to the GUI. More complex output is sent to the GUI on a case-by-case basis using `callbacks', which encapsulate intermediate data to be saved to temporary files or sent directly via interprocess-communication objects. Information sent to the GUI by this mechanism may include warning messages, tables of statistics, plot data, or even models and maps. The ability to propagate higher-level data to the GUI makes it possible to provide fine-grained indicators of program progress using easily interpretable presentations instead of relying on the text output alone.

2.3. Testing

An essential feature of collaborative development, or any long-term software project, is frequent testing of all basic functionality to guard against bugs introduced by code changes. In the context of PHENIX and the cctbx (Adams et al., 2010), this is primarily accomplished by running nightly builds and a set of regression tests on all supported platforms. Automated testing of graphical code is notoriously difficult because of the interactive nature of the software. However, careful separation of core logic and presentation allows the nongraphical components to be tested as part of a nightly build system. This includes consistency checks to ensure that none of the parameters represented in the GUI have been removed from the underlying program. Graphical controls are implemented as separate modules with minimal dependencies where possible, which makes them potentially reusable for other purposes and less prone to errors as a result of changes to other code. Each control includes a simple test program that can be run manually as a standalone command, allowing quick verification of code changes without the burden of running the entire application.

3.1. Overview of the PHENIX GUI

As of the current release (1.7.3, December 2011), most of the major programs in PHENIX are available in the GUI, including phenix.refine (Afonine et al., 2012), the AutoSol, AutoBuild, AutoMR and LigandFit wizards (Terwilliger et al., 2006, 2008, 2009), Phaser (McCoy et al., 2007), Xtriage (Zwart et al., 2005), eLBOW (Moriarty et al., 2009), MR-Rosetta (DiMaio et al., 2011), a comprehensive validation suite largely derived from MolProbity (Chen et al., 2010), POLYGON (Urzhumtseva et al., 2009), and phenix.model_vs_data (Afonine et al., 2010), as well as many simpler programs for building, map calculations, restraints editing, file manipulation, data analysis and visualization.

Nearly all of the programs in PHENIX are now configurable by Phil; in those cases where the original code was designed around this framework (such as phenix.refine and Xtriage), few modifications other than addition of GUI-specific metadata were required. For applications that pre-date Phil, such as the wizards and eLBOW, additional wrapper code was required to provide an interface between Phil and the configuration mechanism used internally. In the case of Phaser, an entirely new command-line interface was added as a bridge between the Phil layer and the existing Python API (McCoy et al., 2007).

The Phil-based system has proved essential for maintaining the interfaces to applications such as phenix.refine, which currently has more than 900 unique parameters. Most changes to parameters can automatically be propagated to the PHENIX GUI, and most customizations are made in the Phil metadata rather than Python code. For particularly complicated input types (especially file input), embedded keywords may specify callback functions to be executed when a parameter is changed. Some common parameters that use the basic types often have additional restrictions on syntax and may be tightly coupled to other parameters, such as the selection of column labels (as comma-separated strings) from reflection data files; these are flagged by style keywords in the Phil syntax and handled by specialized controls. Additional customization of individual applications is often inevitable, and nearly all interfaces contain some custom layout code for the top-level controls. However, for simple programs an entirely automatic interface is possible (Fig. 1c), and even the most complex interface, that for phenix.refine, presents more than 90% of parameters without any customization.

The organization of the central GUI (the `phenix' command) combines a list of user-defined projects and their current status with individual programs grouped by category (Fig. 2). Internally, Phil is also used to define much of the information about available programs, such as module locations and the content displayed in the widgets in the main GUI. On the Macintosh platform, help links embedded in the Phil specification are used to generate buttons to navigate directly to the HTML documentation from anywhere in the GUI. Phil is also the primary storage format for project tracking. Job history for the currently selected project is displayed in a separate window (reached through the `Job history' tab). Both jobs and projects can be sorted on the basis of modification time or current best R_free (Brünger, 1992) (extracted at the end of each job if it is calculated). Currently job tracking is the primary function of projects, but this functionality will be extended in the future to manage default settings and inputs. For demonstration purposes, PHENIX is distributed with a number of tutorial data sets, which can be readily set up as projects directly from the GUI.

Figure 2 Screen capture of the main window of the PHENIX GUI running on a Macintosh computer.

Although they share a common configuration system, the output of the individual programs is heterogeneous, and a variety of mechanisms have been implemented to present the progress and results of each application visually. Larger programs such as phenix.refine send entire molecules and maps via callbacks as they are updated, and this output is displayed in graphics programs such as Coot (Emsley et al., 2010) (see below). Final results may encompass validation reports, models, tables for plotting and any information written to files. These results (and any intermediate results from callbacks) are saved to disk and can be quickly reloaded and displayed in the GUI in future sessions. For many programs, the final result returned is a simple list of files created, which can be automatically displayed using a generic procedure. The heterogeneous nature of the large programs and the complexity of the information returned requires the use of manual layout for many results, although re-use of common classes is made wherever possible.

3.2. Interfacing with molecular graphics

The PHENIX GUI includes basic three-dimensional graphics functionality, for example the ability to display wireframe atomic models and electron density maps. However, manual model building or publication-quality graphics are outside the scope of the PHENIX project. Therefore, we have implemented interface layers for the crystallographic building program Coot and the general-purpose molecular graphics program PyMOL (DeLano, 2002). Both programs are extensible from Python. Most of the communication is through files [PDB, MTZ or CCP4 (Collaborative Computational Project, Number 4, 1994)], but a local network connection is used internally, allowing functions in the Coot and PyMOL Python APIs (for instance, loading and contouring a map file) to be called remotely from within PHENIX. Interactive model validation summaries (`kinemages') may also be loaded in KiNG (Chen et al., 2009), providing visual validation similar to that available in MolProbity.

All applications in the GUI support the loading of data files directly into Coot, and the majority also support PyMOL and/or the built-in graphical viewer. Any necessary file conversions or additional commands, such as a fast Fourier transform to generate a CCP4-formatted map file, are handled internally. Thus, the final model and map coefficients from refinement can be opened in Coot or PyMOL by a simple mouse click in the PHENIX GUI, and PDB files used for input may be opened in one of the viewers instead of a text editor. A more powerful application of the graphics extensions is continuous display of results during computation, by saving intermediate data and re-loading the files as they change. Similar interactivity has previously been implemented in VMD (visual molecular dynamics; Humphrey et al., 1996), both for visualizing molecular dynamics simulations and for NMR refinement (Schwieters & Clore, 2001), and applied to visualizing the progress of the building program ARP/wARP (Payne et al., 2005; Langer et al., 2008). This is most useful in refinement where one can watch model and map changes in real time. Other programs that iteratively rebuild models (currently AutoSol and AutoBuild) produce similar output at longer intervals.

For applications requiring a particularly high degree of interactivity and two-way communication, extending the Coot GUI via the Coot Python API is necessary. This approach has been used to facilitate editing of input models for molecular replacement with the Sculptor utility (Bunkóczi & Read, 2011). In most situations, however, it has been sufficient to use a direct network connection instead. When displaying validation results, therefore, instead of creating a redundant interface in Coot to list outliers, the list controls in PHENIX re-center the attached Coot window on the selected residue (Fig. 3). Controlling PHENIX from Coot via the same mechanism is also possible. Currently this is limited to starting the phenix.refine GUI and loading a selected model, but more complex operations such as atom picking or selecting from a choice of automatically fitted loops could be added.

Figure 3 Integrated validation application, showing a list of outliers for all-atom contact analysis, and the multi-criterion plot combining geometry outliers, electron-density levels and B factors. The model, maps and Probe dots (Word et al., 1999) are automatically loaded into Coot, and the view is re-centered on the selected atom.

3.3. Automation in a GUI framework

Although the new PHENIX GUI has been designed around individual programs, many of the most common automation strategies for phasing and model building are implemented in the wizards (Terwilliger et al., 2006, 2008, 2009). The GUI provides a framework for further simplifying the process, primarily by linking related actions and easing the transition between programs. In the interface for phenix.refine, for instance, common tasks such as setting up restraints and adding hydrogen or deuterium atoms (via the phenix.ready_set program), identification of noncrystallographic symmetry (NCS) groups (phenix.simple_ncs_from_pdb), rapid identification of translation/libration/screw (TLS) groups (phenix.find_tls), and assigning secondary structure for the purpose of generating hydrogen-bond restraints (phenix.secondary_structure_restraints) are easily accessed and the results incorporated into the refinement input. At the end of each run, the validation suite is automatically executed and the results displayed with refinement statistics (Fig. 3).

A simpler, but extremely important, type of automation is for programs to suggest appropriate next steps after a job is complete, and provide an easy and direct transition to other programs. For example, the wizard GUIs display buttons for launching phenix.refine with input fields filled in with data calculated by the wizard, and for running other wizards where appropriate. This allows new `pipelines' to be constructed by chaining of GUI modules instead of integrating programs at a lower level, such as molecular replacement single-wavelength anomalous diffraction (MR-SAD) phasing, which is performed in PHENIX simply by running the molecular replacement GUI, then clicking a button labeled `Run MR-SAD' to launch a separate wizard with the required inputs. Future developments will address common parallel workflows, such as evaluating multiple molecular replacement search models, or validating several closely related structures simultaneously.