Bundle Example: Fetch from Network Database

This example describes how to create a ChimeraX bundle that retrieves data from a network source. In this example, the network source is HomoloGene from NCBI.

The steps in implementing the bundle are:

1. Create a bundle_info.xml containing information about the bundle,

2. Create a Python package that interfaces with ChimeraX and implements the file-reading functionality, and

3. Install and test the bundle in ChimeraX.

The final step builds a Python wheel that ChimeraX uses to install the bundle. So if the bundle passes testing, it is immediately available for sharing with other users.

Source Code Organization

The source code for this example may be downloaded as a zip-format file containing a folder named tut_fetch. Alternatively, one can start with an empty folder and create source files based on the samples below. The source folder may be arbitrarily named, as it is only used during installation; however, avoiding whitespace characters in the folder name bypasses the need to type quote characters in some steps.

Sample Files

The files in the tut_fetch folder are:

• tut_fetch - bundle folder

  • bundle_info.xml - bundle information read by ChimeraX

  • src - source code to Python package for bundle

    • __init__.py - package initializer and interface to ChimeraX

    • fetch.py - source code to retrieve HomoloGene entries

The file contents are shown below.

bundle_info.xml

bundle_info.xml is an eXtensible Markup Language format file whose tags are listed in Bundle Information XML Tags. While there are many tags defined, only a few are needed for bundles written completely in Python. The bundle_info.xml in this example is similar to the one from the Bundle Example: Add a Tool example with changes highlighted. For explanations of the unhighlighted sections, please see Bundle Example: Hello World, Bundle Example: Add a Command and Bundle Example: Add a Tool.

<!--
ChimeraX bundle names must start with "ChimeraX-"
to avoid clashes with package names in pypi.python.org.
When uploaded to the ChimeraX toolshed, the bundle
will be displayed without the ChimeraX- prefix.
-->

<BundleInfo name="ChimeraX-TutorialFetch"
	    version="0.1" package="chimerax.tut_fetch"
  	    minSessionVersion="1" maxSessionVersion="1">

  <!-- Additional information about bundle source -->
  <Author>UCSF RBVI</Author>
  <Email>chimerax@cgl.ucsf.edu</Email>
  <URL>https://www.rbvi.ucsf.edu/chimerax/</URL>

  <!-- Synopsis is a one-line description
       Description is a full multi-line description -->
  <Synopsis>Example for fetching sequence alignment from HomoloGene</Synopsis>
  <Description>Example code for implementing ChimeraX bundle.

Implements capability for fetching and displaying sequence alignments
from HomoloGene.
  </Description>

  <!-- Categories is a list where this bundle should appear -->
  <Categories>
    <Category name="General"/>
  </Categories>

  <!-- Dependencies on other ChimeraX/Python packages -->
  <Dependencies>
    <Dependency name="ChimeraX-Core" version="~=1.1"/>
    <Dependency name="ChimeraX-Alignments" version="~=1.0"/>
  </Dependencies>

  <!-- Register HomoloGene as a fetch source.  The downloaded file
    will (almost) be in FASTA format and should be displayable
    using ChimeraX alignment tools.  If we were using a format
    not supported by ChimeraX, we would need to supply
    "DataFormat" and "Open" ChimeraXClassifiers as well.  -->
  <Providers manager="open command">
    <Provider name="homologene" type="fetch" format_name="fasta" example_ids="87131" />
  </Providers>

  <Classifiers>
    <!-- Development Status should be compatible with bundle version number -->
    <PythonClassifier>Development Status :: 3 - Alpha</PythonClassifier>
    <PythonClassifier>License :: Freeware</PythonClassifier>
  </Classifiers>

</BundleInfo>

The BundleInfo, Synopsis and Description tags are changed to reflect the new bundle name and documentation (lines 8-10 and 17-24).

The Providers sections on lines 42 through 44 use the Manager/Provider protocol to inform the “open command” manager that this bundle supports fetching data from a database named homologene (really HomoloGene, but the user will type “homologene”).

The attributes usable with the “open command” manager (with type="fetch") are described in detail in Fetching Files.

src

src is the folder containing the source code for the Python package that implements the bundle functionality. The ChimeraX devel command, used for building and installing bundles, automatically includes all .py files in src as part of the bundle. (Additional files may also be included using bundle information tags such as DataFiles as shown in Bundle Example: Add a Tool.) The only required file in src is __init__.py. Other .py files are typically arranged to implement different types of functionality. For example, cmd.py is used for command-line commands; tool.py or gui.py for graphical interfaces; io.py for reading and saving files, etc.

src/__init__.py

As described in Bundle Example: Hello World, __init__.py contains the initialization code that defines the bundle_api object that ChimeraX needs in order to invoke bundle functionality. ChimeraX expects bundle_api class to be derived from chimerax.core.toolshed.BundleAPI with methods overridden for registering commands, tools, etc.

# vim: set expandtab shiftwidth=4 softtabstop=4:

from chimerax.core.toolshed import BundleAPI


# Subclass from chimerax.core.toolshed.BundleAPI and
# override the method for fetching from databases,
# inheriting all other methods from the base class.
class _MyAPI(BundleAPI):

    api_version = 1

    @staticmethod
    def run_provider(session, name, mgr):
        # 'run_provider' is called by a manager to invoke the 
        # functionality of the provider.  Since we only provide
        # single provider to a single manager, we know this method
        # will only be called by the "open command" manager to
        # fetch HomoloGene data, and customize this routine accordingly.
        #
        # The 'name' arg will be the same as the 'name' attribute
        # of your Provider tag, and mgr will be the corresponding
        # Manager instance
        #
        # For the "open command" manager with type="fetch", this method
        # must return a chimerax.open_command.FetcherInfo subclass instance.
        from chimerax.open_command import FetcherInfo
        class HomoloGeneFetcherInfo(FetcherInfo):
            def fetch(self, session, identifier, format_name, ignore_cache, **kw):
                from .fetch import fetch_homologene
                return fetch_homologene(session, identifier, ignore_cache=ignore_cache, **kw)
        return HomoloGeneFetcherInfo()


# Create the ``bundle_api`` object that ChimeraX expects.
bundle_api = _MyAPI()

The run_provider() method is called by a ChimeraX manager when it needs additional information from a provider or it needs a provider to execute a task. The session argument is a Session instance, the name argument is the same as the name attribute in your Provider tag, and the mgr argument is the manager instance. These arguments can be used to decide what to do when your bundle offers several Provider tags (to possibly several managers), but since this bundle only declares one provider to one manager, we know it will be called by the “open command” manager to fetch HomoloGene data and don’t need to check the run_provider() arguments.

When called by the “open command” manager (that was given the type="fetch" Provider tag), run_provider() must return an instance of a subclass of chimerax.open_command.FetcherInfo. The methods of the class are thoroughly documented if you click the preceding link, but briefly:

1. The fetch() method is called to actually fetch the data and should return a (models, status message) tuple. Do not add the models to the session — that will done by the calling function.

2. The ignore_cache argument indicates whether your routine should use locally cached data (if any) or instead ignore the cache and fetch the data again. Some types of data fetches may not amenable to caching at all, but for those that are the caching is usually implemented automatically by having the fetching function use the chimerax.core.fetch.fetch_file() routine, which takes an ignore_cache keyword argument.

3. If there are fetch-specific keyword arguments that the open command should handle, then a fetch_args property should be implemented, which returns a dictionary mapping Python keyword names to Annotation subclasses. Such keywords will be passed to your fetch() method, along with format-specific keywords. Note that format-specific keywords are known from the open_args property of the bundle that opens the data’s format, and should not be included in the dictionary returned by fetch_args, so therefore it is rarely necessary to actually implement the fetch_args property.

For this example, the format_name argument is omitted because the bundle only supports FASTA format. All other arguments are passed through to fetch.fetch_homologene to actually retrieve and process the data.

src/fetch.py

# vim: set expandtab shiftwidth=4 softtabstop=4:

_URL = ("https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi"
        "?db=homologene"
        "&id=%s"
        "&rettype=fasta"
        "&retmode=text")


def fetch_homologene(session, ident, ignore_cache=True, **kw):
    """Fetch and display sequence alignment for 'ident' from HomoloGene.

    Use Python library to download the FASTA file and use ChimeraX
    alignment tools for display.
    """
    # First fetch the file using ChimeraX core function
    url = _URL % ident
    session.logger.status("Fetching HomoloGene %s" % ident)
    save_name = "%s.fa" % ident
    from chimerax.core.fetch import fetch_file
    filename = fetch_file(session, url, "HomoloGene %s" % ident, save_name,
                "HomoloGene", ignore_cache=ignore_cache, uncompress=True)

    session.logger.status("Opening HomoloGene %s" % ident)
    models, status = session.open_command.open_data(filename, alignment=False, name=ident)
    return models, status

The fetch_homologene function performs the following steps:

• create an URL for fetching content for the given identifier and an output file name where the content will be saved (lines 17-19),

• call chimerax.core.fetch.fetch_file() to retrieve the actual contents (lines 20-22),

• update status line (line 24),

• open the saved file using the “open command” manager’s open_data() method (line 25), which return a (models, status message) tuple.

• return the list of models created and status message (line 26)

Building and Testing Bundles

To build a bundle, start ChimeraX and execute the command:

devel build PATH_TO_SOURCE_CODE_FOLDER

Python source code and other resource files are copied into a build sub-folder below the source code folder. C/C++ source files, if any, are compiled and also copied into the build folder. The files in build are then assembled into a Python wheel in the dist sub-folder. The file with the .whl extension in the dist folder is the ChimeraX bundle.

To test the bundle, execute the ChimeraX command:

devel install PATH_TO_SOURCE_CODE_FOLDER

This will build the bundle, if necessary, and install the bundle in ChimeraX. Bundle functionality should be available immediately.

To remove temporary files created while building the bundle, execute the ChimeraX command:

devel clean PATH_TO_SOURCE_CODE_FOLDER

Some files, such as the bundle itself, may still remain and need to be removed manually.

Building bundles as part of a batch process is straightforward, as these ChimeraX commands may be invoked directly by using commands such as:

ChimeraX --nogui --exit --cmd 'devel install PATH_TO_SOURCE_CODE_FOLDER exit true'

This example executes the devel install command without displaying a graphics window (--nogui) and exits immediately after installation (exit true). The initial --exit flag guarantees that ChimeraX will exit even if installation fails for some reason.

Distributing Bundles

With ChimeraX bundles being packaged as standard Python wheel-format files, they can be distributed as plain files and installed using the ChimeraX toolshed install command. Thus, electronic mail, web sites and file sharing services can all be used to distribute ChimeraX bundles.

Private distributions are most useful during bundle development, when circulation may be limited to testers. When bundles are ready for public release, they can be published on the ChimeraX Toolshed, which is designed to help developers by eliminating the need for custom distribution channels, and to aid users by providing a central repository where bundles with a variety of different functionality may be found.

Customizable information for each bundle on the toolshed includes its description, screen captures, authors, citation instructions and license terms. Automatically maintained information includes release history and download statistics.

To submit a bundle for publication on the toolshed, you must first sign in. Currently, only Google sign in is supported. Once signed in, use the Submit a Bundle link at the top of the page to initiate submission, and follow the instructions. The first time a bundle is submitted to the toolshed, it is held for inspection by the ChimeraX team, which may contact the authors for more information. Once approved, all subsequent submissions of new versions of the bundle are posted immediately on the site.

What’s Next

• Bundle Example: Hello World

• Bundle Example: Add a Command

• Bundle Example: Add a Tool

• Bundle Example: Read a New File Format

• Bundle Example: Save a New File Format (previous topic)

• Bundle Example: Fetch from Network Database (current topic)

• Bundle Example: Define a Chemical Subgroup Selector (next topic)

• Bundle Example: Define Presets