Visualising a hierarchy of substructures Part II

In an earlier post, I described a simple procedure to generate a hierarchy of substructures, and depicted the hierarchy with GraphViz. Pat Walters at Vertex realised that it is possible to add images as node labels in GraphViz and updated the script so that the image includes the actual chemical depictions (see below). He has also adapted the script to use OpenEye’s OEChem.

Update (8/11/2012): Wolf-D. Ihlenfeldt has implemented this with the Cactvs Toolkit and in addition has shown how to integrate the graph display with Knime.

#!/usr/bin/env python

import copy
import pickle
import os

from openeye.oechem import *
from openeye.oedepict import *

def mol2image(mol,imageFileName,width=200,height=200):
    clearcoords = True
    suppressH   = False
    opts = OE2DMolDisplayOptions(width, height, OEScale_AutoScale)
    itf = OEInterface()
    OESetup2DMolDisplayOptions(opts, itf)
    OEPrepareDepiction(mol, clearcoords, suppressH)
    disp = OE2DMolDisplay(mol, opts)
    ofs = oeofstream(imageFileName)
    name,ext = os.path.splitext(imageFileName)

def create_tree(structures):
    tree = {"Root":{}}
    stack = [(tree["Root"], structures.keys())]

    while len(stack) > 0:
        leaf, subset = stack.pop()

        max_matches = ("", [])
        for name in subset:
            smiles = structures[name]
            smarts = OESubSearch(smiles)
            matches = []
            for nameb in subset:
                if nameb != name:
                    molb = OEGraphMol()
                    if smarts.SingleMatch(molb):
            if len(matches) >= len(max_matches[1]):
                max_matches = (name, matches)
        if False: # Debug statement
            print max_matches

        for name in [max_matches[0]] + max_matches[1]:
        leaf[max_matches[0]] = {}
        if len(subset) > 0:
            stack.append( (leaf, subset) )
        if len(max_matches[1]) > 0:
            stack.append( (leaf[max_matches[0]], copy.deepcopy(max_matches[1])))

    with open("tmp.pickle", "w") as f:
        pickle.dump(tree, f)

def fix(name):
    return name.replace("-", "_").replace("1", "one").replace("2", "two").replace("3", "three").replace("5", "five").replace(",", "_")

def visit(name, leafdict):
    for x, y in leafdict.iteritems():
        if name != "Root":
            print ' %s -> %s;' % (name,x)
        print ' %s [label=<<TABLE><TR><TD><IMG SRC="%s/%s.png"/></TD></TR></TABLE>>];' % (x,os.getcwd(),x)
        visit(x, y)

if __name__ == "__main__":
    structureDict = dict([('001', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@@H](O)[CH]1(O)'),
                       ('002', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@H](O)[CH]1(O)'),
                       ('003', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@@H](O)[C@H]1(O)'),
                       ('004', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@@H](O)[C@@H]1(O)'),
                       ('005', 'C(O)[CH](O1)[CH](O)[CH](O)[CH](O)[CH]1(O)'),
                       ('006', 'c1ccccc1'),
                       ('007', 'c1ccccc1Br'),
                       ('008', 'c1ccc(Br)cc1Br'),
                       ('009', 'c1cccc(Br)c1Br'),
                       ('010', 'c1c(Br)cc(Br)cc1Br')])

    for k in sorted(structureDict.keys()):
        imageFileName = "%s.png" % (k)
        smi = structureDict[k]
        mol = OEGraphMol()

    with open("tmp.pickle", "r") as f:
        tree = pickle.load(f)
    print "digraph graphname {"
    print "node [shape=plaintext];"
    visit("Root", tree["Root"])
    print "}"

Redirect the output to a file and then convert to a graph as follows:
"C:\Program Files (x86)\Graphviz 2.28\bin\dot.exe" -Tpng -ooutput.png dot_input.txt

Ye cannae change the laws of physics

Montgomery "Scotty" ScottOne of the problems with curating chemical databases of small organic molecules is filtering out bogus connection tables from legitimate molecules. One aspect of this challenge has been termed [Ed: “I” have termed] EOCWR standing for “Explodes On Contact With Reality”.

An interesting class of broken molecules, that are often overlooked by reasonableness filters, are those that defy the standard model of physics. In this view of matter, atoms are composed of whole numbers of protons, neutrons and electrons. To paraphrase, Democritus “all that exists are groupings of protons, neutrons and electrons in empty space, all else is opinion”. Naturally, under this model the formal charge on an atom cannot exceed its atomic number. Whilst [Ed: “While”] an arbitrary number of electrons may be associated with an atom, it cannot have fewer electrons than zero; hence the positive charge is bounded by the number of protons in the nucleus. However, many cheminformatics file formats record the formal charge rather than the electron count leading to the ability to represent impossible molecules. Checking for these is relatively trivial and allows compounds such as [H+2] or [C+7] to be flagged as erroneous.

Another example of testing for EOCWR is the work of Dr Jonathan Goodman and colleagues at the University of Cambridge on the challenges on embedding alkanes in three dimensions (here and here). Their work explains that in some molecules, although all atomic valences are reasonable, steric crowding would produce sufficient strain that the molecule would fall apart. Hence although graph theoretically an sp3 carbon may have four neighbours that each itself has three additional neighbours (all unique) in reality there is an energetic upper bound of 10 second neighbours in alkanes. As above, checking the number of second (and third) neighbours of an atom provides a convenient and efficient way of distinguishing plausible molecules from the artifacts of erroneous molecule processing (termed “robochemistry” by NCBI PubChem’s Evan Bolton).

Image credit: Graham Lees (Tram Painter on Flickr)

Using PubChem’s REST interface from Python

Version 1.0 of PubChem’s REST interface (PUG REST) went live on Sept 13. The tutorial gives examples of use, while full details are in the spec.

The following code shows illustrates basic usage from Python. If you want to simultaneously query large numbers of molecules, or do a substructure search, you need to perform a query “in the background” using the ListKey approach shown by SubstructureToCids:

import os
import sys
import time

import urllib2

def getresult(url):
        connection = urllib2.urlopen(url)
    except urllib2.HTTPError, e:
        return ""

def NameToSmiles(name):
    return getresult("" % name)

def NameToCid(name):
    return getresult("" % name)

def CidToSmiles(cid):
    return getresult("" % cid)

def SubstructureToCids(smiles):
    result = getresult("" % smiles)
    if not result:
        return []
    listkey = ""
    for line in result.split("\n"):
        if line.find("ListKey")>=0:
            listkey = line.split("ListKey>")[1][:-2]
    assert listkey

    url = "" % listkey
    delta = 2
    while True:
        time.sleep(delta) # ...perchance to dream
        if delta < 8:
            delta += 1
        result = getresult(url).split("\n")
        if result[0] != "Your request is running":
    return result

if __name__ == "__main__":
    print NameToSmiles("Gleevec")
    print NameToCid("Gleevec")
    print CidToSmiles("123596")
    print SubstructureToCids("CC1=C(C=C(C=C1)NC(=O)C2=CC=C(C=C2)CN3CCN(CC3)C)NC4=NC=CC(=N4)C5=CN=CC=C5")

Lazy File Reading with mmap

Will Fix Defects For CaffeineThe example source code often provided with cheminformatics libraries to demonstrate the available functionality often try to address two conflicting requirements. On the one hand, these attempt to be pedagogical teaching examples, explaining how to perform a task often to a new or inexperienced user. On the other, these also attempt to address realistic problems and have utility in their own right. Alas in some cases, the best or most efficient way to do something is not the easiest to understand, so efficiency is sacrificed for clarity, or the other way around.

A recent example of this dilema arose for an example program to demonstrate how NextMove Software’s CaffeineFix library can be used with jQuery to provide efficient autocompletion/suggestion in chemical search engine text boxes, such as provided by Google. A minor implementation detail of this approach was the reading of CaffeineFix’s binary .cfx file format from disk.

In C or C++, a simple way to perform this shown by LoadDictionary1 below, that uses the standard I/O library to read in the dictionary in sequential chunks of 64 Kbytes at a time.

#include <stdlib.h>
#include <stdio.h>

unsigned char *LoadDictionary1(const char *fname) {
  unsigned int alloc = 65536
  unsigned char *result;
  unsigned int len = 0;

  FILE *fp = fopen(fname,"rb");
  if (!fp) {
    fprintf(stderr,"Error: Unable to read dictionary file %s\n",fname);
    return (unsigned char*)0;

  result = (unsigned char*)malloc(65536);
  if (!result) {
    fprintf(stderr,"Error: Unable to allocate 64K buffer!\n");
    return (unsigned char*)0;

  for(;;) {
    unsigned int chunk = fread(result+len,1,65536,fp);
    len += chunk;

    if (chunk != 65536) break;
    alloc += 65536;
    result = (unsigned char*)realloc(result,alloc);
    if (!result) {
      fprintf(stderr,"Error: Unable to reallocate %uK buffer!\n",alloc>>10);
  return result;

This is functional, highly portable between systems and even works with streams, such as stdin.

Ultimately, however, the most efficient implementation for reading binary CaffeineFix dictionaries on UNIX-like operating systems is to use “mmap“. Memory mapping lets the operating system decide where to return contents of a file, but with a clever trick of virtual memory that the data is (typically) only read from disk when a memory location is accessed.

#include <fcntl.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mman.h>
static unsigned char *LoadDictionary4(const char *fname)
  unsigned char *result;
  unsigned int len;
  struct stat buf;

  int fd = open(fname,O_RDONLY);
  if (fd < 0) {
    fprintf(stderr,"Error: Unable to read dictionary file %s\n",fname);
    return (unsigned char*)0;

  if (fstat(fd,&buf) < 0) {
    fprintf(stderr,"Error: Unable to determine file size\n");
    return (unsigned char*)0;

  len = (unsigned int)buf.st_size;
  result = (unsigned char*)mmap(0,len,PROT_READ,MAP_FILE|MAP_PRIVATE,fd,0);
  if (!result) fprintf(stderr,"Error: Unable to memory map dictionary!\n");
  return result;

For the example use-case described above, autocompletion using CaffeineFix typically only (randomly) accesses a small fraction of the binary file. In practice, this means that only a small part need be read from disk.

To quantify the performance advantage, we consider autocompleting the text “bisul” using NCBI pubchem’s synonym dictionary of 46.6 million compound names. The original 976 Mbyte ASCII dictionary file is very efficiently encoded as a 718 Mbyte binary CFX file. Using LoadDictionary1 above, autocompletion of “bisul” to return 10 possible completions takes 68 seconds on my Apple laptop, over 99.9% spent in file I/O. Using memory mapped file I/O with LoadDictionary4, the same task takes under half a second.

In practice, if using a persistent process on a web server there’s very little difference between approaches, as once the dictionary has been read into memory suggestions can be made as fast as a user types. However, for applications where this start-up time is an issue, memory mapping is clearly superior.

As a final word, I should point out that memory mapped file I/O is also available to programmers on Microsoft Windows using the MapViewOfFile APIs, and even to Java programmers using the method to return a ByteBuffer. [Ed: It’s also available in the Python standard library!]

Image credit: James Nash (aka Cirrus) on Flickr

Identifying suspect InChIs in Wikipedia Chemboxes using Chemical Name to Structure

Wikipedia is a highly useful source of Chemistry and also of chemical nomenclature. A limitation in chemical name to structure software, such as OPSIN, is that trivial names that are similar to systematic names may be misinterpreted if the program has never encountered the trivial names. The nature of Wikipedia means that the most important chemicals and hence the most prevalent trivial names are included so surely Wikipedia would be a great resource to look for name to InChI relationships where the name to structure software was at fault?

I used Matthew Gamble’s code for extracting chemboxes as RDF to quickly grab the contents of all the current chemboxes. From the output of this tool it was simple to get the name/InChI pairs. As I was interested in trivial names I used the title of each Wikipedia page as the input for name to structure.

430 cases were flagged up for a range of reasons: ring/chain tautomerism, intentionally underspecified names, under or over specification of stereochemistry and of course the type of error I was expected. However there were also a significant number of cases where the InChI clearly described a different compound. Upon investigation for the records I’ve corrected so far the root cause appeared to be an incorrect reference to ChemSpider. This then allows script assisted updates to pull in inappropriate InChIs/SMILES.

Example of a previously incorrect page.

Increasing the precision of identificiation of these incorrect name/strucutral identifiers pairs should be possible if the IUPAC names were used as input…

Visualising a hierarchy of substructures

Given a set of chemical structures as SMILES, how can you visualise the substructure/superstructure relationship between them?

For example, the following picture shows the relationship between members of a set of structures containing several benzene derivatives and monosaccharides:
This was created using the following Python script, which iteratively looks for the structure that matches the largest number of molecules in the set, building up a tree as it does so. The output is the tree in a form suitable for depiction using Graphviz’s dot program. The Python script uses Open Babel, but could easily be adapted for other toolkits.

Using Python for batch conversion of ChemSketch files to Mol files

There are several tools out there for batch conversion of chemical file formats. However, you may not have access to those tools or else a particular file format may not be supported. Sometimes your only option is to open each file in the original software, and export it as a more useful file format. When dealing with a large number of files, this can be quite tedious or just impossible.

Recently, faced with the challenge of converting 100s of ChemSketch files to Mol files, myself and Daniel looked up the literature (i.e. googled the web) and found that Rajarshi Guha has described how to automate ChemDraw using the Python module WATSUP. Unfortunately this no longer seems to be available. Rich Apodaca has described an alternative approach using AppleScript but this does not work so well on Windows.

Instead we used a Python library, pywinauto, to automate the process of using the ChemSketch GUI to File/Open the ChemSketch file and File/Export a Mol file. The script is shown below. The trickiest part is choosing the delays so that the process runs as fast as possible without failing (due to a ChemSketch operation taking longer than expected):