Text Mining for a Worthy Cause

I recently received an e-mail from the charity “jeans for genes” introducing me to “black bone disease“, a rare genetic disease without a cure. It is more formally known as “Alkaptonuria” (OMIM entry) and is a defect in the homogentisate 1,2-dioxygenase gene (HGD) which leads to a toxic build-up of homogentisic acid in the blood, causing the symptoms of the disease.

Interestingly a re-purposed herbicide, nitisinone, is currently being investigated as a possible treatment for the disease based on its previous re-purposing as a therapy in related genetic disorder, Type 1 Tyrosinemia.

The story starts in 1977 when a researcher in California observed that relatively few weeds were growing under the bottlebrush (Callistemon) plants in his backyard. Analytical chemistry of the soil fractions revealed the active compound to be the natural product Leptospermone. Traditional ligand based optimization of this compound led to the effective herbicides mesotrione (Syngenta’s Callisto) and nitisinone being synthesized and tested in 1984, with the first patents on this class of herbicides appearing in 1986 (e.g. US 4780127). At the point these patents were filed/granted, the mechanism of action and protein target weren’t yet known, although they were experimentally proven to be toxic to plants but harmless to mammals. Much later it was discovered that these compounds worked by inhibiting the enzyme 4-hydroxyphenylpyruvate dioxygenase (HPPD) which blocks the synthesis of chlorophyll and leads to “bleaching” and eventual plant death.

It is the role that HPPD plays in human metabolism that make these herbicides so interesting as therapeutic agents. The pathway diagram below describes the five enzymatic steps (arrows) in the degradation metabolism of tyrosine.

Defects in these various enzymes responsible for each step lead to a number of related diseases: Problems with the first step, tyrosine-transaminase, cause type 2 tyrosinemia; the second step, p-Hydroxylphenylpyruvate-dioxygenase (HPPD) is our herbicide target for which defects cause type 3 tyrosinemia; step three, homogentisate dioxygenase (HGD) causes alkaptonuria (aka black bone disease); and step 5, 4-fumaryl-acetoacetate hydrolase causes type 1 tyrosinemia.

In the case of type 1 tyrosinemia, it was noticed that those patients with active HPPD had a more severe form of the disease, so it was hypothesized that a HPPD inhibitor may be beneficial. At the time Zeneca worked on both pharmaceuticals and crop protection and were able to evaluate their proven-safe herbicide nitisinone directly in the clinic. In what seems incredible by the standards of today’s pharmaceutical pipelines, their US 5550165 patent filing describes the administration to, and recovery of, sick infants and children, where it is now more usual for a drug candidate to spend years in phase I, II and III clinical trials after a patent is granted before it gets approved by the FDA.

HPPD inhibitors can be anticipated to treat alkaptonuria by much the same mechanism:
By blocking the formation of the toxic metabolite homogentisate, and causing tyrosine
to be metabolised via alternate routes.

One of the goals of modern text mining is to automatically discover links such as those between the above two patents, US4780127 and US5550165. Unfortunately, a range of technical issues complicate the process: In common with many pharmaceutical patent filings, the drug target is not known or not mentioned, so it is necessary to identify and annotate compound classes or modes of action such as “kinase inhibitor”, “beta-blocker”, “herbicide” or “antibiotic”. The large number of synonyms and typographical variants of enzyme and disease names requires the use of synonym dictionaries or ontologies to recognize that “tyrosine transaminase” is the same entity as “tyrosine aminotransferase” is the same as “EC“. Finally, as revealed by the mistake “tyosinemia” in the title of the above US 5550165, documents in real life frequently contain spelling errors, making it impossible to find the most relevant documents when searching for a keyword like “tyrosinemia” (without automatic spelling correction).

These are exactly the types of challenges our LeadMine software attempts to tackle.

Building and climbing a Chemical Ladder

A recent post by Jake Vanderplas described Word Ladders, and gave me an idea. A Word Ladder is a game developed by Lewis Carroll that involves converting one word into another one letter at a time, while passing through only valid words. For example, to convert MAD into HAT, a valid word ladder would be MAD->MAT->HAT or MAD->HAD->HAT.

Here I propose the term Chemical Ladder for a Word Ladder that is restricted to chemical names. For example, try converting from Barbamate to Carbazide in 4 steps only using valid chemical names. Or from Anginine to Arsonite.

So, how did I come up with these examples? Well, NextMove Software’s CaffeineFix can do chemical spelling correction based on a dictionary (or grammar). The spelling suggestions provided by CaffeineFix are substitutions, deletions and insertions, but if we consider just the 1-letter substitutions, this is exactly the transformation needed to build a word ladder, e.g.

>>> from caffeinefix import CaffeineFix
>>> cf = CaffeineFix("mydictionary.cfx")
>>> list(cf.suggest("azite"))
["azote", "azine"]

To begin with I downloaded the list of synonyms from PubChem, and filtered to remove database identifiers and various other cruft. I compiled these into a CaffeineFix dictionary, and edited the suggest method to just return substitutions (suggest_substitutions in the code below). The code shown below then uses the suggested substitutions for each chemical name to create a graph that I visualised to identify Chemical Ladders (see example below). A longer version of the code could be written to identify the Chemical Ladders more automatically.

Just in case any highly-respected and discerning chemistry society wants to include Chemical Ladders in its weekly or monthly magazine, I’ve decided not to include the full output of the program in this blogpost, apart from the image above. What do you think? Could this be the next ChemDoku?

from collections import defaultdict

from caffeinefix import CaffeineFix

def difference(a, b):
    for i, (d, e) in enumerate(zip(a,b)):
        if d != e:
            return i

def nearest(name):
    suggestions = list(cf.suggest_substitutions(name))
    return (name, suggestions)

replacements = [("0", "zero"), ("%", "PERCENT"), ("+", "PLUS"), ("4", "FOUR"),
                ("7", "SEVEN"), ("9", "NINE"), ("6", "SIX"), ("8", "EIGHT"),
                (")", "RBRACKET"), ("'", "APOSTROPHE"), ("@", "AT"),
                ("}", "RBRACKETB"), ("{", "LBRACKETB"), (":", "COLON"),
                ("/", "FSLASH"), (".", "PERIOD"), ("&", "AMPERSAND"),
                ("^", "CIRCUMFLEX"), ("[", "LBRACKETC"), ("]", "RBRACKETC"),
                ("|", "PIPE"), (";", "SEMICOLON")]

def fix(name):
    name = name.replace("-", "_").replace("1", "one").replace("2", "two").replace("3", "three").replace("5", "five").replace(",", "_").replace("?", "Q").replace("(", "LB").replace(" ", "SPACE")
    for x, y in replacements:
        name = name.replace(x, y)
    return name

if __name__ == "__main__":
    cf = CaffeineFix(r"C:\Tools\LargeData\PubChem_Synonyms\pubchem.cfx")
    names = [x.strip() for x in open(r"C:\Tools\LargeData\PubChem_Synonyms\lowercase_sorted_uniq.txt", "r") if x.rstrip()
             and len(x) == 10]
    results = map(nearest, names)

    output = open("wordladder_10.txt", "w")
    output.write("graph graphname {\n")

    for x, y in results:
        if len(y) > 1:
            collate = defaultdict(list)
            for w in y:
                collate[difference(x, w)].append(w)
            if len(collate) > 1:
                for v in collate.values():
                    output.write('%s [label="%s"];\n' % (fix(x), x))
                    for z in v:
                        output.write("%s -- %s\n" % (fix(x), fix(z)))
                        output.write('%s [label="%s"];\n' % (fix(z), z))


Note: A couple of people have asked why are there two edges for only some of the connections in the graph. This would be the case if I retained all of the original edges, as if A is a spelling correction of B, then B is a spelling correction of A. However, since a word ladder can only exist if a node in the graph has at least two connections, I filter out all those cases where a node has only a single connection (otherwise you end up with a lot of ‘word ladders’ composed of just two words). So, if I have A->B, B->(A, C), C->(B, D), D->C, then A->B and D->C will be removed, and the graph will be A-B=C-D.

Molecular Half-life: The light that burns twice as bright burns for half as long

Trinity 007In a previous post I described how to use cheminformatics to determine whether or not a compound was radioactive by checking for unstable atomic nuclei.  Alas, such a binary yes-or-no classification is a clumsy tool for eliminating dubious structures, or even identifying non-drugs in chemical databases.  For example, bismuth is unfortunate enough to have no perfectly stable isotopes, making drugs such as GSK’s Pylorid (ranitidine bismuth citrate) technically but negligibly radioactive, even though bismuth’s half-life is 1.9×1019 years, or over a billion times the life of the universe.  Likewise, although [7H] has been observed experimentally, its half-life of 23 yoctoseconds (1×10-24 seconds) makes it use impractical for traditional drug discovery.

As an interesting aside, measurements of half-lives form an interesting exception to usual SI units, with small values being in fractions of a second (milliseconds, microseconds, nanoseconds), intermediates values being in minutes, hours and days, and large values being in multiples of years (kiloyears, megayears, gigayears and so on).  Converting units to normalized form, for example times in terms of seconds, is all part of scientific computing.

To refine the filtering of plausible/reasonable neutron counts we propose the use of molecular half-life, rather than binary categories such radioactive vs. stable or experimentally observed vs. purely hypothetical.  The one subtlety with this definition is that a molecule’s half-life is determined from the half-lives of all of the unstable atoms it contains.  As hinted in the title, a molecule with two copies of the same unstable nuclide, has a half-life half that of an individual atom.  In general, the formula for a molecule’s half-life is 1/Σi{1/t½(i)}, or the reciprocal of sum of the reciprocals of constituent atomic half-lives.

At NextMove Software, we currently use the nuclide half-lives tabulated by the Nubase2003 database (downloadable as an ASCII file). The resulting “half-life validity” check can be used to identify dubious structures in chemical databases using a suitable threshold. For example, the most suspicious isotopic specification in NCBI’s PubChem database belongs to CID 11635947. This is a structure deposited by NextBio that contains an erroneous [8C]. Although [8C] has been experimentally observed (and PubChem should be congratulated for containing no nuclides that haven’t been observed), it has an impressively low half-life of only two zeptoseconds (2×10-19 seconds).

A more reasonable threshold might be around the 1223 second (~20 minute) half-life of [11C], which legitimately appears as the least stable compound in Accelrys’ MDDR database. 11C is used as a radiotracer in Positron Emission Tomography (PET), where compounds have to be used within about three half-lives of their manufacture. When filtering compound screening collections, threshold half-lives much higher might be reasonable.

My final observation is that even more accurate calculations of molecular half-life is possible by taking into account the influence of chemical environment on atomic half-life.
For example, metallic Berylium-7, [7Be], has a different half-live to covalently bound Berylium-7, such as in Berylium-7 fluoride, F[7Be]F, or Berylium-7 oxide, [7Be]=O, and fully ionizied Rhenium-187, [187Re+75] has a half-life of 33 years, significantly lower than that of metallic Rhenium-187, [187Re], which has a half-life of 41 gigayears (41×109 years).

Image credit: Ed Siasoco (aka SC Fiasco) on Flickr

Radioactivity — It’s in the air for you and me

Radioactive Materials AreaRadioactivity, discovered by Madame Curie, is the process by which the nucleus of an “unstable” atom decays to a different form.  As mentioned in a previous blog post, atoms are composed of protons, neutrons and electrons; protons are easy to handle in cheminformatics, electrons are incredibly difficult and here we discuss the neutrons.

Checking whether the number of neutrons specified with an atom (the isotope) is plausible and reasonable is a non-trivial challenge.  At the most simplistic level, many cheminformatics applications and file formats ignore isotopes altogether and assume every atom has the default terrestrial isotopic composition/abundance as prescribed/recommended by IUPAC. The next level of sophistication is to treat the atomic symbols “D” and “T” as corresponding to deuterium, [2H], and tritium, [3H] respectively.

More usually, such as with MDL’s SD files or SMILES, allow the optional specification of a mass number (number of nucleons, i.e. protons+neutrons, in the nucleus).  If not specified, the element again has the IUPAC recommended composition.  A common misunderstanding with these semantics is that [12C] is not the same as [C].  Although terrestrial carbon is predominantly carbon-12 (98.89% by the latest 2009 recommendations) the presence of trace amounts of [13C] keep these distinct.  Having said that, 22 elements do have a unique isotope officially used to determine their atomic weight and hence [4Be], [9F], [11Na], [13Al], [15P], [21Sc], [25Mn], [27Co], [33As], [39Y], [41Nb], [45Rh], [53I], [55Cs], [59Pr], [65Tb], [67Ho], [69Tm], [79Au], [83Bi], [90Th] and [91Pa] may legitimately be canonicalised without the isotopic specification, i.e. [Be], F, [Na] and so on.

The most advanced cheminformatics file formats, such as Perkin Elmer Informatics’ (formerly CambridgeSoft’s) ChemDraw CDX and CDXML file format can even specify enrichment and depletion in specific isotopes. 

Unfortunately, having a specified isotope is often confused with being radioactive.  For example, RSC’s ChemSpider abuses the international icon for radioactivity to actually mean “Non-standard isotope”, though this is clearly stated.  This is because testing for a specified mass number is relatively easy, with many toolkits supporting the SMARTS semantics that [!0*] matches any specified isotope.  Although this is useful for identifying compounds whose isotopes need to be checked, it doesn’t correspond to radioactivity.  For example, deuterium, [2H] has a specified isotope but isn’t radioactive, whilst uranium, [U], even without a specified isotope is radioactive.

To address this I’ll describe how to ascertain whether a compound is radioactive, a useful descriptor especially when dealing with the “Health & Safety” parts of a pharmaceutical company.  A molecule is radioactive if any of its atoms is radioactive, and an atom is radioactive if it isn’t stable.  If an isotope isn’t specified, the element must have at least one stable isotope to be considered stable (these are the elements from hydrogen, [#1], to lead, [#82], with the exceptions of technetium [#43] and prometium [#61]), otherwise the specified isotope must correspond to one of the 255 known stable nuclides.  Hence, SMARTS pattern is_stable corresponds to [0#1,1#1,2#1,0#2,3#2,4#2,...].  Using De Morgan’s laws this atom expression can be negated to produce is_radioactive as [!0,!#1;!1,!#1;!2,!#2;…].

The complete SMARTS pattern for is_radioactive is shown below:


Image credit: LimeTech on Flickr

Patently wrong – Tracing the origin of an unusual molecule in PubChem

Greg Landrum of Novartis posted a link to PubChem structure CID60140829 on Google Plus, with the comment:

This one is from a patent with the title “Apparatus and method for encoding and decoding block low density parity check codes with a variable coding rate”. I bet it’s the result of an overly zealous (and insufficiently error checked) image->structure conversion.

A good guess, but not quite right…

First of all, let’s look at the PubChem image for the deposited structure, SID143481705 (see right). Hmmm…

These structures are from the US Patent 20050283708 via SCRIPDB. For each patent the USPTO makes available a PDF, and very usefully for chemists, the ChemDraw files associated with the patent (along with MOL, and TIFF). NextMove Software’s PatFetch (bundled with LeadMine) makes it easy to extract the corresponding PNG, CDX and MOL files for a particular structure (it runs in a web browser, and you just click on an image to obtain the CDX file). In this case, the image corresponding to the structure is as follows:
Don’t ask me what it is, but I can confirm that it’s not a crossword.

But here’s the thing. If you download the CDX file and open it in ChemDraw…you get exactly this image. 🙂 In other words, the good people at the USPTO appear to use ChemDraw as a generic drawing tool, and in particular, seem to favour carbon-carbon bonds over the actual box or line tools. Actually, now that I see how useful a grid of carbon-carbon bonds can be to create a nice table, I think I might dump Excel for good too.

Visualising a hierarchy of substructures Part II

In an earlier post, I described a simple procedure to generate a hierarchy of substructures, and depicted the hierarchy with GraphViz. Pat Walters at Vertex realised that it is possible to add images as node labels in GraphViz and updated the script so that the image includes the actual chemical depictions (see below). He has also adapted the script to use OpenEye’s OEChem.

Update (8/11/2012): Wolf-D. Ihlenfeldt has implemented this with the Cactvs Toolkit and in addition has shown how to integrate the graph display with Knime.

#!/usr/bin/env python

import copy
import pickle
import os

from openeye.oechem import *
from openeye.oedepict import *

def mol2image(mol,imageFileName,width=200,height=200):
    clearcoords = True
    suppressH   = False
    opts = OE2DMolDisplayOptions(width, height, OEScale_AutoScale)
    itf = OEInterface()
    OESetup2DMolDisplayOptions(opts, itf)
    OEPrepareDepiction(mol, clearcoords, suppressH)
    disp = OE2DMolDisplay(mol, opts)
    ofs = oeofstream(imageFileName)
    name,ext = os.path.splitext(imageFileName)

def create_tree(structures):
    tree = {"Root":{}}
    stack = [(tree["Root"], structures.keys())]

    while len(stack) > 0:
        leaf, subset = stack.pop()

        max_matches = ("", [])
        for name in subset:
            smiles = structures[name]
            smarts = OESubSearch(smiles)
            matches = []
            for nameb in subset:
                if nameb != name:
                    molb = OEGraphMol()
                    if smarts.SingleMatch(molb):
            if len(matches) >= len(max_matches[1]):
                max_matches = (name, matches)
        if False: # Debug statement
            print max_matches

        for name in [max_matches[0]] + max_matches[1]:
        leaf[max_matches[0]] = {}
        if len(subset) > 0:
            stack.append( (leaf, subset) )
        if len(max_matches[1]) > 0:
            stack.append( (leaf[max_matches[0]], copy.deepcopy(max_matches[1])))

    with open("tmp.pickle", "w") as f:
        pickle.dump(tree, f)

def fix(name):
    return name.replace("-", "_").replace("1", "one").replace("2", "two").replace("3", "three").replace("5", "five").replace(",", "_")

def visit(name, leafdict):
    for x, y in leafdict.iteritems():
        if name != "Root":
            print ' %s -> %s;' % (name,x)
        print ' %s [label=<<TABLE><TR><TD><IMG SRC="%s/%s.png"/></TD></TR></TABLE>>];' % (x,os.getcwd(),x)
        visit(x, y)

if __name__ == "__main__":
    structureDict = dict([('001', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@@H](O)[CH]1(O)'),
                       ('002', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@H](O)[CH]1(O)'),
                       ('003', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@@H](O)[C@H]1(O)'),
                       ('004', 'C(O)[C@@H](O1)[C@@H](O)[C@H](O)[C@@H](O)[C@@H]1(O)'),
                       ('005', 'C(O)[CH](O1)[CH](O)[CH](O)[CH](O)[CH]1(O)'),
                       ('006', 'c1ccccc1'),
                       ('007', 'c1ccccc1Br'),
                       ('008', 'c1ccc(Br)cc1Br'),
                       ('009', 'c1cccc(Br)c1Br'),
                       ('010', 'c1c(Br)cc(Br)cc1Br')])

    for k in sorted(structureDict.keys()):
        imageFileName = "%s.png" % (k)
        smi = structureDict[k]
        mol = OEGraphMol()

    with open("tmp.pickle", "r") as f:
        tree = pickle.load(f)
    print "digraph graphname {"
    print "node [shape=plaintext];"
    visit("Root", tree["Root"])
    print "}"

Redirect the output to a file and then convert to a graph as follows:
"C:\Program Files (x86)\Graphviz 2.28\bin\dot.exe" -Tpng -ooutput.png dot_input.txt

Ye cannae change the laws of physics

Montgomery "Scotty" ScottOne of the problems with curating chemical databases of small organic molecules is filtering out bogus connection tables from legitimate molecules. One aspect of this challenge has been termed [Ed: “I” have termed] EOCWR standing for “Explodes On Contact With Reality”.

An interesting class of broken molecules, that are often overlooked by reasonableness filters, are those that defy the standard model of physics. In this view of matter, atoms are composed of whole numbers of protons, neutrons and electrons. To paraphrase, Democritus “all that exists are groupings of protons, neutrons and electrons in empty space, all else is opinion”. Naturally, under this model the formal charge on an atom cannot exceed its atomic number. Whilst [Ed: “While”] an arbitrary number of electrons may be associated with an atom, it cannot have fewer electrons than zero; hence the positive charge is bounded by the number of protons in the nucleus. However, many cheminformatics file formats record the formal charge rather than the electron count leading to the ability to represent impossible molecules. Checking for these is relatively trivial and allows compounds such as [H+2] or [C+7] to be flagged as erroneous.

Another example of testing for EOCWR is the work of Dr Jonathan Goodman and colleagues at the University of Cambridge on the challenges on embedding alkanes in three dimensions (here and here). Their work explains that in some molecules, although all atomic valences are reasonable, steric crowding would produce sufficient strain that the molecule would fall apart. Hence although graph theoretically an sp3 carbon may have four neighbours that each itself has three additional neighbours (all unique) in reality there is an energetic upper bound of 10 second neighbours in alkanes. As above, checking the number of second (and third) neighbours of an atom provides a convenient and efficient way of distinguishing plausible molecules from the artifacts of erroneous molecule processing (termed “robochemistry” by NCBI PubChem’s Evan Bolton).

Image credit: Graham Lees (Tram Painter on Flickr)

Using PubChem’s REST interface from Python

Version 1.0 of PubChem’s REST interface (PUG REST) went live on Sept 13. The tutorial gives examples of use, while full details are in the spec.

The following code shows illustrates basic usage from Python. If you want to simultaneously query large numbers of molecules, or do a substructure search, you need to perform a query “in the background” using the ListKey approach shown by SubstructureToCids:

import os
import sys
import time

import urllib2

def getresult(url):
        connection = urllib2.urlopen(url)
    except urllib2.HTTPError, e:
        return ""
        return connection.read().rstrip()

def NameToSmiles(name):
    return getresult("http://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/%s/property/IsomericSMILES/TXT" % name)

def NameToCid(name):
    return getresult("http://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/%s/cids/TXT" % name)

def CidToSmiles(cid):
    return getresult("http://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/cid/%s/property/IsomericSMILES/TXT" % cid)

def SubstructureToCids(smiles):
    result = getresult("http://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/substructure/smiles/%s/XML" % smiles)
    if not result:
        return []
    listkey = ""
    for line in result.split("\n"):
        if line.find("ListKey")>=0:
            listkey = line.split("ListKey>")[1][:-2]
    assert listkey

    url = "http://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/listkey/%s/cids/TXT" % listkey
    delta = 2
    while True:
        time.sleep(delta) # ...perchance to dream
        if delta < 8:
            delta += 1
        result = getresult(url).split("\n")
        if result[0] != "Your request is running":
    return result

if __name__ == "__main__":
    print NameToSmiles("Gleevec")
    print NameToCid("Gleevec")
    print CidToSmiles("123596")
    print SubstructureToCids("CC1=C(C=C(C=C1)NC(=O)C2=CC=C(C=C2)CN3CCN(CC3)C)NC4=NC=CC(=N4)C5=CN=CC=C5")

Lazy File Reading with mmap

Will Fix Defects For CaffeineThe example source code often provided with cheminformatics libraries to demonstrate the available functionality often try to address two conflicting requirements. On the one hand, these attempt to be pedagogical teaching examples, explaining how to perform a task often to a new or inexperienced user. On the other, these also attempt to address realistic problems and have utility in their own right. Alas in some cases, the best or most efficient way to do something is not the easiest to understand, so efficiency is sacrificed for clarity, or the other way around.

A recent example of this dilema arose for an example program to demonstrate how NextMove Software’s CaffeineFix library can be used with jQuery to provide efficient autocompletion/suggestion in chemical search engine text boxes, such as provided by Google. A minor implementation detail of this approach was the reading of CaffeineFix’s binary .cfx file format from disk.

In C or C++, a simple way to perform this shown by LoadDictionary1 below, that uses the standard I/O library to read in the dictionary in sequential chunks of 64 Kbytes at a time.

#include <stdlib.h>
#include <stdio.h>

unsigned char *LoadDictionary1(const char *fname) {
  unsigned int alloc = 65536
  unsigned char *result;
  unsigned int len = 0;

  FILE *fp = fopen(fname,"rb");
  if (!fp) {
    fprintf(stderr,"Error: Unable to read dictionary file %s\n",fname);
    return (unsigned char*)0;

  result = (unsigned char*)malloc(65536);
  if (!result) {
    fprintf(stderr,"Error: Unable to allocate 64K buffer!\n");
    return (unsigned char*)0;

  for(;;) {
    unsigned int chunk = fread(result+len,1,65536,fp);
    len += chunk;

    if (chunk != 65536) break;
    alloc += 65536;
    result = (unsigned char*)realloc(result,alloc);
    if (!result) {
      fprintf(stderr,"Error: Unable to reallocate %uK buffer!\n",alloc>>10);
  return result;

This is functional, highly portable between systems and even works with streams, such as stdin.

Ultimately, however, the most efficient implementation for reading binary CaffeineFix dictionaries on UNIX-like operating systems is to use “mmap“. Memory mapping lets the operating system decide where to return contents of a file, but with a clever trick of virtual memory that the data is (typically) only read from disk when a memory location is accessed.

#include <fcntl.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mman.h>
static unsigned char *LoadDictionary4(const char *fname)
  unsigned char *result;
  unsigned int len;
  struct stat buf;

  int fd = open(fname,O_RDONLY);
  if (fd < 0) {
    fprintf(stderr,"Error: Unable to read dictionary file %s\n",fname);
    return (unsigned char*)0;

  if (fstat(fd,&buf) < 0) {
    fprintf(stderr,"Error: Unable to determine file size\n");
    return (unsigned char*)0;

  len = (unsigned int)buf.st_size;
  result = (unsigned char*)mmap(0,len,PROT_READ,MAP_FILE|MAP_PRIVATE,fd,0);
  if (!result) fprintf(stderr,"Error: Unable to memory map dictionary!\n");
  return result;

For the example use-case described above, autocompletion using CaffeineFix typically only (randomly) accesses a small fraction of the binary file. In practice, this means that only a small part need be read from disk.

To quantify the performance advantage, we consider autocompleting the text “bisul” using NCBI pubchem’s synonym dictionary of 46.6 million compound names. The original 976 Mbyte ASCII dictionary file is very efficiently encoded as a 718 Mbyte binary CFX file. Using LoadDictionary1 above, autocompletion of “bisul” to return 10 possible completions takes 68 seconds on my Apple laptop, over 99.9% spent in file I/O. Using memory mapped file I/O with LoadDictionary4, the same task takes under half a second.

In practice, if using a persistent process on a web server there’s very little difference between approaches, as once the dictionary has been read into memory suggestions can be made as fast as a user types. However, for applications where this start-up time is an issue, memory mapping is clearly superior.

As a final word, I should point out that memory mapped file I/O is also available to programmers on Microsoft Windows using the MapViewOfFile APIs, and even to Java programmers using the FileChannel.map method to return a ByteBuffer. [Ed: It’s also available in the Python standard library!]

Image credit: James Nash (aka Cirrus) on Flickr

Identifying suspect InChIs in Wikipedia Chemboxes using Chemical Name to Structure

Wikipedia is a highly useful source of Chemistry and also of chemical nomenclature. A limitation in chemical name to structure software, such as OPSIN, is that trivial names that are similar to systematic names may be misinterpreted if the program has never encountered the trivial names. The nature of Wikipedia means that the most important chemicals and hence the most prevalent trivial names are included so surely Wikipedia would be a great resource to look for name to InChI relationships where the name to structure software was at fault?

I used Matthew Gamble’s code for extracting chemboxes as RDF to quickly grab the contents of all the current chemboxes. From the output of this tool it was simple to get the name/InChI pairs. As I was interested in trivial names I used the title of each Wikipedia page as the input for name to structure.

430 cases were flagged up for a range of reasons: ring/chain tautomerism, intentionally underspecified names, under or over specification of stereochemistry and of course the type of error I was expected. However there were also a significant number of cases where the InChI clearly described a different compound. Upon investigation for the records I’ve corrected so far the root cause appeared to be an incorrect reference to ChemSpider. This then allows script assisted updates to pull in inappropriate InChIs/SMILES.

Example of a previously incorrect page.

Increasing the precision of identificiation of these incorrect name/strucutral identifiers pairs should be possible if the IUPAC names were used as input…