{"id":2746,"date":"2018-04-09T13:17:44","date_gmt":"2018-04-09T12:17:44","guid":{"rendered":"https:\/\/nextmovesoftware.com\/blog\/?p=2746"},"modified":"2020-08-25T15:28:33","modified_gmt":"2020-08-25T14:28:33","slug":"textmining-pubmed-abstracts-with-leadmine","status":"publish","type":"post","link":"https:\/\/nextmovesoftware.com\/blog\/2018\/04\/09\/textmining-pubmed-abstracts-with-leadmine\/","title":{"rendered":"Textmining PubMed Abstracts with LeadMine"},"content":{"rendered":"

The US National Library of Medicine provide<\/a> an annual baseline of PubMed Abstracts freely available for download, along with daily updates throughout the year. You can download the baseline with a command such as the following, if you have wget available:<\/p>\n

wget --mirror --accept \"*.xml.gz\" ftp:\/\/ftp.ncbi.nlm.nih.gov\/pubmed\/baseline\/\n<\/pre>\n
Once downloaded, you will end up with close to 100 gzipped xml files, each one containing a large number of abstracts along with bibliographic data.<\/p>\n
While it is possible to use the LeadMine library and an XML reader to textmine these any way one wants, it is convenient to use LeadMine’s command-line application to do the textmining if possible, as this has built-in reporting of results, makes use of multiple processors, and well, you don’t need to write any Java to use it. However, without transforming these data somehow, the command-line application is not going to work very well as (a) it process each entire XML file in one go, with resulting large memory usage and a lack of correspondence between a particular PMID and its results, (b) using more than one processor simply exacerbates the memory problem, and (c) while LeadMine handles XML without problems, it will inevitably end up textmining information that is not in the abstract (e.g. part of a journal name).<\/p>\n
Fortunately, it is not difficult to transform the data into a more easily digestible form. For each gzipped XML file, the Python script below generates a zip containing a large number of small XML files, each one corresponding to a single PubMed abstract. Bibliographic information is included in XML attributes so that the only text that will be textmined is that of the title and abstract (indicated by T and A in the results below). Once the script is run, LeadMine can be used to textmine as shown below. Here I focus on diseases and ClinicalTrials.gov (NCT) numbers; note that while PubMed provide manually curated entries for both of these in the original XML, they are missing from the most recent abstracts (presumably due to a time lag).<\/p>\n
java -jar leadmine-3.12.jar -c diseases_trails.cfg -tsv -t 12 -R D:\\PubMedAbstracts\\zipfiles > diseases_trials.txt\n<\/pre>\n
Fifteen minutes later (on my machine), I get an output file that includes the following where the PMID (and version) appears in the first column:<\/p>\n
DocName BegIndex        EndIndex        SectionType     EntityType      PossiblyCorrectedText   EntityText      CorrectionDistance      ResolvedForm\n29170069.1\t1272\t1279\tA\tDisease\tphobias\tphobias\t0\tD010698\n29170069.1\t1290\t1307\tA\tDisease\tanxiety disorders\tanxiety disorders\t0\tD001008\n29170072.1\t325\t358\tA\tDisease\tExocrine pancreatic insufficiency\tExocrine pancreatic insufficiency\t0\tD010188\n29170073.1\t1856\t1860\tA\tDisease\tpain\tpain\t0\tD010146\n29170073.1\t2087\t2098\tA\tTrial\tNCT02683707\tNCT02683707\t0\t\n29170074.1\t334\t349\tA\tDisease\tcystic fibrosis\tcystic fibrosis\t0\tD003550\n29170075.1\t127\t146\tT\tDisease\tdepressive symptoms\tdepressive symptoms\t0\tD003866\n29170075.1\t419\t438\tA\tDisease\tdepressive symptoms\tdepressive symptoms\t0\tD003866\n29170075.1\t476\t495\tA\tDisease\tdepressive symptoms\tdepressive symptoms\t0\tD003866\n29170075.1\t1579\t1598\tA\tDisease\tdepressive symptoms\tdepressive symptoms\t0\tD003866\n29170075.1\t2191\t2202\tA\tTrial\tNCT02860741\tNCT02860741\t0\t\n29170076.1\t198\t221\tT\tDisease\tend-stage renal disease\tend-stage renal disease\t0\tD007676\n29170076.1\t240\t262\tA\tDisease\tCardiovascular disease\tCardiovascular disease\t0\tD002318\n29170076.1\t320\t342\tA\tDisease\tchronic kidney disease\tchronic kidney disease\t0\tD051436\n29170076.1\t485\t507\tA\tDisease\tvascular calcification\tvascular calcification\t0\tD061205\n29170076.1\t583\t588\tA\tDisease\ttumor\ttumor\t0\tD009369\n29170076.1\t589\t597\tA\tDisease\tnecrosis\tnecrosis\t0\tD009336\n29170076.1\t765\t788\tA\tDisease\tend-stage renal disease\tend-stage renal disease\t0\tD007676\n<\/pre>\n
Python script<\/b><\/p>\n