Changes between Version 7 and Version 8 of TaxonomyLearningImplementation


Ignore:
Timestamp:
06/08/10 18:48:24 (9 years ago)
Author:
grimnes
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • TaxonomyLearningImplementation

    v7 v8  
    44The relations between terms (broader/narrower terms, hierarchic relations, related terms), and possible alternative spellings, can be extracted from existing data sources. For example, DBPedia and linked open data can be used. 
    55 
    6 == How to use it == 
    7 The component is run from command-line. It is a separate application written in Java, Python, and bash. To run the application on a unix/linux server, use the command 
    8 {{{ 
    9 runAll.sh 
    10 }}} 
    11 It will read the configuration file in {{{config2}}}, to configure it to run on a dataset called "mydataset" use 
    12 {{{ 
    13 NAME="mydataset" 
    14 }}} 
     6== First time Taxonomy Learning == 
     7The taxonomy learning is separate from Drupal and is written in Java.  
     8For details on how to actually download and run it please refer to:  
     9http://organik.opendfki.de/wiki/AdministratorTutorial/TaxonomyLearning 
    1510 
    16 This will analyze text files inside the data-file-folder {{{./data/mydataset}}}. The algorithm will do the following: 
    17  * read all files in the data-file-folder and extract relevant terms using !eu.organik.ontolearn.TermExtractor  
    18  * match the terms with the public open linked data base dbpedia using !eu.organik.ontolearn.DBpediaTagger 
    19  * rank and filter the terms according to relevancy 
    20  * call !eu.organik.ontolearn.DBpediaLookUp to create a '''SKOS output file''' 
    21  
    22 The result is a [wiki:SKOS] thesaurus which contains interesting terms which can now be used as tags. 
    23 Take the resulting SKOS thesaurus from {{{results/mydataset/taxonomy_skos.rdf}}} and upload it to OrganiK using Drupal/TaxonomyImport.  
     11This will analyze text from the nodes in your drupal installation and the result is a [wiki:SKOS] thesaurus which contains interesting terms which can now be used as tags. 
     12Take the resulting SKOS thesaurus and upload it to OrganiK using Drupal/TaxonomyImport.  
    2413After uploading, the terms are available in the system. 
    2514 
    2615== How it works == 
    27 '''TODO: this is outdated. We do more. Gunnar Grimnes will update this page according to the documentation in our recent paper and give some links to the source. 
    28 ''' 
    2916 
    30 Before going into details: the term-extraction used by this component is described at ContentAnalyserImplementation#TermExtractor . 
    3117 
    32 As input for the bootstrapping process all content existing in OrganiK (or imported into OrganiK) is taken. The text sources are analysed with natural language processing techniques and post-processed to create a taxonomy.  
    33 Overall, taxonomy learning includes the following steps: ''term extraction'',  
    34 ''finding synonyms'', ''identification of concepts'', and placing concepts into a ''hierarchy''.  
    3518 
    36 In the first step, all sentences in the text are parsed to identify noun phrases ("NP").  
    37 [http://opennlp.sf.net OpenNLP] is used to generate parse trees of the sentences.  
    38 In the next step, for each noun phrase, matching DBPedia resources are identified. Here, DBPedia is used as ''background knowledge'' to evaluate the usefulness of found strings in a taxonomy. The string representation of a noun phrase is matched with labels from DBPedia. Partial matches are included. The assumption behind this approach is that noun phrases mentioned on DBPedia are known to a broad audience and can serve as taxonomy terms for the SME in question.   
    39 For example the noun phrase ''Julius Caesar's Conquests'' matches both ''Julius Caesar'' and ''Military career of Julius Caesar'' on DBPedia. 
    40 Obviously, this fuzzy matching may find too many candidates in the background knowledge. The list of candidate is ranked by what we refer to as ''DBPedia index''. 
     19 * The input text is either read from files in a folder or extracted from Drupal Nodes. HTML tags etc. are stripped.  
    4120 
    42 {{{ 
    43 #!latex 
    44 $\displaystyle (DBPedia index)=\frac{(number of common words for matching DBPedia label)}{(number of words in the concepts)} $ 
    45 }}} 
    46 The rank for found resources on DBPedia is a weighted combination of a total of three scores: 
     21 * Using a machine-learning based noun-phrase chunker, the ''noun-chunks'' are extracted from the text, these are the candidate terms. They are counted and only the most frequent are kept. We use [http://opennlp.sf.net OpenNLP] for this - and have trained our own noun-chunking model for german. (source:trunk/OrganikOntologyLearning/src/eu/organik/ontolearn/NPExtractor.java) This same method is used for the term-suggestion servlet described at ContentAnalyserImplementation#TermExtractor . 
    4722 
    48   
    49  *  percentile rank of term  
    50   
    51  *  inverse length (1/(number of words in the term))  
    52   
    53  *  DBPedia match index (how well a term matches with any dpedia term) 
     23 * The candidate terms are matched with the public open linked project DbPedia. (source:trunk/OrganikOntologyLearning/src/eu/organik/ontolearn/dbpedia/DBpediaTagger.java) Here, DBPedia is used as ''background knowledge'' to evaluate the usefulness of found strings in a taxonomy. The string representation of a noun phrase is matched with labels from DBPedia. Partial matches are included. The assumption behind this approach is that noun phrases mentioned on DBPedia are known to a broad audience and can serve as taxonomy terms for the SME in question.   
    5424 
    55 Only terms with a score higher than a threshold-K are chosen as concepts to be included in the taxonomy.  
    56 Result of the ranking step is to identify concepts from DBPedia that possibly match keywords found in the text. 
     25 * The terms with dbpedia matches are ranked according to their frequency combined with how well the term matched the dbpedia article label.  
    5726 
    58 After ranking, terms are ''positioned in a hierarchy'' based on narrower and broader relations in DBPedia. These existing SKOS relations\footnote{Broader and narrower are properties defined in the SKOS vocabulary and express taxonomic structure.} are already building a hierarchy in DBPedia, which can be adopted for the SME case. 
    59 All broader and narrower relations {{{ {r} }}} of found DBPedia resources are collected and kept in a buffer.  
    60 When one relation {{{ r }}} connects two noun phrases found in the overall SME corpus, {{{ r }}} is a good candidate for a hierarchy and is adopted with a high rank.  
    61 Also indirect relations can be used, when two hierarchical relations {{{ {r1, r2} }}} contain two noun phrases from the corpus and an additional new resource {{{ n }}}. The new resource {{{ n }}} can be either a broader topic of both phrases, or an intermediate topic in a larger hierarchy.  
    62 For example, when both the resource ''Julius Caesar'' and ''Mark Antony'' are mentioned, the broader resource ''Ancient Roman generals'' of both will be included in the resulting taxonomy. 
     27 * Terms with a score lower than some threshold K are discarded. (this can be set with the commandline option --taxonomyRankingThreshold) 
     28 
     29 * After ranking and filtering, terms are ''positioned in a hierarchy'' based on narrower and broader relations in DBPedia. These existing SKOS relations\footnote{Broader and narrower are properties defined in the SKOS vocabulary and express taxonomic structure.} are already building a hierarchy in DBPedia, which can be adopted for the SME case. (source:trunk/OrganikOntologyLearning/src/eu/organik/ontolearn/dbpedia/DBpediaTaxonomyBuilder.java) 
     30 
     31 * Now ''implicit terms'' are added - i.e. terms that are connected to two or more of our terms in DbPedia but were not found in our original texts. For example, when both the resource ''Julius Caesar'' and ''Mark Antony'' are mentioned, the broader resource ''Ancient Roman generals'' of both will be included in the resulting taxonomy. The new term can be either a broader topic of both phrases, or an intermediate topic in a larger hierarchy.  
     32 
     33 * Now some amount of ''spreading activation'' is done, this helps us focus on parts of the found taxonomy that are well connected. The degree of spreading activation can be adjusted with the options: spreadingChildFlow, spreadingParentFlow, spreadingRelatedFlow, and spreadingIterations. 
     34 
     35 * Finally, a  ''SKOS'' file is output with the top ranked terms, the number of terms to output can be configured with the ''--taxonomyNoTerms'' option. 
     36 
     37 
     38 
     39For a more technical description of the process, see our recent paper at i-Semantics'10: Gunnar Aastrand Grimnes, Remzi Celebi and Leo Sauermann, Using Linked Open Data to bootstrap corporate Knowledge Management in the OrganiK Project. 
     40 
     41 
    6342 
    6443== Sourcecode ==