Ontology Evaluation In Use: Exploring Methods and Metrics of Construction, Use, and Re-Use

An ICBO informal meeting


Abstract We propose an informal meeting on the topic of Ontology Evaluation In Use: Exploring Methods and Metrics of Construction, Use, and Re-Use. The informal meeting has two goals. First is a presentation/workshop aimed at raising the awareness of an approach to the objective evaluation of an ontology (or part of an ontology) for use in a specific application (as described in a use case). The approach presented is currently underdevelopment at RPI and seeking input from and adoption by the broader community. The second goal is the planning of a three day workshop to further identify stakeholder needs and develop the framework further to meet those needs.

The informal meeting contributes to the field of biomedical ontology in that it furthers the current understanding and practice in ontology evaluation and re-use, provides a framework for modular ontology construction, and identifies the specific challenges the community needs to address going forward. The workshop aims to engage the biomedical community in the discussion and development of new metrics for ontology evaluation by engaging the participants in discussion of the challenges they face or faced in their current or past projects. This workshop will help establish a foundation for objective and quantitative evaluation of the applicability of a particular ontology for a particular use case, thus further enabling the sharing, re-using, and reviewing biomedical ontologies.

This workshop is an ideal forum for this discussion since there are a growing number of major incentives for work in health and wellbeing underway, and it the need is overdue for an implementation, which will need to rely upon a number of focused and coordinated efforts.

Proposed format The first part of the meeting will introduce the generalized framework for ontology evaluation in use and depending on time could include an exercise demonstrating the approach. If time does not permit, this will be limited to a demonstration. The second part of the meeting will be a discussion resulting in a set of concrete next steps and assignments to those willing and interested in contributing, along with the outline of a workshop proposal to further advance the development of ontology evaluation tools.

  • Target audience: Ontology developers and users of biomedical ontology.
  • Level of the event: Introductory (the framework will be introduced during this session)
  • Required background knowledge from the participants: It is assumed that the participant has a working knowledge of ontology, but does not have to be an expert in ontology development. It will be assumed and required that participants have looked at and/or used at least one ontology in some application, or aiming at this and want to understand how to evaluate an ontology for their intended purpose. It is desirable if participants bring an ontology and use case they wish to evaluate and work through during the workshop.


Please contact the proposers for further information:

Joanne S. Luciano, PhD
email: jluciano [at] rpi.edu

Dr. Joanne S. Luciano is a Research Associate Professor at the Tetherless World Constellation, Rensselaer Polytechnic Institute. For nearly two decades, Dr. Luciano has been pioneering the use of computational and knowledge-based approaches in biomedical applications. She has organized and co-organized many workshops and symposiums, including the BioPathways Consortium (for approximately a decade), in which she co-authored BioPAX standard, an OWL-based pathway ontology for bio-research. She was instrumental in the creation of the W3C Health Care and Life Science SIG, and is conference co-chair for Conference on Clinical Semantics in Health Care and Life Science (CSHALS), as the International Society for Computational Biology (ISCB) scientific representative since its inception. Dr. Luciano leads the RPI hands-on tutorial for CSHALS and has taught graduate courses in semantic and advanced semantic technologies at Rensselaer and is currently leading the research effort in ontology evaluation.
Note: The current framework originated from research carried out at MITRE lead by Luciano and included Cohen as a collaborator. The work was internally funded and proposed to NIST, who intended to fund the work going forward. A number of factors unrelated to the merit of the work kept it from advancing at that time. Research and work has resumed now, some initial funding is in place and further funding being sought. We aim to offer collaboration opportunities to participants of the meeting.

Kevin B. Cohen, PhD
email: Kevin.Cohen [at] gmail.com

Dr. Kevin B. Cohen leads the Biomedical Text Mining Group in Larry Hunter's Computational Bioscience Program at the University of Colorado Health Sciences Center and is an adjunct assistant professor in the Department of Linguistics at the University of Colorado Boulder. Kevin spent the first half of his adult life doing medical/technical work, and the second half doing linguistics and software development. Dr. Cohen was originally trained in phonetics, but most of his linguistic work has been in the area of natural language processing. Kevin collaborated on the initial ontology evaluation work that was done by Luciano at MITRE and has extensive experience in evaluation of NLP from his work in the BioCreAtIvE (Critical Assessment of Information Extraction systems in Biology) challenge evaluation. The BioCreAtIvE challenge evaluation consists of a community-wide effort for evaluating text mining and information extraction systems applied to the biological domain.

Local Organizers

Stefan Schulz
Catalina Martínez
Markus Kreuzthaler

Institute for Medical
Informatics, Statistics
and Documentation

Medical University of Graz

Auenbruggerplatz 2
8036 Graz, Austria

stefan.schulz [at] medunigraz.at


Feb 1: ICBO and FOIS workshops and tutorials announced

FOIS deadline extended to Feb 12, 2012 (strict)

Jan 25: ICBO workshops and tutorials online

Jan 6: FOIS call for posters open.

Dec 8: Keynote Speakers Announced.

Maintainer: Stefan Schulz