^{†}

^{†}

^{*}

Edited by: Nicholas J. Tustison, University of Virginia, USA

Reviewed by: Robert C. Cannon, Textensor Limited, UK; Richard Beare, Monash University, Australia

*Correspondence: Sean G. Megason, Department of Systems Biology, Harvard Medical School, 200 Longwood Avenue, WAB 536, Boston, MA 02134, USA e-mail:

This article was submitted to the journal Frontiers in Neuroinformatics.

†These authors have contributed equally to this work as first authors.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

In the last decade, level-set methods have been actively developed for applications in image registration, segmentation, tracking, and reconstruction. However, the development of a wide variety of level-set PDEs and their numerical discretization schemes, coupled with hybrid combinations of PDE terms, stopping criteria, and reinitialization strategies, has created a software logistics problem. In the absence of an integrative design, current toolkits support only specific types of level-set implementations which restrict future algorithm development since extensions require significant code duplication and effort. In the new NIH/NLM Insight Toolkit (ITK) v4 architecture, we implemented a level-set software design that is flexible to different numerical (continuous, discrete, and sparse) and grid representations (point, mesh, and image-based). Given that a generic PDE is a summation of different terms, we used a set of linked containers to which level-set terms can be added or deleted at any point in the evolution process. This container-based approach allows the user to explore and customize terms in the level-set equation at compile-time in a flexible manner. The framework is optimized so that repeated computations of common intensity functions (e.g., gradient and Hessians) across multiple terms is eliminated. The framework further enables the evolution of multiple level-sets for multi-object segmentation and processing of large datasets. For doing so, we restrict level-set domains to subsets of the image domain and use multithreading strategies to process groups of subdomains or level-set functions. Users can also select from a variety of reinitialization policies and stopping criteria. Finally, we developed a visualization framework that shows the evolution of a level-set in real-time to help guide algorithm development and parameter optimization. We demonstrate the power of our new framework using confocal microscopy images of cells in a developing zebrafish embryo.

The automated identification of anatomical structures found in medical and microscopy images is an important step in any imaging-based quantitative analysis pipeline. Large variations in image quality arising from differences in acquisition protocols, anisotropic point-spread functions, and image noise complicate the task of automated image analysis tools. To overcome the disadvantages associated with simple heuristic methods, a class of methods for contour evolution known as geometric active contour models, or level-sets, have been actively developed (Osher and Sethian, _{0}(

The function

In general, there are two main classes of the level-set methodology that arise from using a variational approach that minimizes an energy functional: (i) edge-based and (ii) region-based methods.

The goal of edge-based approaches is to evolve the contour until it finds an object edge where its speed is gradually reduced to 0. An edge-based level-set based on mean curvature motion is given by (Caselles et al., _{1} and _{2} are the maximum and minimum values of the image gradient (|∇

The first (curvature) and last (propagation) terms in Equation 4 approach zero at the object boundary. The middle (advection) term ensures that contour motion is always directed toward the boundary.

Many additional variants of edge-based concepts have been published. An abstract representation common to all edge-based partial differential equation (PDE) is as follows:

Region-based level-sets segment the image into objects based on region statistics (rather than just object edges) of intensity, texture, or color values. For example, the region mean intensity for foreground (_{1}) and background (_{2}) is a popularly used statistic for defining an energy functional

Minimizing the energy functional using the standard gradient descent (or steepest descent) method leads to the following gradient-flow equation:
_{e} is the smoothed Dirac-Delta function, and μ and ν regularize the curve length and foreground area, respectively. To account for image inhomogeneities and large gradients that may be present in an image, piecewise-smooth extensions were proposed in Tsai et al. (_{1} and _{2} and the energy functional was defined in terms of the fit with respect to these functions. Nevertheless, the final equation form is similar to Equation 7.

Within the last decade, region-based techniques, as well as graph-partitioning-based active contours (Sumengen et al.,

One of the primary disadvantages with using level-set methods for image segmentation is that they are slow and memory-intensive. The level-set function is typically discretized on the entire image grid to hold floating-point values (Figure

During the evolution of the level-set function, the differential movement of different level-curves can cause the function to develop gradients that are too steep and/or too flat. This negatively impacts the numerical stability in subsequent iterations leading to unpredictable outcomes. Therefore, one needs to perform a distance reinitialization step at the end of every few iterations, such that the zero level-curve location remains unchanged. One way to perform such an initialization is to evolve the PDE to a steady-state according to Sussman et al. (_{0} represents the level-set function before the reinitialization. The end result will be a signed distance function to the interface (ϕ_{0} = 0). Another approach is to solve the Eikonal equation using fast-marching methods (Peng et al.,

In this method, the signed distance function is used to fix the level-set function in a narrow band around the zero curve as boundary conditions. Fast marching methods are then used to solve the Eikonal equation.

In another development, Li

The addition of this energy followed by the steepest descent flow leads to the addition of a new term

Similar to the reinitialization problem, the stopping criterion also presents several choices to the user. The evolution of a level-set function is typically halted by a threshold set on the number of iterations (

In biomedical and biological image analysis applications, one is often interested in segmenting more than a single object (of the same or different kind) from a given image. This especially happens when the objects to be segmented are adjacent to each other and the delineation of one object automatically affects the neighboring object. For example, in MRI imagery, white and gray matter regions in the brain have a common interface. Moreover, spatial constraints often specify relationships between classes of level-sets. For example, microscopy images depict thousands of cells in terms of their plasma membranes, nuclear membranes, organelles, and proteins. Here, the nucleus and organelles are always contained within the membrane. In such situations, concurrently interacting and evolving level-sets provide optimal and efficient image segmentation solutions. Multi-object methods (Dufour et al.,

The goal of the new inheritance framework is to permit the implementation of a variety of level-set methods with minimal code-duplication and allow end-users to explore and customize the method to their data. The new framework is designed to be modular, and therefore, reduces the effort needed to expand the level-set code base. An overview of the level-set algorithm with components of our framework is provided in Figure

While the level-set function has typically been implemented on underlying image grids, it also theoretically extends to unstructured grids or surface meshes and continuous representations. In the popular ITK v3 framework, for example, an image representation was used throughout the code which impedes the development of mesh-based level-set segmentation methods. Additionally, in a continuous representation, the level-set function is not discretized on a grid but instead represented analytically in terms of linear combination of base functions [e.g., radial basis functions (Bernard et al.,

Our code is heavily templated to allow for a great deal of flexibility in level-set methods in an efficient manner. In our new framework, we define an abstract level-set function base class (

where TInput defines the input type where the level-set function will be evaluated, VDimension is dimension of the input space, TOutput is the returned type when evaluating the level-set function (for the general case when it is not a scalar), TDomain is the discretization of the level-set function (e.g., ImageBase or QuadEdgeMesh). In the case of an image representation, this specializes as only the first three parameters and inherits from the

While this definition accounts for an image representation (continuous or discrete), we further specialize into a discrete image representation with TInput as

All of the above level-set function classes implement specific member functions for returning the level-set value [ϕ(^{2} ϕ), Laplacian (ϕ_{xx} + ϕ_{yy}), gradient norm (|∇ϕ|), and mean curvature (

The level-set quantities are stored as specific instances of a templated class

The discrete image representation

For the convenience of end-users, adaptors for converting from binary images to level-set function objects were developed in ITK v4. The supplied binary image could be the output of a preprocessing segmentation pipeline. We consider the dense level-set function as well as the three types of sparse-field representations: (1) Whitaker, (2) Shi, and (3) Malcolm. The adaptor base class is templated over the input image type and the output level-set function type. It contains member variables for the input image and the computed level-set function representation and a pure virtual member function for computing the level-set function.

The derived class

The large computational costs associated with level-set methods is a particularly severe problem in multi-level set implementations. Typically, depending on the computing environment, one cannot evolve more than tens of level-set functions without running into memory problems. To address this problem, our new framework handles level-set functions defined on a subset of the input image domain (Ω). The interaction among level-set functions is limited to those functions whose domains overlap (Figures

The output of the helper class is to define an image of list pixel types. Each list holds the ids of level-set functions active at that pixel. By using another helper class (

As an illustrative example,Figure

The level-set equation is a weighted sum of terms. Each term is a simple function of the level-set (e.g., gradient, Hessian, divergence, etc.), the input image (intensity, gradient, etc.), or involving the other level-set functions concurrently evolving in the system. As mentioned before, level-set computations occur repeatedly in each term and therefore values can be computed, cached, and stored in the level-set object and shared among the various terms. The term base class implements functions [

Different types of terms arising from edge-based and region-based level-set methods such as the propagation, Laplacian, advection, curvature, and region-based terms described in Equations 4 and 7 derive directly from

Consider a concurrently evolving system of level-set functions {ϕ_{1}, ϕ_{2}, …, ϕ_{M}} with input image _{i}_{ij}.

The set of terms that constitute each equation may vary across all the

In the case of multiple level-set functions, a special container class (

Using the partial template specialization mechanism, the class

For each level-set function defined by its identifier (

The class is templated over the input image type and the level-set container type. The class is not derived further since there is no further specialization of the individual terms. In a given term container, the end-user can combine edge-based and region-based terms for customizing the level-set method which is yet another aspect of flexibility enabled by our framework.

For a system of level-set equations, each equation (_{i}

The class

Dense and sparse-strategies are different in terms of how the update buffers are setup. Sparse methods visit only the narrow band layers while the dense method visits every pixel in the level-set domain. Therefore, as before we use partial template specialization to specialize the implementation depending on the level-set representation.

Figure

The concurrent segmentation of all cells and corresponding cell nuclei in ITKv3 is intractable given that each level-set function would occupy the same domain as the image and 946 level-set functions would not fit in memory. Secondly, the level-sets would have to be segmented independently one function at a time. This would lead to a running time of many hours. Thirdly, level-set functions cannot interact with each other. Figure

In ITKv4, domain partitioning happens automatically and the user exactly specifies the number of threads to use. We provided running times for the procedure (≈21 min).

The iterations in the level-set method end when the functions finish evolving (converge) or when they satisfy some user-defined criterion. A specialized stopping criterion class (

In order to enable communication among filters and with graphical user-interfaces (GUIs), we used the ^{*})

To enable the real-time visualization and recording of level-set function evolution, we provide a pipeline linking our level-set framework to The Visualization Toolkit (VTK). The process of visualization consists of first mapping a level-set function to a VTK dataset. The base class for converting a

Given the dense and three sparse-field representations of level-set functions implemented (Figures

We chose to implement our framework using C++ in the NIH/NLM Insight Toolkit (ITK) v4 because of its solid software engineering practices, permissive license, community-support, and its popularity in the biomedical imaging community. The level-set classes, examples, and associated tests have been integrated into the ITKv4 toolkit. Detailed instructions for downloading ITK are available at:

After compiling and installing ITK, users can navigate to the downloaded source directory and find a total of 123 level-set classes at the following location:

Additional examples of the level-set API used to generate the results for this article (Figure

Thus, our framework addresses a long-standing challenge in the image analysis community for developing a state-of-the art ITKv4 segmentation and analysis toolkit. Our framework achieves the following goals:

Current medical and microscopic imaging modalities feature rich datasets that requires the continuous development of new algorithms and performance optimization strategies to handle large data. In our framework, we combined the two major classes of edge-based and region-based methods seamlessly which should promote the development and usage of hybrid methods. We refactored sparse-field implementations to enable the addition of Whitaker, Malcolm, and Shi methods for improving performance. We enabled the implementation of multi level-set methods to handle multiple objects in images and also implemented domain partitioning techniques for improved performance. Thus, our framework supports large datasets, incorporates all the major level-set technologies, and enables the research and development of new ideas in the near future.

Our implementation allows the user to customize all aspects of the level-set method. The user can optimize the number of level-sets to use and specify how they interact. Users can specify the exact terms in the level-set update equations and choose the reinitialization and stopping criteria to use after each iteration. The large number of customization options available means that the user is better equipped to handle the varying challenges presented by different biomedical datasets. Users can also seed or terminate level-set functions, or manipulate the terms and term weights at the beginning of each iteration in the system. This paves the way for the exploration and development of real-time level-set systems in the future.

The three sparse-field strategies significantly improve level-set performance on large datasets. Domain partitioning significantly reduces the memory utilized per level-set especially when the objects occupy a small portion of the large input space. By caching and reusing level-set quantities in the term calculations (see Section 3.2), we further improve performance. We also implemented multithreaded execution of the iterations to take advantage of multicore processor systems. Multithreading is accomplished by allocating threads to iterate on (i) different regions in the domain partitioned image or (ii) across individual level-sets. This choice is decided at run-time depending on the maximum number of threads available to the system, the number of level-sets initialized by the user, and the number of domain partitions present.

Currently, there is no other comprehensive framework or toolkit similar to the ITK framework in terms of implementing a multitude of level-set technologies in a scalable and efficient manner. However, upon further testing we do find that ITKv3 performs 2–2.5× faster for very basic segmentation using just one thread. This speed difference appears to result from the use of image iterators in ITKv3 compared with GetPixel() and SetPixel() methods in ITKv4 so future work could see a speed improvement by transitioning to image iterators. In the future, we plan to develop user-friendly APIs for developers familiar with ITKv3 and GUIs that will benefit new users. For the purpose of cell segmentation in microscopy, we have already begun developing a image analysis GUI called GoFigure2 (

Arnaud Gelas designed the software. Kishore R. Mosaliganti and Arnaud Gelas implemented and tested the level-set software in C++. Arnaud Gelas developed the level-set visualization code. Kishore R. Mosaliganti and Arnaud Gelas ported the code into the official NIH ITK repository and continue to provide technical support. Kishore R. Mosaliganti analyzed the microscopy data reported in the article. Kishore R. Mosaliganti and Sean G. Megason wrote the manuscript. All authors discussed the manuscript and contributed to writing.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

We thank Dr. Luis Ibáñez (Kitware Inc), Dr. Luca Antiga (Orobix Inc), Dr. Ross Whitaker (University of Utah), and Dr. Jim Miller (GE USA), for comments on the level-set design. We thank Dr. Matthew McCormick (Kitware Inc) for technical assistance in improving our C++ code and for participating in code reviews prior to submission into the Insight Toolkit v4. We thank members of Megason lab for comments and support, Mr. Dante D'India for fish care, and Ms. Suzanne Mosaliganti for proof-reading and editing the manuscript. This work was supported by NLM Contracts (RFQ: NLM-10-029-CYC), NLM Contract (RFQ: NLM-10-029-CYC) and NIDCD R01 DC010791-01 that were awarded to Sean G. Megason. Kishore R. Mosaliganti is also supported by the Eunice Kennedy Shriver National Institute Of Child Health & Human Development of the National Institutes of Health under Award Number K25HD071969. The content in this article is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

The Supplementary Material for this article can be found online at: