^{1}

^{2}

^{2}

^{1}

^{*}

^{1}

^{2}

Edited by: Trygve B. Leergaard, University of Oslo, Norway

Reviewed by: Boudewijn Lelieveldt, Leiden University Medical Center, Netherlands; Rembrandt Bakker, Radboud University Nijmegen, Netherlands

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Histological brain slices are widely used in neuroscience to study the anatomical organization of neural circuits. Systematic and accurate comparisons of anatomical data from multiple brains, especially from different studies, can benefit tremendously from registering histological slices onto a common reference atlas. Most existing methods rely on an initial reconstruction of the volume before registering it to a reference atlas. Because these slices are prone to distortions during the sectioning process and often sectioned with non-standard angles, reconstruction is challenging and often inaccurate. Here we describe a framework that maps each slice to its corresponding plane in the Allen Mouse Brain Atlas (2015) to build a plane-wise mapping and then perform 2D nonrigid registration to build a pixel-wise mapping. We use the L2 norm of the histogram of oriented gradients difference of two patches as the similarity metric for both steps and a Markov random field formulation that incorporates tissue coherency to compute the nonrigid registration. To fix significantly distorted regions that are misshaped or much smaller than the control grids, we train a context-aggregation network to segment and warp them to their corresponding regions with thin plate spline. We have shown that our method generates results comparable to an expert neuroscientist and is significantly better than reconstruction-first approaches. Code and sample dataset are available at

Neuroanatomical studies have traditionally been performed in histological sections, followed by manually annotating data based on histological stains in comparison with a brain atlas. For large-scale analyses, this procedure is labor-intensive, time-consuming, variable and sometimes subjective. It is crucial to standardize and digitalize anatomical information to allow data from multiple brains to be compared in the same reference brain. To this end, detailed anatomical brain reference atlases have been established for both human and animal model studies (Lein et al.,

Histological images suffer from multiple artifacts.

In this work, we introduce a method to register a sequence of coronal histological sections of mouse brain to the grayscale Nissl volume of the Allen Mouse Brain Atlas (2015) (ABA) (Lein et al.,

Mapping a sequence of histological mouse brain slices to the atlas volume of ABA (horizontal view of coronal sections). Left side shows a real histological stack. Right hand side is the ABA.

The problem of mapping a sequence of histological slices to a reference brain has been well studied (Pichat et al.,

To improve overall reconstruction results and reduce error propagation, some methods align each slice with multiple neighboring images. For example, Ju et al. (

Now the main challenge is finding the corresponding plane for each slice (Yang et al.,

To avoid these issues, we concurrently estimate the sectioning angle difference and the best matching planes in the atlas volume for each slice. This approach requires us to find the best matching slice in the reference before applying nonrigid deformations. Since the resulting slice comparisons are noisy, we aggregate information from all slices and use information about the brain's structure to find the best match. Our method does not have a reconstruction step, therefore completely eliminating the z-shift problem. The details of our method are given in the next section.

After each matching reference slice has been determined, we need to perform a 2D registration between it and its matching histological slice. Free Form Deformation (FFD) (Rueckert et al.,

Instead, we find that the L2 norm of histogram of oriented gradients (HOG) (Dalal and Triggs,

Our strategy makes the maximum use of the reference volume, successfully deals with the non-standard sectioning angle problem, preserves the curvature of the object—eliminating the z-shift problem (Adler et al.,

This section describes in more detail how we find the sectioning angle difference and the best matching plane in the reference volume for each histological slice (Section 2.1) and nonrigidly register each slice to the corresponding sectioning plane in the atlas (Section 2.2).

Both in the 2D to 3D localization and the 2D nonrigid registration steps, a relatively sensitive and quantitative similarity measure is needed. The state of art is to use normalized mutual information (Jefferis et al.,

When searching for a better metric, we also wanted to find one that would work well for our images. Our image characteristics include:

Staining reagent and microscopic setting difference can cause direct comparison of intensities to be not useful. Even worse, due to the non-uniformly applied staining reagent, some slices are unevenly illuminated.

Nissl-stained (Glaser and Van der Loos,

Sparsely scattered or densely populated cell bodies make images low-contrast and noisy. Many descriptors that work with man-made scenes do not perform well.

Distortions caused by brain's elasticity require metrics that work even when the two images are slightly distorted from each other. This distortion tolerance also allows it to compare a distorted histological slice to a reference slice.

Even though the newest Nissl volume of the Allen Mouse Brain Atlas (2015) is smoother than the Allen Mouse Brain Atlas (2011), it is still Nissl-stained volume constructed from physically-sectioned mouse brain slices and is not perfectly aligned. So an ideal metric should be somewhat tolerant to this imperfect alignment.

As shown by Dalal and Triggs (

Since histological slices are often cut with near constant angles with a microtome (Gibson et al.,

The following sections give our dynamic programming formulation to solve the alignment problem to determine the slicing angle and a simple method to increase sensitivity to angular shifts.

The best cutting angle is the angle that maximizes the similarity between all histological slices and their corresponding best matching slices in the atlas. Because in-plane rotation can be fixed, we only consider rotation angle α about the superior-interior (y) axis and β about the left-right (x) axis. To solve the problem, we first find the best matching slice for each experimental slice given a potential cutting angle. The problem can be represented as follows: Let I_{1…N} with spacing s_{E} be the experimental slice sequence, and _{A} be an isotropic atlas with voxel dimension s_{A}, defined on the domain Ω. After rotating the atlas with potential best rotation _{αβ}, we reslice the rotated atlas into coronal slices and re-index them as atlas slice sequence J_{1…M}. Using the L2 norm of HOG differences described in Section 2, we aim to find a mapping that matches each slice in I to a slice in J which minimizes the overall difference.

Taking into account potential compression along the longitudinal axis, slice quality variation, and intersubject variation, we formulate the problem with a single subset A of all slices I, where A is an ordered selection of 1…_{A} and s_{E}). A is chosen to span the full sequence while avoiding damaged slices.

We formulate this slice mapping and difference minimization problem as a dynamic program. Let I_{A} be the ordered selection of experimental slices, and let J be the resliced atlas sequence ordered from the same direction along the longitudinal axis and spacing s_{A}. The cost, ^{th} slice has to be mapped to the ^{th} slice:

where

where _{i} is the original index of the ith slice in the selected sequence and

We denote the best ^{*}. The best intermediate steps are saved by updating the three-dimensional array

_{αβ}(

where

The cost of mapping all slices in A to resectioned slices in J with atlas rotated by αβ is given by ^{*}).

After running this dynamic program with different sectioning angles we should be able to directly choose the angle that gives minimum cost score to be the best cutting angle. However, since HOG is relatively insensitive to local distortions and each slice is slightly distorted, when summing up all the costs we also sum up a lot of noise. Therefore when the angle is very close to the true sectioning angle, the difference among neighboring angles is not substantial. To improve our robustness, we use a different approach. This approach also predicts how we should adjust the rotation and prevents exhaustive searching in the previous approach.

Biological structures change quickly along the posterior-anterior direction. It is not hard to tell if an experimental brain is sectioned with a different angle than the atlas volume, even if the angle deviation is only several degrees, because structures that appear in the same slice in the atlas will be in different slices in the experimental slice sequence. For example, if the left side of the brain is tilted to be more anterior, on average the right hand half coronal brain slice will appear to be more posterior to the left half. Thus if we match the left and right half slices of an experimental brain separately to the atlas, we will see that the slice number of the matching slices of the left half brain will be on average higher than that of the right half. Based on this idea, we use matching slice index differences of half brains to determine if a rotation angle best fixes the cutting angle difference between the experimental brain and the atlas.

Because mouse brains have left-right symmetry, the rotation angle α about the superior-interior (y) axis tends to be around zero. The rotation angle β about the left-right (x) axis tends to be larger because the mouse brain is not flat at the bottom and can easily be set tilted on the microtome plate. Here we use the determination of angle β as an example; the flowchart is shown in Figure

Flow chart for determining sectioning angle about the left-right (x) axis. In the matching half slices step, atlas is stretched for better illustration.

To find the best rotation angle β about the x axis, we solve the slice mapping problem with the method described in Section 2.1.1 on the upper half slices and the lower half slices respectively. We take the index difference between the optimal mapping given by:

where M_{upper} and M_{lower} denote binary masks to apply to both experimental image and resliced atlas image to include only half of a slice. Positive

After finding the optimal rotation

After the 2D to 3D registration, we map all the experimental slices to their computed corresponding slices in the optimally rotated atlas with the deformable registration to build a pixel-wise mapping from the 2D slice sequence to the atlas volume. Let an experimental image g that is globally transformed to the same coordinates of its corresponding slice be the target image, and its best matching slice f be the source image, where Ω⊂ℤ^{2} is the image domain. In the task of 2D registration, we aim at finding a transformation

where g and f become equivalent in terms of anatomical structures.

For registering histological images, the most common approach has been mutual information based free form deformation (FFD) (Rueckert et al., _{p} ∈ _{p} corresponds to a displacement _{p} in a predefined displacement set _{p}. We define the bijective function _{p} between _{p} and _{p} for each node _{p}: _{p} → _{p}.

The ventricular system spans throughout the brain, providing fluid pathways in the brain and creating regions of empty space in almost every histological slice. Those cavities are easily deformed during slice preparation procedures and have much inter-subject variation. Thus when computing this MRF warp field, one needs to take into account the elasticity of different regions in the brain. By warping images to match with each other, we are essentially warping tissues: the more two adjacent control points are displaced, the more tension accumulates, if the two control points are connected through coherent tissue. In contrast, if they are separated by any hollow structure or empty space, no tension should be built in between.

The traditional and most common interpolation method for biomedical image analysis has been the B-spline model (Rueckert et al.,

Our system does so with a very simplistic model. We divide each target slice into two regions: free (ventricular system and background), and coherent (other areas) based on the annotation volume of ABA. We then classify the nodes as coherent (red) or free (green in Figure

Illustration of the coherency model and grid refinement from level t to level t+1. Grids are overlaid on an atlas image (contrast adjusted for better illustration) to show the coherency model. Green nodes are free nodes which include nodes in the free regions—ventricle systems and background—and affect real tissue. Red nodes are coherent nodes which cannot be seen since they become part of the tension edges, represented by red line segments between coherent nodes. During refinement, image resolution is 2 × the resolution of last level. The grid spacing remains the same. Therefore the grid quadruples in each direction, which is shown in the lower grid. The motion of existing nodes are carried onto the next level. The motion of non-existing nodes in the lower grids are interpolated from the motion of the existing nodes.

We first extract mask images _{c} and _{e} representing coherent region—tissue—and empty space region— ventricles and background—respectively from the annotation volume of the ABA and project them to the source image. We further group control points as coherent or free in Equation 8. Coherent control points are inside coherent regions. Free control points are the control points inside an empty space, and moving the control point will affect pixels inside coherent regions.

where the inverse influence function, η(.)^{−1}, adopted in Glocker et al. (

We further define a tension edge set, _{pq} is in _{c}. Because the spring potential energy is proportional to the square of displacement, we use squared difference as the pairwise term:

where λ is a regulation parameter.

We need to be able to both correct large distortions and make small changes to achieve good results. For both computation efficiency and quality of results, we use a multilevel approach. Since we are trying to model the tension that the deformations create, we need the pairwise energy terms to accumulate as we refine the grids. This requirement means that we cannot use the approach used by Glocker et al. (

The conventional multilevel approach (Glocker et al.,

To correctly carry on calculated displacements to the next level, we first need to compute the set of possible locations for each grid point, which depends on the results from the prior level. To do this, we denote the grid at level t as ^{t} and the influence function as η^{t}. Let ^{t} and bilinearly interpolate them to get the initial displacement for each node in ^{t+1} at the next level. We denote this preset displacement at node

where Θ is the allowable additional displacements and is the same for every node.

Having created a set of possible locations for each grid point, we next need to create the image that we will compare at this level to compute the similarity. In previous work, this warped image is input to this level, but we need to compute the image from the displacements of the previous level's control points and the labels associated with the node we are evaluating. When estimating the local patch around node

We denote the patch that is affected by p in the first level function _{0, p}. The control points in the patch at level t+1 is defined as:

To create the image that we will compare, we set the nodes in _{p} at the values from level _{0, p} when we associate label _{p} with node

Thus the unary term is given by the similarity measure between the warped patch and target patch:

where _{0, p} and ρ measures the difference score between two images. Since at every level, the only region that changes when we approximate the change for each possible label is centered around the node being evaluated, we approximate this change by simply translating the patch centered at the approximate new node coordinates

where _{t, p} denotes the patch centered at node

Eventually we formulate the MRF energy function at level t as the summation of the normalized unary similarity term to the corresponding atlas image, the unary similarity term to the warped previous image, and the pairwise term:

where

Because the free nodes are not constrained with any pairwise terms, they are essentially assigned labels that minimize the unary terms:

where

While HOG matches internal features well, we find it hard to align the contour of images. The atlas has very low-intensity pixels around real brain tissues as shown in Figure

where _{p} represents the intensity at pixel _{average} is the average intensity of all nonzero pixels in the image. Solving this energy function, we can obtain satisfactory result except that some very dark tissue regions near a slice's contour will be removed in some slices. To fix it, we keep the otherwise removed regions if the area is well-connected with its surrounding regions. This is accomplished by morphological eroding and dilating the to-be-removed regions with a disk of 20 pixels. We keep a region if it survives the opening operation. Same numbers are used across slices. We fill in holes in the computed mask so that the mask consists of a single piece. Even though the experimental slices are often preprocessed by neuroscientists to remove the nonzero-intensity background and keep only the tissues, this procedure is not quality-controlled. Therefore we refine the experimental images again with a similar method that is used to preprocess the atlas images, except that the masks are morphologically eroded and dilated with a disk of 3 pixels to encourage smoothness, and the to-be-removed regions will be kept if its area is greater than 50 pixels. These parameters were selected based on experiments on several slices in one of our experimental brains. Same parameters are used for all experimental slices. In the case that a slice is missing a relatively large portion of tissue, after the plane correspondence is found and before the 2D nonrigid registration, a manual preprocessing is done on the corresponding atlas slice to crop out the same corresponding portion that is missing in the corresponding atlas plane returned by our algorithm.

Example atlas slices with nonzero intensity regions circled by white contours. Scale, 1 mm.

After fixing the background noise, we find it still hard to use HOG difference to align image contours because HOG difference reduces dramatically only when after transformation the contours overlap or are separated by a distance smaller than the HOG cell size. Moreover, since the atlas is not a smooth volume, after rotation, the contours may be jagged - creating unwanted gradients. A more sensitive and more robust metric is needed. When displacing nodes that affect image contours, we are essentially warping the contour to maximize overlapped region of the two images or equivalently to minimize the symmetric difference of image foregrounds. Therefore, if a node

where the contour of the experiment image's real tissue is denoted by _{e}, and the real tissue in image _{f} and _{g}. In Equation 17 we estimate the warped patch by translating the patch center. This estimation improves computation speed while retaining performance when the similarity measure is HOG difference or another metric that involves more internal information. The shape of the experimental images is often deformed in the preparation process. However, simple translation does not change the shape. It only reduces the disjoint area but cannot find a transformation that reverts the deformation. Therefore instead of simply translating the patch, we warp the binary masks to evaluate this symmetric difference term.

Our framework was used in a systematic anatomical study in the hindbrain to map the brain regions containing the dorsal raphe nuclei to the ABA to study the organization of the dorsal raphe serotonin system and its behavioral functions related to depression and anxiety (Ren et al.,

We solve the squeezed aqueduct problem by warping the segmented aqueduct to the corresponding annotated atlas aqueduct with thin plate spline (TPS) (Bookstein,

With point correspondence on the aqueducts' contours, we add another term to the unary term so that the aqueduct is brought closer before reshaped with TPS. The term measures the Euclidean distance between the warped experimental aqueduct contour points and their corresponding atlas aqueduct contour points:

where an experimental aqueduct contour point _{p}, if its influence to node p - η_{t+1}(|_{p} are the corresponding aqueduct contour points in the atlas image, and

where _{a}, _{b}, _{d} and _{p} are the coefficients before the energy terms. We assign labels that minimize the new combined unary term to the free nodes:

where

We use a cell size of 15 pixels to measure the image similarity (see Section 3.1.1). This relatively large cell size allows us to capture structural similarities even with uncorrected small distortions. For nonrigid registration in Section 2.2, we decrease the cell size to 4 pixels, because the purpose of this step is to correct distortions. In both steps, the block size is 2 × 2. HOG is computed with the Vlfeat toolbox (Vedaldi and Fulkerson,

We select a subset A from all the slices I to find the best cutting angle and the best corresponding slices. For full brain data, which contains about 200 slices, we use about 30 slices with minimal artifacts - 1/6 to 1/7 of the whole sequence. In the anatomical study, each sectional brain consists of about 35 slices. We use every third image for most of the brains - about 12 slices for each experimental brain. For some brains with relatively more damaged slices, we manually checked the automatic selection and replaced slices with significant damage with a nearby good quality slice. Since many slices are used to find the best cutting angle, this manual check and replacement is only performed when many automatically selected slices are damaged. With Matlab, it takes 38.8s on a 12-core 3GHz Linux machine to evaluate a set of 12 slices, or equivalently to evaluate a cutting angle on a sectioned brain.

All terms—the unary terms and the pairwise term—in the energy functions presented in this paper are normalized to the range [0, 1]. Since the experimental images and the atlas volume are of the same modality, and the terms are normalized, the parameters before each term do not need to be heavily tuned to yield good results. Our original energy function consists of only one unary term—the HOG similarity term to the atlas slice—and the pairwise term. With several experiments, we find equal weight between the unary and the pairwise term generates the best visual result. In the general energy function in Equation 16, the HOG similarity term to the previous slice is added to encourage smoothness in the “reconstructed” volume and make the features that do not exist in the atlas but exist in the experimental volumes more consistent. We add an additional Euclidean distance term between the two aqueduct contour point sets in Equation 22 to suit our dataset better. Since the HOG difference to the corresponding atlas slice is the dominant term, we set it to be three times as strong to the HOG difference term to the previous slice and the Euclidean distance term between the two aqueduct contour point sets in both forms of the energy function. The coefficients before the pairwise term is set to be the sum of the previous coefficients to maintain the equal weight between unary and pairwise terms.

We use three iterations to complete the 2D nonrigid registration described in Section 2.2. The grid spacing is 16 × 16 in all iterations. In the first iteration, we downsample both images 4 × in both horizontal and vertical directions. In the second iteration, images are downsampled by 2 × . In the final iteration, we use the original resolution. The maximum displacement at each level is set to be half of the grid spacing. Therefore the total number of labels are 17 × 17 in each iteration. The optimization is computed with tree-reweighted message passing (TRW), more specifically TRW-S (Kolmogorov,

We chose five brains from all our brains that could represent the variability of aqueduct appearance and label the aqueduct of all slices in the selected brains. Both the experimental slices and aqueduct masks were downsampled to 512 × 512. One brain in these five brains happens to be in the five brains that we evaluate in the Evaluation Section 3.2. Because our training data consists of only five brains-about 150 slices, we data-augmented the training data and predicted the aqueduct of all other brains with this trained model. The quality of the prediction is correlated with the quality of the dataset. The generated masks are manually corrected if necessary which is about 10% of the total number of slices. The segmentation network consists of 9 layers. The input and output image has dimensions 512 × 512 × 1 where the input is the image to be segmented, and the output image is the predicted mask of the aqueduct. The first seven layers have dimension 512 × 512 × 32 with dilation rate doubling the rate of the previous layer starting from 1. The convolution kernel size is 3 × 3. The largest receptive field in the network is the seventh layer - 256 × 256. The last two layers consist of an undilated smooth layer with the same kernel size and a linear transformation layer. We use the intersection over union as the loss function and take 8 points on each half of the segmented aqueduct to compute the point distance term in the revised similarity term function in Equation 22.

Our framework was developed to register a full mouse brain slice sequence consisting of 202 60 μm-thick slices to the atlas and was also used in a systematic anatomical study in the hindbrain to study the organization of the dorsal raphe serotonin system (Ren et al.,

We use 5 brains in the anatomical study (Ren et al.,

The most common metric for evaluating image registration is the target registration error (TRE) measured as the Euclidean distance between landmark point coordinates in the target image mapped by a computed transformation to the source image and the corresponding landmark points in the source image. We asked one of our neuroanatomist coauthors to identify 20 sparsely-scattered landmarks in the hindbrain of the atlas which she would be confident in locating in both simulated and real experimental brains. The points are enough to cover all the significant brain areas in this study, because 1) an experimental brain has around 35 slices, 2) on a representative experimental slice, there are roughly 30 nuclei identified by its anatomical properties based on neuroscientists' historical consensus, 3) almost all the nuclei are shown on at least 5 slices. The corresponding points of these 20 points are marked by the same neuroscientist in the brains that we evaluated. In the full brain, we select 17 regions - 81 lateral ventricle, 581 triangular nucleus of septum, 286 suprachiasmatic nucleus, 338 subfornical organ, 223 arcuate hypothalamic nucleus, 830 dorsomedial nucleus of the hypothalamus, 470 subthalamic nucleus, 884 amygdalar capsule, 587 nucleus of darkschewitsch, 214 red nucleus, 931 pontine gray, 872 dorsal nucleus raphe, 642 nucleus of the trapezoid body, 574 tegmental reticular nucleus, 169 nucleus prepositus, 222 nucleus raphe obsurus, 207 area postrem (the numbers in front of region names are the region ID in the annotation volume) and generate landmarks automatically by sampling 100 points along these brain region boundaries. These regions show contrast to their neighboring regions in at least a few slices that contain them. The sampled points cover 32% of the total number of slices and 72% of the entire brain length. We then map the selected landmark points with the known transformation to obtain the ground truth.

In the sectional simulated brain, we can compute the true error of both our method and of the expert, since we have ground truth, as well as the TRE - expert and computation combined error. This information can help interpret the results in the five experimental brains, where we can only compute the TRE. For the full brain, we are only able to measure the computation error since the landmark points are generated automatically, but the information from expert error in the simulated sectional brain can help us interpret the results.

To compare our results to previous work, we use Ju's method (Ju et al.,

Figure

Boxplots showing the experiment results on the sectional simulated brain and the full simulated brain. The boxplots in the left two columns show the sectional brain results. We measured the intrinsic expert error, pure computation error, and the TRE - combined expert error and computation error of the reconstruction-first method and our method. The third column shows the results on the full simulated brain where we measured the pure computation error. The lines on the boxes represent the minimum, first quartile, median (red), third quartile, and maximum respectively. The star denotes the average.

Figure

Boxplots of the TRE on evaluated experimental brains.

For the simulated full brain, we display the sagittal view (ventricle systems masked out) of the results generated with the reconstruction-first method and our method. The reconstruction-first method first reconstructs the brain and then aligns the reconstructed brain to the atlas. Our method approaches the problem differently by first finding the best matching angle and the corresponding slices in the resliced atlas for each experimental image, then registers each experimental slice to their corresponding slice individually. To show the “reconstructed” brain with our method, we place each slice to the coordinates of the atlas volume rotated with the best cutting angles and interpolate the volume in the anterior-posterior direction to fill in the “missing” slices. The results are shown in Figure

Full simulated brain results, sagittal view. Scale, 1 mm.

Because sectional brains only consist of about 1/7 to 1/6 of a full brain length, showing the sagittal view of these thin stacks does not exhibit the correctness of alignment. Instead, we show four evenly-spaced slices in each experimental brain and their corresponding planes in the atlas volume after we mapped them to the same coordinates. Figure

Real experiments results: experimental images and their corresponding atlas planes after registration. Intensity 2 × in all images for visualization purposes. Scale, 1 mm.

Real experiments results: experimental images and their corresponding atlas planes after registration. Intensity 3 × in experimental images, 2 × in atlas images for visualization purposes. Scale, 1 mm.

Histological sectioning is the most commonly used method to investigate organizations of normal and diseased brains. Individual brain variations and distortions and intensity inconsistencies caused by sample preparations make aligning histological brain slices to a reference a challenging task for both experts and computer algorithms. To address these challenges, we put together a direct approach to solving the mapping problem between a 2D histological sequence and a reference volume that allows us to determine the best corresponding slice for each experimental slice before attempting any nonrigid alignment. It uses the L2 norm of HOG difference as the image comparison metric and the average matching index difference between half-images to create a sectioning angle measurement. The HOG metric enables image similarity comparison without the need of deformable registration. This produces a robust framework that leverages brain structural characteristics and symmetry to determine the cutting angle and matching slices without initial reconstruction. Avoiding reconstruction improves accuracy by preventing z-shift problems as validated by our comparison experiments. In 2D nonrigid registration, we augmented the standard MRF on medical image registration to model accumulated tension when deforming tissues to more naturally deal with the easily-deformed cavities throughout the brain. This requires us to use squared distance pairwise term and pass simulated stress across iterations.

Interestingly, the results from the comparison experiment between the reconstruction-first method and our method show that using sectional reconstruction for registration still introduces small errors. These methods must compromise between thinner sections, with less z-shift issues, and thicker sections that contain better matching information. As a result, our method has better accuracy for registrations of sections with only 1/7 of the full brain.

Since our method is mostly automatic, and the accuracy is similar to or better than an expert neuroscientist even for datasets where many slices are corrupted, we have successfully used our method to map multiple brain datasets in a recent anatomical study (Ren et al.,

The ABA (2015) also contains a population average of serial two-photon (STP) tomography images. While we used the grayscale Nissl volume of the ABA in our project, because our method is very robust to intensity variation, we tested aligning a Nissl-stained experimental image to the corresponding STP plane of the ABA. The STP volume is easier to prepare because the quality of imaging is overall better. The results are promising. In fact, we get similar qualitative results as the Nissl-stained atlas slice. Clearly, while further work will be needed in this multi-modality task, it seems this method might be useful to these applications as well.

Code and sample dataset in this study are available at the project's website,

JX wrote the software to register the histological slice sequence to the ABA. JR prepared the datasets used in this paper and has been significantly involved in evaluation. MH provided important advice and oversees the entire project with LL.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

This work is supported by the Hughes Collaborative Innovation Award and a BRAIN initiative grant (R01 NS104698). We thank Qifeng Chen for suggestions on segmentation networks and donating a GPU for our research, Steven Bell for discussion, proofreading, and hardware maintenance, Tao Ju for sharing his source code and advice on using the stack Aligner software, Allen Institute for Brain Science for the reference atlas, and Terri Gilbert for the advice on using the atlas.

This manuscript has been released as a pre-print at BioRxiv (Xiong et al.,