A Computer Tool for 3D Shape Recovery of Fruits

Characterizing a fruit’s mechanical behavior is an important step towards reducing economic losses due to bruising. Several 3D scanning technologies allow to obtain the external geometry of a fruit, but no easily accessible tools exist for the acquisition of the geometry of internal structures such as the core. We propose a low-cost destructive method for tomographic reconstruction of a fruit from scanned slices. A method for overcoming the difficulties in registering the different images is also presented.


Introduction
The importance of the geometry of fruits for the study of bruising is a paramount for fruit producers, distributors and sellers. High percentage of loss due to fruit bruising during harvesting, transportation and storage has been estimated to be close to 25% [1]. Therefore there is a high social and economical value in understanding the effect of different handling in fruit bruising. Of particular interest are the apple and the pear.
Several studies have been published on this subject, attempting to characterize the mechanical behaviour of different fruit varieties, both by experimental means and by computational simulations [2][3][4][5][6][7][8]. Numerical simulation is one of the most powerful techniques, however, in order to obtain reliable results a realistic model should be used. We have previously [1] experimentally determined the mechanical properties of the apple and the pear. The results showed that both properties varied clearly along the axis as well as radially. It is thus clear that a proper model of the fruit must not only use the external surface geometry but also include some internal structure. Nevertheless most authors have modeled the fruits as spherical in shape and uniform in their mechanical properties which doesn't mimic the interior or exterior geometry of each fruit.
Previous works. In previous works [10,11] we have explored the automated reconstruction of the outer surface of fruits from digitized images of slices, using a low-cost approach. This is a type of destructive tomography. Destructive tomographic techniques have been used for a long time, but they are usually labour-intensive, and require a trained operator (for more details see [9]). To keep the procedure as simple as possible, we did not use any type of fiducial markers, relying instead on the algorithms for registration of the images (tomograms). We also explored different ways of slicing and of surface reconstruction (isosurfaces versus contour-based vector surfacing) which allowed us to obtain low-fidelity reconstructions of the outer surface. Fig. 1 Previous work on the reconstruction of the outer surface of an apple fruit using wedgeshaped slices and contour-based registration. Left: two scanned slices with superimposed contours. Center: matching two contours for registration using cross-correlation. Right: outer surface reconstructed from the contours. In the presented work we focus on obtaining the internal structure of the pear, necessary for a better mechanical model of the fruit. Therefore it was essential to align and process the raw slice images in order to obtain a proper stack of images forming a meaningful volumetric representation of the fruit. This could then be used with standard tomography software tools, or further processed using custom algorithms.

Slicing and digitization.
A pear fruit (Rocha Portuguese variety) was sliced using a household rotary slicer (Krups TR 522). The slicing was performed with a fixed thickness of 6.7 mm, starting from the top (stem). We performed eleven parallel cuts, resulting in twelve slices. For reference, we have digitized the surface of the same specimen prior to slicing, using a Comet 5 3D scanner from Steinbichler. The model obtained and a set of simulated slices are presented in figure 3.
For the digitization, a Ricoh Aficio MPC 2051 photocopier and printer was used. The slices were immediately placed on the surface of the photocopier, in their natural order, and scanned in grayscale at 600 dpi resolution (see figure 4). The cover of the scan surface was not lowered, instead we turned off the lights to avoid casting shadows or undesired stray light and to ensure a dark background. Two scans where made, one with the slices' bottom face facing the scanner, and another one with the opposite orientation. The scans where saved in two PDF format files, from which the image was later extracted using the open source software 'poppler'.

182
Direct Digital Manufacturing and Polymers Fig. 4 The slicing (left, center) and scanning (right) procedure. Refer to the text for details.
Reconstruction of volumetric data. Tomographic reconstruction relies on some conditions, of which we highlight the following: consistency and independence. The former is related to the interpretation of the intensity levels in each tomogram, which should be consistent across all tomograms in a set; lighting conditions and other imaging factors should be kept constant throughout the digitization of all slices. The latter relates to the spatial interpretation of the tomograms: each tomogram should contain only data associated with a single plane. Enforcing these conditions in destructive physical-optical tomography usually involves a laborious preparation of the specimen and complex imaging setups.
In non-destructive tomography devices, the subject is usually mechanically immobilized and the acquired scans have a common spatial reference; in our case, however, we will have each slice image affected by a different translation and rotation. Additionally, each slice image captures not only the exposed interior of the fruit along one cut plane, but also a possibly part of the outside of the fruit between the cut plane and the adjacent one. In order to obtain a meaningful data volume, we must consistently align all frames and use only the data (pixel) that corresponds to the interior of the fruit.
It is important to introduce some terminology. We will refer to each individual image of a slice as a frame (reserving the term tomogram for a frame that respects the consistency and independence constraints), and for each slice we have a top and a bottom frame. A frame may present only the interior, the corresponding back face is occluded, and no part of the outer surface of the fruit is presented, and in this situation it is named a 'clean' frame. Otherwise, we call it a 'mixed' frame. A set of frames from distinct slices will be designated as a series, and we are particularly interested in the top (and bottom) series formed by all the top (respectively, bottom) frames, and also the clean (and mixed) series.

Results
Slicing and Digitization. The first step of the procedure consists of the digitization of slices of the fruit. To that we used an electric slicer, as shown in figure 4.
Tests indicated that slices tend to get damaged on the side farthest from the blade, which can be seen in some of the slices in the form of a non-planar cut. The smallest thickness we could reliably use was about 5 mm. We also found that it is easier to start slicing from the top (narrower side) in the case of the pear.
The slicing was done with a thickness of 6.7 mm. All the slices were placed in the scanner in order (left to right, then top to bottom), and scanned on both sides. The scan was performed in grayscale at a resolution of 600 dpi. Figure 5 below presents the raw images obtained1. Image Processing and Segmentation. Previously, we have scanned each slice of the fruit separately, but this time we scanned all slices in one single image. This has the benefit of all slices have the exact same exposure settings, so that it is easier to have a common interpretation of different intensity levels in the images which will help to identify specific regions of the fruit.
As we want to have each slice's image separately accessible, we started by identifying the positions of the slices by means of a Hough Transform. This was implemented in Matlab [13] using the function imfindcircles. Using the centers and radii, the regions containing each slice were cropped, and padded to guarantee that they all have the same dimensions using padarray function with the 'replicate' option. The result was then cleaned up by imclearborder, removing any part of adjacent slices that was included in the crop area. Finally, in this particular case we re-assigned the resulting frames to the correct series. Figure 6 shows all the frames. During this process, we also identify the largest slice. This procedure separates each series in two sub-sequences, one of increasing radius and the other of decreasing radius. Swapping these between the top view and the bottom view, we obtained two other complete sets: one where all frames contain a view of the two contours of a slice (back face and front face), and another one where each 1 Note that due to an mistake in the procedure some of the slices are not consistently oriented, i.e. in both images some of the slices appear with bottom-side up.

184
Direct Digital Manufacturing and Polymers frame contains only a foreground view. We will refer to these two series as 'mixed series' and 'clean series' respectively. Note that in the clean series we have two views of the same cut -frames 5 and 6; this should be taken into account when reconstructing the surface (we only have eleven cuts, if we have twelve frames). Also note that before any further processing or combination of the two series, the frames from one of the original series (either top or bottom) must be mirrored (flipped), since they represent views from two different sides.

Image Registration and Preparation of Volumetric Data.
Our previous use of the contours for registration [11] relied on the existence of easily identifiable features, persistent across all slices, which in the case of radial slicing can be associated with the distinct geometry of the top and bottom of the fruit. In the case of parallel slicing, however, the overall central symmetry of the fruit results in a lack of clear features making the contour-based registration difficult and error prone. We have thus opted to perform an image-based registration, using the imregister/imregtform functions from Matlab's Image Processing Toolbox. We could not find a detailed description of the algorithm used in those functions 2 .
In each of the two sets of images, there are some slice views that include a full or partial view of the back-face and the skin. For the surface reconstruction this represents a problem because not all of the frame pixels correspond to the same plane, with the same z level (violating the 'slice independence' criteria). Therefore, for the reconstruction, combine the frames from each set where the back-face was fully occluded. For this, we needed to align those frames to a common reference coordinate system. However due to the thickness of the slices, the successive cuts were too different to be successfully registered.
The solution to this problem was to use an indirect registration (see figure 7). This was possible because the frames that contain a non-occluded view of the back-face (the mixed series) provided a reference for the alignment of the adjacent frame. Then, the adjacent frames in the mixed series, and the corresponding frames in the mixed and clean series can be registered; these transformations can then be composed in order to obtain the registration of adjacent frames in the clean series. For clarity, we consider the transformation as vi, when it aligns adjacent frames in the mixed series, and the transformation that aligns the i th mixed frame to the i th clean frame as ui, then wi = ui+1 ○ vi ○ ui -1 is the desired transformation that aligns the clean frames. 2 Matlab's documentation describes it only as Intensity-based registration techniques correlate image intensity in the spatial or frequency domain. The moving image undergoes a single global transformation to maximize the correlation of its intensity with the intensity of the fixed image. The registration of the frames between the top and bottom series, i.e. the top and bottom view of each slice, was totally successful, therefore it was possible to correctly align all frames without any tweaking of parameters (see figure 8, left).
Frame alignment was assessed visually by means of composite false color images, where one of the frames' intensity was assigned to green channel and the other to magenta (red and blue channels). This procedure contribute to a very clear identification of the mismatches, while the matched regions were still clearly visible as neutral gray. This visualization however was not suitable for reproduction in grayscale.
The first attempts at registering the adjacent frames in the mixed set, however, gave poor results. Since both an inner and outer contour were presented in both frames, but only one pair of contours matches, the registration fails. We realized however that it could be overcome by matching a frame which was presented only one contour to an adjacent frame presenting exactly one contour in common -as we had found already with the registration of top and bottom series. Referring to figure 7, we can match the bottom left frame to the top center frame, thus obtaining vi ○ ui -1 directly. Although some pairs where perfectly registered, other pairs gave ambiguous or clearly wrong alignments. The registration procedure turns out to be somewhat inflexible, and prone to stick to local extrema. Experimentation with the optimizer and metric parameters improved the quality of the successful alignments, but did not solve the other cases. We believe a different metric function could solve these problems, but the image registration routines provided in Matlab do not support a custom metric. The solution was to force the optimizer to look for a different local extrema, by providing an 180 degree rotated input image; this resulted in a successful alignment of the pairs that previously failed, and failure in the previously successful cases (see figure 8, right).

Surface Reconstruction and Extraction of the Core Geometry.
Having a set of aligned images, we can now export it as a multi-frame dicom file for processing in a dedicated software such as Slicer [14] or InVesalius [15]. We opted for performing the remaining processing in Matlab in view of the integration of all steps in a single tool.

Direct Digital Manufacturing and Polymers
Since the number of slices is very small, the volumetric data was interpolated to a resolution five times higher using interp3. After that the correct physical dimensions can be set using the known resolution in the slice plane (the imaging resolution being known) and the known slice thickness.
Although we have previously adopted a vector (or contour) based surface reconstruction, in this application we relied on the isosurface method (via Matlab's isosurface function). A straightforward application of the isosurface function to this dataset with different thresholds is shown in figure 9. This already shows several different identifiable structures such as, for example, the seeds. For the extraction of the core, we started by finding a proper set of thresholds for separating the different features. These thresholds are used with the grayslice function. Since the core comprises some dark levels in the same range of the background, we then applied imclearborder to remove the background, resulting in a binary 3D array representing the pixels of the core. A morphological opening followed by a morphological closing eliminates the smaller isolated regions, while also smoothing the boundary.
A different approach to the extraction of the boundaries of inner structures from the volumetric data at a voxel level was explored in [12].
Processing (cleanup) of the surface and alignment with the outer surface. In the isosurface obtained in the previous step the core is connected to the outer surface. The surface was decomposed by cutting the bottom of the interior structure. For illustrative purposes, we further processed the obtained mesh in MeshLab [16], in order to obtain a single and smoother closed surface, more suitable for use in simulations. A 'Uniform Mesh Resampling' filter was applied twice with a positive offset and with a negative offset, thus performing a morphological opening, whereby the smaller features are eliminated as well as small holes that still exist. We also combined the isolated core geometry with the high-resolution scan of the outer surface, by first visually aligning the former to the low-resolution outer surface. The result is presented in figure 10.

Applied Mechanics and Materials Vol. 890
The reconstruction of the outer surface and the extraction and processing of the core represent preliminary work which will be improved in the future, by using an interpolation method similar to the one used in a previous work as mentioned in the introduction (see figure 1).

Conclusions
In this paper we presented a volumetric reconstruction of the interior of a pear fruit using parallel slices. We were able to successfully overcome the difficulties in the registration of the tomograms by taking advantage of the possibility of scanning both sides of the slices and using an indirect registration procedure.
Although the procedure presented allowed us to obtain a usable triangulated surface representing an internal structure of the fruit, the fidelity of this reconstruction is hindered by the need for a human intervention, introducing a subjective factor due to the assessment of the best alignment of the different slice views. The same applies to the matching of the obtained internal structure to the outer surface, which could only be roughly reconstructed from this procedure alone. Also, since no staining was used, the interpretation of the intensity levels as representing different internal tissues requires further validation. Better results could probably be obtained by introducing clearly identifiable features for the alignment procedure (fiducial markers), perhaps scoring or otherwise marking the surface before slicing. Still, we consider this first result useful, providing usable results through a low cost procedure and little user intervention. The applicability of this procedure to other fruits should be further explored.
This results will allow us to mimic the geometry of the pear which will be very useful to create the numerical model of this fruit. The numerical results of compression, vibration and impact simulations obtained will help to reduce fruit damage and can be further explored on other fruits and vegetables.