Detecting Nanosheet Objects from Noisy CLSM Images Using Deep Learning Approach

This paper considers a problem of detecting nanosheets which are moving in colloidal liquid from confocal laser scanning microscopy (CLSM) images. Introducing the deep learning approach, we particularly develop a scheme for constructing the so-called `detection map’ consisting of the brightness value information on the area of nanosheets in CLSM images. Therein, we use an architecture of deep learning network ‘U-net’ and present how to implement such a network. The performance is demonstrated by some experimental studies.


Introduction
Inorganic nanosheets are highly anisotropic plate-like materials with the thickness of ~1 nm and lateral size of microns [1].Inorganic nanosheets are produced as colloidal dispersions through the exfoliation of a layered solid in a solvent.Many kinds of nanosheet materials with various functions have been developed, opening applications as optical and electronic devices, polymer/inorganic nanocomposite gels for soft robotics, etc. (see e.g.[2]) In particular, curious aspect of nanosheet colloid is that the nanosheet colloids form a liquid crystal phase, in which nanosheets are aligned along a director and are forming periodic array on the mesoscopic scale (several nm to sub-μ m).Although the liquid crystal phase formation of nanosheet colloids has been basically formulated on the basis of Onsager's theory [3], further understanding of fundamental aspects are still required.
To proceed the above mentioned research, in-situ structural characterization of the colloidal nanosheets is indispensable.For characterization of nanomaterials, scanning electron and transmission electron microscopy are generally used; however, these methods are not applicable to colloidal systems because samples should be placed under high vacuum.X-ray scattering techniques are very important and effective methods; however, they only give statistical information, that is an averaged information over a certain range of time and space.Optical microscopy gives real time direct images, but space resolution is not enough.In this situation, Miyamoto et al. recently reported that back scattering mode of confocal laser scanning microscope (CLSM) equipped with resonant scanner is suitable for directly imaging of colloidal nanomaterials dispersed in a solvent [4].However, in the case of colloidal dispersion of nanosheets, the nanosheets are moving around (Brownian motions) not only in translational manner but also in rotational manners.It is also complicated that the shape of the nanosheet is not perfectly flat sheet; the shape is changing time-by-time due to flexibility of the thin sheet.Thus, one of our concerns is to quantitatively understand the motions and deformation of nanosheets through the observations using CLSM in combination with effective image analysis technique.
For solving such an issue, the so-called 'segmentation' play a key role to detect nanosheets on CLSM images.In general, a brightness value information of CLSM images corresponds to a depth from focal plane of CLSM.Thus, such a brightness value information on area of the detected nanosheets may be used in order to understand their motions.However, no generic method for solving the image segmentation problem has been existed due to an existence of observation noises.CLSM images contain some noises due to the microscope instruments.In particular, unlike the ordinary observations of solid materials, the material which we here observe using CLSM is nanosheets in the colloidal liquid.Thus, laser light scattering due to a mechanism of CLSM may be considerably contained as noises on images.Also, recording a sequence of CLSM images with high sampling rate is desirable from the viewpoint of understanding whole nanosheet motions -such as changes on deformation motions during an observation.But, as the sampling rate becomes high, the noise level of images increases due to ISO sensitivity.Thus, we may readily see that most conventional segmentation approaches (see e.g.[5]) -such as morphological segmentation approach may not work well in the above case.
A main purpose of this study is to develop a robust scheme for detecting nanosheet objects from noisy CLSM images.Introducing the deep learning approach, we particularly develop a method for constructing the so-called `detection map' consisting of the brightness value information on the area of nanosheets in CLSM images.As a deep learning network, we here use the so-called 'U-net' which has been developed by Ronneberger [6].Thus, the basic idea of this study is similar to Ronneberger's work in [6], but a big difference is that we attempt to detect nanosheets from noisy CLSM images and to construct a detection map.The performance is demonstrated by some experimental studies.

Nanosheets and CLSM System
As preliminaries, we briefly present an overview of nanosheets and confocal laser scanning microscopy (CLSM).
Nanosheets are generally obtained as colloidal dispersions through the exfoliation of a layered solid in a solvent.In such a way, the colloidal hexaniobate nanosheets is here produced by exfoliating single crystal of layered niobate K4Nb6O17 (see [7] for the details).Figure 1 shows a photograph of such colloidal hexaniobate nanosheets.For the sake of simplicity, we will refer these colloidal hexaniobate nanosheets as 'nanosheets' in the sequel.
Figure 2 illustrates an overview of CLSM system (Nikon Eclipse Ti).This system consists of a florescence microscope and the confocal part, including scan head, laser optics and PC (see e.g.[8] for the details of CLSM principle).A small amount of nanosheets in Figure 1 is dripped into a plastic bottom dish and is observed by CLSM.Although CLSM can do 'z-stack' which captures multiple two-dimensional images at different depths in a sampling rate and reconstruct three-dimensional structures, we here fix the depth in some height.That is, only CLSM images at a depth (i.e.focal plane) are observed at a sampling rate, and then a sequence of such images are stored in PC.In Figure 3, an example of CLSM image observed at a high sampling rate are shown, where the sampling rate is 15 [frames/sec].The observation is performed using a laser with wavelength 405 nm.In the image, a brightness value (red-and-black color) are equivalent to a depth from a focal plane of CLSM, where the color 'red-and-black' is set by a software attached to CLSM system.Therein, darkest red color corresponds to the depth corresponding to the focal plane.As the color becomes black, it means that the depth become low or high away from the focal plane.
On the other hand, what we want to observe using this CLSM is nanosheets which are moving in a colloidal liquid.However, due to the property of colloidal liquid, laser light scattering may be observed as noises around the center of CLSM images.Moreover, as in Figure 3, the high sampling Figure 1: A sample of colloidal nanosheets.
Figure 2: Overview of CLSM system.
rates yields the noisy images due to ISO sensitivity.Therefore, our task is to detect the nanosheets and construct its detection map through the observation using CLSM at a high sampling rate as in Fig. 3. Figure 3: An example of CLSM image observed at a high sampling rate.

Detecting Nanosheets Using Deep Learning
We are now in the position to develop a method for detecting nanosheets from a set of CLSM images using deep learning approach.We first present the network architecture 'U-net' [6], which is used to detect nanosheets from CLSM images.Next, we present the implementation, say, how to train U-net and get a detection map.U-net is a network with a variant form of convolutional neural networks for image-to-image conversion.Basically, this has been developed for image segmentation task with some microscopy images in the biomedical field.U-net can convert input images into different images that usually contain some kind of novel information which can be expected, predicted, or estimated from the input images.The network architecture is based on the fully convolutional encoder-decoder network with some skip connections in order to transfer low-level features from the encoder layers into the corresponding decoder layers (see [6] for the details).Such architecture makes it possible to yield more precise conversion results with fewer training images.
Here, we build the network with five encoding blocks followed by five decoding blocks.Each encoder block is composed by two 3×3 convolution layers with a rectified linear unit (ReLU) activation, and then a 2×2 max pooling with strides of two in each dimension.In the decoder layer, each decoder block contains a 2×2 up-sampling convolution with strides of two in each dimension, followed by two 3×3 convolution layers with a ReLU activation.In the final layer, we use a sigmoid activation function.As input to this network, we use an RGB image (i.e. three channels) with a size of 256×256 [pixel].In the final layer of network, output is 256×256 [pixel], but is a gray scale image (i.e. one channel).
Here, the network is implemented by employing 'Keras' with 'TensorFlow' backend (see e.g.[9]), which are programming framework for deep learning.In order to train the network, we need some sets of paired data consisting of the input CLSM images observed at high sampling rate (e.g. an image in Figure 3) and their corresponding outputs (i.e.detection map).Unfortunately, we here have no their ground truth on the detection map.A natural idea of generating such a ground truth data may be to manually detect the nanosheets from input CLSM images using the conventional image processing techniques and to store the corresponding brightness values as a detection map.But, the resulting detection map may be out of an allowable region due to noises.
We thus consider to generate a training dataset from a set of CLSM images observed at a low sampling rate (e.g. 1 [frame/sec]), in which the level of ISO noise is quite smaller than the case of high sampling rate.The procedure for generating a set of training data is as follows.As the input of training dataset, we generate noisy CLSM images by adding zero-mean Gaussian white noise with various variances σ 2 to CLSM images in the case of low sampling rate.On the other hand, the output of training data is generated as follows.We first estimate and remove a background of input CLSM images using morphological opening and its subtracting method.Then, the image contrast is increased and is converted to gray-scale image.Also, we reduce the gray-scale image to a binary image by using Otsu's method (see e.g.[5]).Using the Hadamard product, we then take the 'detection map' as Key Engineering Materials Vol.804 output (gray-scale image) of training dataset.Such a paired data of input CLSM image and output detection map is used to train the network with the cross entropy loss and ADAM optimizer [10].
In Figure 4, we summarize our network architecture and the procedure on data generation described in the above.After the training, the input images for the network are replaced to CLSM images observed at a high sampling rate (e.g. 15 [frame/sec]).Even though such images might contain high ISO noise, we can expect that the trained network correctly detect nanosheets from them if the training using artificially noised images was appropriately conducted.

Experimental Studies
We demonstrate the performance of our method by some experimental studies.For conducting the experiments, we observed nanosheets using CLSM (Figure 2) in low and high sampling rates (see e.g. Figure 3 for the case of high sampling rates) and a sequence of their observed images are stored, where the number of images is 2089 and 2357 for the both cases of low and high sampling rates, respectively.Note here that all the observed CLSM images in low sampling rate are used for training the network.Then, each input data is generated by artificially adding zero-mean Gaussian white noise with various variances σ 2 to the CLSM image.Here, we set σ 2 as σ 2 = 0, 0.2,0.4,0.6, 0.8.The corresponding output data is also generated by the method in Fig. 5. Hence, 10445 (= 2089×5) paired data is totally used to train the network.Also, the batch size and epoch number for training are set as 10 and 100, respectively.For this implementation, we used PC (CPU: Intel Core-i9-7900X, RAM: 128GB) with GPU (NVIDIA GeForce GTX1080Ti).
Applying such a trained network to CLSM images observed in high sampling rate, we demonstrate the performance of constructing detection map. Figure 5 illustrate an experimental result.Here, Figure 5 (a) shows input CLSM images for the network, and (b) detection map as output of network in the above section.For the sake of comparison, we show the results corresponding to the detection map in Figure 5 (c), which are obtained by using a function on the threshold-based segmentation of the software 'Image J' [11].From these results, we may see that our method works well and the nanosheets are detected appropriately even from CLSM images which are considerably corrupted by noises.

Conclusion
This paper considered a problem of detecting nanosheets which are moving in colloidal liquid from confocal laser scanning microscopy (CLSM) images.By introducing the deep learning approach, we particularly developed a scheme for constructing the so-called `detection map' consisting of the brightness value information on the area of nanosheets in CLSM images.Therein, we presented the architecture of deep learning network 'U-net' which is used here.We then present how to generate a training dataset in order to implement such a network.The performance was demonstrated by some experimental studies.

Figure 4 :
Figure 4: The architecture of U-net and the procedure on data generation.