掲載種別：研究論文（国際会議プロシーディングス）   出版者・発行元：IEEE  

DOI： 10.1109/sii55687.2023.10039109

researchmap

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）   出版者・発行元：IEEE  

DOI： 10.1109/sii55687.2023.10039474

researchmap

記述言語：日本語   掲載種別：研究論文（学術雑誌）  

DOI： 10.2493/jjspe.89.90

researchmap

掲載種別：研究論文（学術雑誌）   出版者・発行元：Informa UK Limited  

DOI： 10.1080/01691864.2022.2084346

researchmap

掲載種別：研究論文（国際会議プロシーディングス）   出版者・発行元：IEEE  

DOI： 10.1109/sii52469.2022.9708610

researchmap

掲載種別：研究論文（国際会議プロシーディングス）   出版者・発行元：SPIE  

DOI： 10.1117/12.2590827

researchmap

担当区分：筆頭著者   掲載種別：研究論文（国際会議プロシーディングス）   出版者・発行元：IEEE  

DOI： 10.1109/IEEECONF49454.2021.9382719

researchmap

その他リンク： https://dblp.uni-trier.de/db/conf/sii/sii2021.html#PathakULMMYA21

掲載種別：研究論文（学術雑誌）  

DOI： 10.1080/01691864.2020.1857305

researchmap

掲載種別：研究論文（国際会議プロシーディングス）   出版者・発行元：Springer  

DOI： 10.1007/978-3-030-95892-3_47

researchmap

その他リンク： https://dblp.uni-trier.de/db/conf/ias/ias2021.html#SugasawaCKKPYHN21

掲載種別：研究論文（学術雑誌）  

DOI： 10.3390/s21216975

researchmap

掲載種別：研究論文（国際会議プロシーディングス）   出版者・発行元：IEEE  

DOI： 10.1109/SMC52423.2021.9658793

researchmap

その他リンク： https://dblp.uni-trier.de/db/conf/smc/smc2021.html#KawataHPYA21

researchmap

掲載種別：研究論文（学術雑誌）  

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. Self-localization enables a system to navigate and interact with its environment. In this study, we propose a novel sparse semantic self-localization approach for robust and efficient indoor localization. “Sparse semantic” refers to the detection of sparsely distributed objects such as doors and windows. We use sparse semantic information to self-localize on a human-readable 2D annotated map in the sensor model. Thus, compared to previous works using point clouds or other dense and large data structures, our work uses a small amount of sparse semantic information, which efficiently reduces uncertainty in real-time localization. Unlike complex 3D constructions, the annotated map required by our method can be easily prepared by marking the approximate centers of the annotated objects on a 2D map. Our approach is robust to the partial obstruction of views and geometrical errors on the map. The localization is performed using low-cost lightweight sensors, an inertial measurement unit and a spherical camera. We conducted experiments to show the feasibility and robustness of our approach.

DOI： 10.3390/s20154128

Scopus

PubMed

researchmap

掲載種別：研究論文（学術雑誌）  

© 2013 IEEE. In this paper, we propose SelfSphNet, that is, a self-supervised learning network to estimate the motion of an arbitrarily moving spherical camera without the need for any labeled training data. Recently, numerous learning-based methods for camera motion estimation have been proposed. However, most of these methods require an enormous amount of labeled training data, which is difficult to acquire experimentally. To solve this problem, our SelfSphNet employs two loss functions to estimate the frame-to-frame camera motion, thus giving two supervision signals to the network with the usage of unlabeled training data. First, a 5 DoF epipolar angular loss, which is composed of a dense optical flow of spherical images, estimates the 5 DoF motion between two image frames. This loss function utilizes a unique property of the spherical optical flow, which allows the rotational and translational components to be decoupled by using a derotation operation. This operation is derived from the fact that spherical images can be rotated to any orientation without any loss of information, hence making it possible to 'decouple' the dense optical flow between pairs of spherical images to a pure translational state. Next, a photometric reprojection loss estimates the full 6 DoF motion using a depth map generated from the decoupled optical flow. This minimization strategy enables our network to be optimized without using any labeled training data. To confirm the effectiveness of our proposed approach (SelfSphNet), several experiments to estimate the camera trajectory, as well as the camera motion, were conducted in comparison to a previous self-supervised learning approach, SfMLearner, and a fully supervised learning approach whose baseline network is the same as SelfSphNet. Moreover, transfer learning in a new scene was also conducted to verify that our proposed method can optimize the network with newly collected unlabeled data.

DOI： 10.1109/ACCESS.2020.2977109

Scopus

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2020 SPIE. With the advantage of having a large field of view, fisheye cameras are widely used in many applications. In order to generate a precise view, calibration of the fisheye cameras is very important. In this paper, we propose a method of extrinsic parameters calibration of multiple fisheye cameras working in man-made structures. A Manhattan Worlds space assumption is used, which describes man-made structures as sets of planes that are either orthogonal or parallel to each other. The orientation of the cameras is obtained by extracting vanishing points that denote orthogonal principal directions in different images captured by the cameras at the same time. With the proposed method, the calibration of extrinsic parameters is very convenient and the system can be recalibrated remotely.

DOI： 10.1117/12.2566562

Scopus

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2019 IEEE. Comparing to perspective cameras, the all-round 3D measurement of the environment can be done by spherical cameras in a more efficient way. However, the measurement using binocular spherical stereo has two singularity points at the epipoles of each spherical camera, where the measurement result gets extremely sensitive to the error when getting close to the epipoles and along the epipolar directions. This affects the accuracy of 3D reconstruction along with the epipolar directions. A three-way measurement method using three spherical cameras with trinocular spherical stereo setup is proposed in this paper to achieve accurate all-round 3D measurement. The improved accuracy of 3D measurement by the implementation of weighted reprojection error optimization was verified in experiments.

DOI： 10.1109/ISM46123.2019.00021

Scopus

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2019 IEEE. Spherical cameras are suitable for all-round 3D measurement due to their wide field of view. When performing 3D measurement using binocular spherical stereo, the areas in the epipolar directions cannot be reconstructed accurately due to the low measurement confidence in the epipolar direction. By adding one more spherical camera, perpendicular to the epipolar direction, to the binocular spherical stereo to form a trinocular spherical stereo, all-round 3D measurement can be achieved. In this paper, we propose a method to improve the 3D measurement accuracy by minimizing the overall reprojection error. Effectiveness of the proposed method is verified through experiments.

DOI： 10.1109/GCCE46687.2019.9015551

Scopus

researchmap

記述言語：日本語   出版者・発行元：公益社団法人 精密工学会  

<p>Localization is an important function for a mobile robot. In this research, a new localization method using a spherical stereo camera is proposed. Spherical stereo cameras are effective for motion estimation as they can visualize and measure all directions of the environment. Instead of using conventional sparse feature points, dense information of all pixels in images is used to estimate motion accurately. However, because of the wide field of view, the 3D measurement uncertainty is not uniform across the image. Hence, geometric uncertainty of every 3D point is calculated and used as weights to improve accuracy. Experiments show the effectiveness of using uncertainty information to increase the accuracy of motion estimation.</p>

DOI： 10.2493/jjspe.85.568

Scopus

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2019 IEEE. Spherical cameras, which can acquire all-round information, are effective to estimate rotation for robotic applications. Recently, Convolutional Neural Networks have shown great robustness in solving such regression problems. However they are designed for planar images and cannot deal with the non-uniform distortion present in spherical images, when expressed in the planar equirectangular projection. This can lower the accuracy of motion estimation. In this research, we propose an Equirectangular-Convolutional Neural Network (E-CNN) to solve this issue. This novel network regresses 3D spherical camera rotation by uniformizing distorted optical flow patterns in the equirectangular projection. We experimentally show that this results in consistently lower error as opposed to learning from the distorted optical flow.

DOI： 10.1109/ICASSP.2019.8682203

Scopus

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2019 IEEE. Self-localization in indoor environments is a critical issue for visually impaired people. Most localization approaches use low-level features and metric information as input. This can result in insufficient output for visually impaired people since humans understand their surroundings from high-level semantic cues. They need to be provided their location with respect to the objects in their surroundings. Thus, in this work, we develop a novel framework that uses semantic information directly for localization, which can also be used to inform the user about his surroundings. The developed framework directly uses sparse semantic information such as the existence of doors, windows, tables, etc. directly within the sensor model and localizes the user within a 2D semantic map. It does not make use of any distance information to each semantic landmark, which is usually quite difficult to obtain. Nor does it require any kind of data association- the objects need not be uniquely identified. Hence, it can be implemented with simple sensors like a camera, with object detection software. For our framework, one of the most popular game engines, Unity was chosen to create a realistic office environment, consisting of necessary office items and an agent with a wide-angle camera representing the user. Experimentally, we show that this semantic localization method is an efficient way to make use of sparse semantic information for locating a person.

DOI： 10.1109/SII.2019.8700370

Scopus

researchmap

担当区分：筆頭著者   掲載種別：研究論文（国際会議プロシーディングス）  

© COPYRIGHT SPIE. In this research, we propose a novel distortion-resistant visual odometry technique using a spherical camera, in order to provide localization for a UAV-based, bridge inspection support system. We take into account the distortion of the pixels during the calculation of the 2-frame essential matrix via feature-point correspondences. Then, we triangulate 3D points and use them for 3D registration of further frames in the sequence via a modified spherical error function. Via experiments conducted on a real bridge pillar, we demonstrate that the proposed approach greatly increases the accuracy of localization, resulting in an 8.6 times lower localization error.

DOI： 10.1117/12.2520206

Scopus

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2018 IEEE. Localization is an important task for mobile service robots in indoor spaces. In this research, we propose a novel technique for indoor localization using a spherical camera. Spherical cameras can obtain a complete view of the surroundings allowing the use of global environmental information. We take advantage of this in order to estimate camera position and the orientation with respect to a known 3D line map of an indoor environment, using a single image. We robustly extract 2D line information from the spherical image via spherical-gradient filtering and match it to 3D line information in the line map. Our method requires no information about the 3D-2D line correspondences. In order to avoid a complicated six degrees of freedom (6 DoF) search for position and orientation, we use a Manhattan world assumption to decompose the line information in the image. The 6 DoF localization process is divided into two phases. First, we estimate the orientation by extracting the three principle directions from the image. Then, the position is estimated by robustly matching the distribution of lines between the image and the 3D model via a spherical Hough representation. This decoupled search can robustly localize a spherical camera using a single image, as we demonstrate experimentally.

DOI： 10.1109/ICRA.2018.8460920

Scopus

researchmap

担当区分：筆頭著者   掲載種別：研究論文（国際会議プロシーディングス）  

© 2018 IEEE. Conventional techniques for frame-to-frame camera motion estimation rely on tracking a set of sparse feature points. However, images taken from spherical cameras have high distortion which can induce mistakes in feature point tracking, offsetting the advantage of their large fields-of-view. Hence, in this research, we attempt a novel approach of using dense optical flow for distortion-robust spherical camera motion estimation. Dense optical flow incorporates smoothing terms and is free of local outliers. It encodes the camera motion as well as dense 3D information. Our approach decomposes dense optical flow into epipolar geometry and the dense disparity map, and reprojects this disparity map to estimate 6 DoF camera motion. The approach handles spherical image distortion in a natural way. We experimentally demonstrate its accuracy and robustness.

DOI： 10.1109/ICIP.2018.8451406

Scopus

researchmap

担当区分：筆頭著者   掲載種別：研究論文（国際会議プロシーディングス）  

© 2017 IEEE. Virtual reality (VR) systems using head-mounted displays (HMDs) can render immersive views of environments, allowing change of viewpoint position and orientation. When there is a change in the position of the viewpoint, different objects in the scene undergo different displacements, depending on their depth. This is known as 'Motion Parallax' and is important for depth perception. It is easy to implement for computer-generated scenes. Spherical cameras like the Ricoh Theta S can capture an all-round view of the environment in a single image, making VR possible for real-world scenes as well. Spherical images contain information from all directions and allow all possible viewpoint orientations. However, implementing motion parallax for real-world scenes is tedious as accurate depth information is required, which is difficult to obtain. In this research, we propose a novel method to easily implement motion parallax for real world scenes by automatically estimating allround depth from two arbitrary spherical images. The proposed method estimates dense optical flow between two images and decomposes it to the depth map. The depth map can be used to reproject the scene accurately to any desired position and orientation, allowing motion parallax.

DOI： 10.1109/SII.2017.8279335

Scopus

researchmap

記述言語：日本語   出版者・発行元：公益社団法人 精密工学会  

In this paper, a novel method for 6 degrees of freedom (DoF) localization of a single spherical camera in a man-made environment is proposed. Taking advantage of the various line features that are usually present in such an environment, a technique to match the 2D line feature information inside a spherical image to the 3D line segment information available in a known 3D model of the environment is developed. There are two main challenges to be overcome. First is the detection of the line feature information in a spherical image and its abstraction into a descriptor that is compatible with the 3D line feature information in the model. Second is to evaluate similarity of the line feature information from the 2D image and that from arbitrary 6 DoF poses in the 3D environment model in order to localize the camera. To deal with the former, a randomized hough transform with spherical gradient-based filtering is used to accurately detect line features in the image and create a line feature descriptor. The same descriptor is created from arbitrary 6 DoF poses in the 3D model. Then, to deal with the latter, the Earth Mover's Distance (EMD) is used to evaluate their similarity. The proposed method was evaluated in a real environment with its 3D model. The results demonstrated that it can effectively estimate the 6 DoF pose of a spherical camera using a single image.

DOI： 10.2493/jjspe.83.1209

Scopus

researchmap

掲載種別：研究論文（学術雑誌）  

© 2016 IEEE. Spherical cameras are widely used due to their full 360° fields of view. However, a common but severe problem is that anything carrying the camera is always included in the view, occluding visual information. In this letter, we propose a novel method to remove such occlusions in videos taken from a freely moving spherical camera. Our method can recover the occluded background accurately in distorted spherical videos by inpainting the color and motion information of pixels. The missing color and motion information inside the occluded region is iteratively recovered in a coarse-to-fine optimization. Spatial and temporal coherence of color and motion information is enforced, considering spherical image geometry. Initially, feature-point matching is used to remove the effect of camera rotation in order to deal with large pixel displacements. Following this, the iterative optimization process is bootstrapped using a reliable estimate of motion information obtained by interpolating it from surrounding regions. We demonstrate its effectiveness by successfully completing videos and recovering occluded regions recorded in various practical situations and by quantifying it against other state-of-the-art methods.

DOI： 10.1109/LRA.2017.2718106

Scopus

researchmap

担当区分：筆頭著者   記述言語：英語   出版者・発行元：公益社団法人 計測自動制御学会  

Stereo vision is a well-known technique for vision-based 3D reconstruction of environments. Recently developed spherical cameras can be used to extend the concept to all 360&deg; and provide LIDAR-like 360 degree 3D data with color information. In order to perform accurate stereo disparity estimation, the accurate relative pose between the two cameras, represented by the five degree of freedom epipolar geometry, needs to be known. However, it is always tedious to mechanically align and/or calibrate such systems. We propose a technique to recover the complete five degree of freedom parameters of the epipolar geometry in a single minimization with a dense approach involving all the individual pixel displacements (optical flow) between two camera views. Taking advantage of the spherical image geometry, a non-linear least squares optimization based on the dense optical flow directly minimizes the angles between pixel displacements and epipolar curves in order to align them. This approach is particularly suitable for dense 3D reconstruction as the pixel-to-pixel disparity between the two images can be calculated accurately and converted to a dense point cloud. Further, there are no assumptions about the direction of camera displacement. We demonstrate this method by showing some error evaluations, examples of successfully rectified spherical stereo pairs, and the dense 3D models generated from them.

DOI： 10.9746/jcmsi.10.476

researchmap

掲載種別：研究論文（学術雑誌）  

© 2017, Fuji Technology Press. All rights reserved. Recently, many countries have faced serious problems associated with aging civil infra structures such as bridges, tunnels, dams, highways and so on. Aging infrastructures are increasing year by year and suitable maintenance actions are necessary to maintain their safety and serviceability. In fact, infrastructure deterioration has caused serious problems in the past. In order to prevent accidents with civil infrastructures, supervisors must spend a lot of money to maintain the safe conditions of infrastructures. Therefore, new technologies are required to reduce maintenance costs. In 2014 the Japanese government started the Cross-Ministerial Strategic Innovation Promotion Program (SIP), and technologies for infrastructure maintenance have been studied in the SIP project [1]. Fujitsu Limited, Hokkaido University, The University of Tokyo, Nagoya Institute of Technology and Docon Co. Limited have been engaged in the SIP project to develop a bridge inspection support system using information technology and robotic technology. Our system is divided into the following two main parts: bridge inspection support robots using a two-wheeled multicopter, and an inspection data management system utilizing 3D modeling technology. In this paper, we report the bridge inspection support system developed in our SIP project.

DOI： 10.20965/jdr.2017.p0593

Scopus

researchmap

担当区分：筆頭著者   掲載種別：研究論文（学術雑誌）  

© 2017, Fuji Technology Press. All rights reserved. We propose a method for stabilizing spherical videos by estimating and removing the effect of camera rotation using dense optical flow fields. By derotating each frame in the video to the orientation of its previous frame in two dense approaches, we estimate the complete 3 DoF rotation of the camera and remove it to stabilize the spherical video. Following this, any chosen area on the spherical video (equivalent of a normal camera’s field of view) is unwarped to result in a ‘rotation-less virtual camera’ that can be oriented independent of the camera motion. This can help in perception of the environment and camera motion much better. In order to achieve this, we use dense optical flow, which can provide important information about camera motion in a static environment and can have several advantages over sparse feature-point based approaches. The spatial regularization property of dense optical flow provides more stable motion information as compared to tracking sparse points and negates the effect of feature point outliers. We show superior results as compared to using sparse feature points alone.

DOI： 10.20965/jrm.2017.p0566

Scopus

researchmap

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2016 IEEE. Gradient histogram based descriptors, that are constructed using the gradient magnitudes as votes to orientation bins, are successfully used for Pedestrian Detection. However, their performance is hampered when presented with datasets having many variations in properties such as appearance, texture, scale, background, and object pose. Such variations can be reduced by smoothing the images. But, the performance of the descriptors, and their classifiers is affected negatively by this, due to the loss of important gradients along with the noisy ones. In this work, we show that the ranks of gradient magnitudes stay resilient to such a smoothing. We show that a combination of image smoothing and the ranks of gradient magnitudes yields good detection performances, especially when the variations in a dataset are large or the number of training samples is less. Experiments on the challenging Caltech and Daimler Pedestrian datasets, and the Inria Person dataset illustrate these findings.

DOI： 10.1109/CVPRW.2016.150

Scopus

researchmap

掲載種別：研究論文（国際会議プロシーディングス）  

© 2016 IEEE. Spherical cameras are widely used for robot perception because of their full 360 degree fields of view. However, the robot body is always seen in the view, causing occlusions. In this paper, we propose a video completion method which is able to remove occlusions fast and recover the occluded background accurately. We first estimate motion in a 2D dense spherical optical flow field. Then we interpolate the motion in the occlusion regions by solving least square minimization problems using polynomial models. Based on the interpolated motion, we recover the occluded regions by tracing the optical flow trajectory to find the corresponding pixels in other frames and warping them back to fill in the occlusions. We also provide a simple, yet fast solution to effectively remove occlusions in all regions of the image by utilizing the continuity of field of view of spherical images. In experiments, quantitative methods are conducted to demonstrate the effectiveness and efficiency of our proposed method by comparing the results with those from state-of-the-art methods.

DOI： 10.1109/ROBIO.2016.7866353

Scopus

researchmap

担当区分：筆頭著者   掲載種別：研究論文（国際会議プロシーディングス）  

© 2016 IEEE. In this research, a technique for dense 3D reconstruction from two spherical images clicked at displaced positions near a large structure is formulated. The technique is based on the use of global variational information i.e. The dense optical flow field in a unique rectification based refinement of the epipolar geometry between the two images in a vertically displaced orientation. A non-linear minimization is used to globally align the 2D equirectangular optical flow field (i.e. pixel displacements). The magnitude component of the resultant optical flow field can directly be converted to a dense 3D reconstruction. Thus, the epipolar geometry as well as the 3D structure can be estimated in a single minimization. This method could be useful in measurement and reconstruction of large structures such as bridges, etc. using a robot equipped with a spherical camera, thus helping in their inspection and maintenance.

DOI： 10.1109/IST.2016.7738212

Scopus

researchmap

担当区分：筆頭著者   掲載種別：研究論文（国際会議プロシーディングス）  

© 2016 IEEE. In camera-equipped teleoperated robots, it is often tedious for the operator to manage both the viewpoint and the shaky/unstable navigation, leading to disorientation. Our proposal is to create a virtual, freely rotatable camera that is decoupled from the robot's rotation. It is implemented using a complete spherical camera and removing its rotation in-image with a novel algorithm based on aligning the dense spherical optical flow field along the epipolar direction. Finally, any area on the rotation-less image sequence can be undistorted, resulting in the desired decoupled camera. We illustrate the concept by showing the effect on some videos taken from a spherical camera under different robot motions.

DOI： 10.1109/ICIP.2016.7533209

Scopus

researchmap

担当区分：筆頭著者   掲載種別：研究論文（国際会議プロシーディングス）  

© 2016 Institute of Control, Robotics and Systems - ICROS. In this research, a method for dense 3D reconstruction of structures from small motion of a spherical camera is proposed. Spherical cameras can capture information from all directions enabling measurement of the entire surrounding structure at once. The proposed technique uses two spherical images clicked at slightly displaced positions near the structure, followed by a combination of feature-point matching and dense optical flow. Feature-point matching between two images alone is usually not accurate to give a dense point cloud because of outliers. Moreover, calculation of the epipolar direction with feature point matching is susceptible to noise with small displacements. However, spherical cameras have unique parallax properties allowing use of dense, global information. Taking advantage of this, the global, dense optical flow field is used. The epipolar geometry is densely optimized based on the optical flow field for an accurate 3D reconstruction. A possible use of this research could be to measure large infrastructures (bridges, tunnels, etc.) with minimal robot motion.

DOI： 10.1109/ICCAS.2016.7832307

Scopus

researchmap

記述言語：英語   出版者・発行元：一般社団法人日本機械学会  

Omnidirectional cameras are widely used for robot inspection for their wild fields of view. However, the robot body will always be included in the view, causing occlusions. This paper deals with one such example of occlusion and proposes an inpainting based solution to remove it. Our method could generate a clean video automatically, without the need for a manually given mask. We propose an approximate shape fitting method combined with color information to generate the mask of robot body occlusion and followed by video inpainting. In experiments, the effectiveness of our proposed method is demonstrated by successfully removing robot body occlusions in omnidirectional videos.

DOI： 10.1299/jsmeicam.2015.6.49

CiNii Books

researchmap

担当区分：筆頭著者   記述言語：英語   出版者・発行元：一般社団法人日本機械学会  

Omnidirectional cameras are often used with robots to provide an immersive view of the surroundings. However, robots often have unstable motion and undergo rotation. In this work, we formulate a method to stabilize the viewpoint of Omnidirectional videos by removing the rotation using dense optical flow fields. The method works by first projecting each Omnidirectional video frame on a unit sphere, measuring the optical flow at every point on the sphere, and finding the direction that minimizes its rotational component in a frame by frame manner. The Omnidirectional video is de-rotated and a 'rotation-less, translation-only' viewpoint is generated. Such a technique is well suited to work in any environment, even with sparse texture or repeating patterns where feature correspondence based methods may fail.

DOI： 10.1299/jsmeicam.2015.6.51

CiNii Books

researchmap

開催年月日： 2016年9月 　 　

researchmap

開催年月日： 2015年9月 　 　

researchmap