Patent US10633007

Autonomous driving assistance glasses that assist in autonomous driving by recognizing humans' status and driving environment through image analysis based on deep neural network
[+] Show Details

US10633007

1. A method for providing safe-driving information via assistance glasses worn by a driver, comprising steps of: (8) (2)

(a) if at least one visual-dependent driving image corresponding to perspective of the driver taken from at least one camera installed on the assistance glasses worn by the driver of a vehicle, acceleration information and gyroscope information from one or more sensors installed on the assistance glasses are acquired, a safe-driving information analyzing device performing (i) a process of inputting the visual-dependent driving image into a convolution network, to thereby allow the convolution network to generate at least one feature map by applying convolution operation to the visual-dependent driving image, and a process of inputting the feature map respectively into a detection network, a segmentation network, and a recognition network, to thereby allow the detection network to detect at least one object located on the visual-dependent driving image by using the feature map, allow the segmentation network to detect one or more lanes on the visual-dependent driving image, and allow the recognition network to detect driving environment corresponding to the visual-dependent driving image, (ii) a process of inputting the acceleration information and the gyroscope information into a recurrent network, to thereby allow the recurrent network to generate status information on the driver corresponding to the acceleration information and the gyroscope information; and

(b) the safe-driving information analyzing device performing (i) a process of notifying the driver of information on an estimated probability of a collision between the vehicle and the object via an output unit of the assistance glasses by referring to the object detected by the detection network, a process of notifying the driver of lane departure information on the vehicle via the output unit by referring to the lanes detected by the segmentation network, and a process of notifying the driver of the driving environment detected by the recognition network via the output unit, and (ii) a process of giving a safe-driving warning to the driver via the output unit by referring to the status information on the driver detected by the recurrent network.

2. The method of claim 1, wherein the safe-driving information analyzing device performs a process of inputting the feature map into the detection network, to thereby allow the detection network to (i) generate proposal boxes, corresponding to one or more regions where the object is estimated as located, on the feature map by using a region proposal network, (ii) generate at least one object feature vector by applying pooling operation to each of one or more areas, corresponding to each of the proposal boxes, on the feature map via a pooling layer, and (iii) generate multiple pieces of object information corresponding to the proposal boxes by applying object-detecting fully-connected operation to the object feature vector via an object-detecting fully-connected layer, and as a result, a process of detecting the object located on the visual-dependent driving image. (0)

3. The method of claim 1, wherein the safe-driving information analyzing device performs a process of inputting the feature map into the segmentation network, to thereby allow the segmentation network to generate at least one deconvolutional feature map by applying deconvolution operation to the feature map via at least one deconvolutional layer, and a process of inputting the deconvolutional feature map into at least one lane-detecting fully-connected layer, to thereby allow the lane-detecting fully-connected layer to detect the lanes located on the visual-dependent driving image by applying lane-detecting fully-connected operation to the deconvolutional feature map. (0)

4. The method of claim 1, wherein the safe-driving information analyzing device performs a process of inputting the feature map into the recognition network, to thereby allow the recognition network to (i) generate at least one global feature vector by applying global-pooling operation to the feature map via a global-pooling layer, and (ii) generate driving environment information by applying recognition fully-connected operation to the global feature vector via a recognition fully-connected layer, and as a result, a process of detecting the driving environment corresponding to the visual-dependent driving image. (0)

5. The method of claim 1, wherein the safe-driving information analyzing device performs a process of inputting the acceleration information and the gyroscope information into the recurrent network, to thereby allow the recurrent network to generate the status information on the driver corresponding to behavioral patterns of the driver by applying recurrent operation to one or more change statuses of the acceleration information and the gyroscope information during a preset time period by using one or more Long Short Term Memories (LSTM). (0)

6. The method of claim 1, wherein the safe-driving information analyzing device performs a process of adjusting transparency of lenses of the assistance glasses in response to illumination information acquired from at least one illumination sensor installed on the assistance glasses. (0)

7. The method of claim 1, wherein the output unit includes (i) at least one speaker to be positioned at a location corresponding to at least one ear of the driver if the assistance glasses are worn by the driver and (ii) at least one Virtual Retinal Display (VRD) to be positioned at a location corresponding to at least one eye of the driver if the assistance glasses are worn by the driver. (0)

8. The method of claim 1, wherein the convolution network, the detection network, the segmentation network, and the recognition network have been learned by a 1-st learning device repeating (i) a process of inputting at least one visual-dependent driving image for training into the convolution network, to thereby allow the convolution network to generate at least one feature map for training by applying convolution operation using at least one previously learned convolution parameter to the visual-dependent driving image for training, (ii) (ii-1) a process of inputting the feature map for training into the detection network, to thereby allow (ii-1-1) a region proposal network of the detection network to generate one or more proposal boxes for training, corresponding to one or more regions where at least one object for training is estimated as located, on the feature map for training, (ii-1-2) a pooling layer of the detection network to generate at least one object feature vector for training by applying pooling operation to one or more regions, corresponding to each of the proposal boxes for training, on the feature map for training, and (ii-1-3) an object-detecting fully-connected layer of the detection network to generate multiple pieces of object information for training corresponding to the proposal boxes for training by applying object-detecting fully-connected operation using at least one previously learned object-detection parameter to the object feature vector for training, (ii-2) a process of inputting the feature map for training into the segmentation network, to thereby allow (ii-2-1) a deconvolutional layer of the segmentation network to generate at least one deconvolutional feature map by applying deconvolution operation using at least one previously learned deconvolution parameter to the feature map for training, and (ii-2-2) a lane-detecting fully-connected layer to detect one or more lanes for training located on the visual-dependent driving image for training by applying lane-detecting fully-connected operation using at least one previously learned lane-detection parameter to the deconvolutional feature map, (ii-3) a process of inputting the feature map for training into the recognition network, to thereby allow (ii-3-1) a global-pooling layer of the recognition network to generate at least one global feature vector for training by applying global-pooling operation to the feature map for training, and (ii-3-2) a recognition fully-connected layer of the recognition network to generate driving environment information for training by applying recognition fully-connected operation using at least one previously learned recognition parameter to the global feature vector for training, and (iii) (iii-1) a process of updating the previously learned object-detection parameter of the object-detecting fully-connected layer such that one or more 1-st losses are minimized which are outputted from a 1-st loss layer by referring to the multiple pieces of the object information for training and their corresponding 1-st ground truths, (iii-2) a process of updating at least one of the previously learned lane-detection parameter of the lane-detecting fully-connected layer and the previously learned deconvolution parameter of the deconvolutional layer such that one or more 2-nd losses are minimized which are outputted from a 2-nd loss layer by referring to the lanes for training and their corresponding 2-nd ground truths, (iii-3) a process of updating the previously learned recognition parameter of the recognition fully-connected layer such that one or more 3-rd losses are minimized which are outputted from a 3-rd loss layer by referring to the driving environment information for training and its corresponding 3-rd ground truth, and (iii-4) a process of updating the previously learned convolution parameter of the convolution network such that at least one 4-th loss is minimized which is created by weighted summation of the 1-st losses, the 2-nd losses, and the 3-rd losses or their processed values. (0)

9. The method of claim 1, wherein the recurrent network has been learned by a 2-nd learning device performing a process of inputting acceleration information for training and gyroscope information for training from a current point of time t to a previous point of time (t−k) into each of LSTMs, to thereby allow each of the LSTMs to output driving status information for training corresponding to behavioral patterns of the driver by applying forward operation to the acceleration information for training and the gyroscope information for training from the current point of time t to the previous point of time (t-k), and a process of instructing a 4-th loss layer to adjust one or more parameters of the LSTMs such that one or more 5-th losses are minimized which are created by referring to the driving status information for training and its corresponding 4-th ground truth. (0)

10. A safe-driving information analyzing device for providing safe-driving information via assistance glasses worn by a driver, comprising: (8) (2)

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to perform or support another device to perform: (I) if at least one visual-dependent driving image corresponding to perspective of the driver taken from at least one camera installed on the assistance glasses worn by the driver of a vehicle, acceleration information and gyroscope information from one or more sensors installed on the assistance glasses are acquired, (I-1) a process of inputting the visual-dependent driving image into a convolution network, to thereby allow the convolution network to generate at least one feature map by applying convolution operation to the visual-dependent driving image, and a process of inputting the feature map respectively into a detection network, a segmentation network, and a recognition network, to thereby allow the detection network to detect at least one object located on the visual-dependent driving image by using the feature map, allow the segmentation network to detect one or more lanes on the visual-dependent driving image, and allow the recognition network to detect driving environment corresponding to the visual-dependent driving image, (I-2) a process of inputting the acceleration information and the gyroscope information into a recurrent network, to thereby allow the recurrent network to generate status information on the driver corresponding to the acceleration information and the gyroscope information, and (II) (II-1) a process of notifying the driver of information on an estimated probability of a collision between the vehicle and the object via an output unit of the assistance glasses by referring to the object detected by the detection network, a process of notifying the driver of lane departure information on the vehicle via the output unit by referring to the lanes detected by the segmentation network, and a process of notifying the driver of the driving environment detected by the recognition network via the output unit, and (II-2) a process of giving a safe-driving warning to the driver via the output unit by referring to the status information on the driver detected by the recurrent network.

11. The safe-driving information analyzing device of claim 10, wherein the processor performs a process of inputting the feature map into the detection network, to thereby allow the detection network to (i) generate proposal boxes, corresponding to one or more regions where the object is estimated as located, on the feature map by using a region proposal network, (ii) generate at least one object feature vector by applying pooling operation to each of one or more areas, corresponding to each of the proposal boxes, on the feature map via a pooling layer, and (iii) generate multiple pieces of object information corresponding to the proposal boxes by applying object-detecting fully-connected operation to the object feature vector via an object-detecting fully-connected layer, and as a result, a process of detecting the object located on the visual-dependent driving image. (0)

12. The safe-driving information analyzing device of claim 10, wherein the processor performs a process of inputting the feature map into the segmentation network, to thereby allow the segmentation network to generate at least one deconvolutional feature map by applying deconvolution operation to the feature map via at least one deconvolutional layer, and a process of inputting the deconvolutional feature map into at least one lane-detecting fully-connected layer, to thereby allow the lane-detecting fully-connected layer to detect the lanes located on the visual-dependent driving image by applying lane-detecting fully-connected operation to the deconvolutional feature map. (0)

13. The safe-driving information analyzing device of claim 10, wherein the processor performs a process of inputting the feature map into the recognition network, to thereby allow the recognition network to (i) generate at least one global feature vector by applying global-pooling operation to the feature map via a global-pooling layer, and (ii) generate driving environment information by applying recognition fully-connected operation to the global feature vector via a recognition fully-connected layer, and as a result, a process of detecting the driving environment corresponding to the visual-dependent driving image. (0)

14. The safe-driving information analyzing device of claim 10, wherein the processor performs a process of inputting the acceleration information and the gyroscope information into the recurrent network, to thereby allow the recurrent network to generate the status information on the driver corresponding to behavioral patterns of the driver by applying recurrent operation to one or more change statuses of the acceleration information and the gyroscope information during a preset time period by using one or more Long Short Term Memories (LSTM). (0)

15. The safe-driving information analyzing device of claim 10, wherein the processor performs a process of adjusting transparency of lenses of the assistance glasses in response to illumination information acquired from at least one illumination sensor installed on the assistance glasses. (0)

16. The safe-driving information analyzing device of claim 10, wherein the output unit includes (i) at least one speaker to be positioned at a location corresponding to at least one ear of the driver if the assistance glasses are worn by the driver and (ii) at least one Virtual Retinal Display (VRD) to be positioned at a location corresponding to at least one eye of the driver if the assistance glasses are worn by the driver. (0)

17. The safe-driving information analyzing device of claim 10, wherein the convolution network, the detection network, the segmentation network, and the recognition network have been learned by a 1-st learning device repeating (i) a process of inputting at least one visual-dependent driving image for training into the convolution network, to thereby allow the convolution network to generate at least one feature map for training by applying convolution operation using at least one previously learned convolution parameter to the visual-dependent driving image for training, (ii) (ii-1) a process of inputting the feature map for training into the detection network, to thereby allow (ii-1-1) a region proposal network of the detection network to generate one or more proposal boxes for training, corresponding to one or more regions where at least one object for training is estimated as located, on the feature map for training, (ii-1-2) a pooling layer of the detection network to generate at least one object feature vector for training by applying pooling operation to one or more regions, corresponding to each of the proposal boxes for training, on the feature map for training, and (ii-1-3) an object-detecting fully-connected layer of the detection network to generate multiple pieces of object information for training corresponding to the proposal boxes for training by applying object-detecting fully-connected operation using at least one previously learned object-detection parameter to the object feature vector for training, (ii-2) a process of inputting the feature map for training into the segmentation network, to thereby allow (ii-2-1) a deconvolutional layer of the segmentation network to generate at least one deconvolutional feature map by applying deconvolution operation using at least one previously learned deconvolution parameter to the feature map for training, and (ii-2-2) a lane-detecting fully-connected layer to detect one or more lanes for training located on the visual-dependent driving image for training by applying lane-detecting fully-connected operation using at least one previously learned lane-detection parameter to the deconvolutional feature map, (ii-3) a process of inputting the feature map for training into the recognition network, to thereby allow (ii-3-1) a global-pooling layer of the recognition network to generate at least one global feature vector for training by applying global-pooling operation to the feature map for training, and (ii-3-2) a recognition fully-connected layer of the recognition network to generate driving environment information for training by applying recognition fully-connected operation using at least one previously learned recognition parameter to the global feature vector for training, and (iii) (iii-1) a process of updating the previously learned object-detection parameter of the object-detecting fully-connected layer such that one or more 1-st losses are minimized which are outputted from a 1-st loss layer by referring to the multiple pieces of the object information for training and their corresponding 1-st ground truths, (iii-2) a process of updating at least one of the previously learned lane-detection parameter of the lane-detecting fully-connected layer and the previously learned deconvolution parameter of the deconvolutional layer such that one or more 2-nd losses are minimized which are outputted from a 2-nd loss layer by referring to the lanes for training and their corresponding 2-nd ground truths, (iii-3) a process of updating the previously learned recognition parameter of the recognition fully-connected layer such that one or more 3-rd losses are minimized which are outputted from a 3-rd loss layer by referring to the driving environment information for training and its corresponding 3-rd ground truth, and (iii-4) a process of updating the previously learned convolution parameter of the convolution network such that at least one 4-th loss is minimized which is created by weighted summation of the 1-st losses, the 2-nd losses, and the 3-rd losses or their processed values. (0)

18. The safe-driving information analyzing device of claim 10, wherein the recurrent network has been learned by a 2-nd learning device performing a process of inputting acceleration information for training and gyroscope information for training from a current point of time t to a previous point of time (t-k) into each of LSTMs, to thereby allow each of the LSTMs to output driving status information for training corresponding to behavioral patterns of the driver by applying forward operation to the acceleration information for training and the gyroscope information for training from the current point of time t to the previous point of time (t−k), and a process of instructing a 4-th loss layer to adjust one or more parameters of the LSTMs such that one or more 5-th losses are minimized which are created by referring to the driving status information for training and its corresponding 4-th ground truth. (0)

19. Assistance glasses for providing a driver with safe-driving information, comprising: (2) (4)

the assistance glasses wearable by the driver;

one or more sensors, including a camera for taking at least one visual-dependent driving image corresponding to perspective of the driver, an acceleration sensor, and a gyroscope sensor, which are installed on the assistance glasses; and

an output unit for providing the driver with the safe-driving information, of the assistance glasses;

wherein the assistance glasses includes a safe-driving information analyzing device for performing (I) (I-1) a process of inputting the visual-dependent driving image, acquired from the camera, into a convolution network, to thereby allow the convolution network to generate at least one feature map by applying convolution operation to the visual-dependent driving image, and a process of inputting the feature map respectively into a detection network, a segmentation network, and a recognition network, to thereby allow the detection network to detect at least one object located on the visual-dependent driving image by using the feature map, allow the segmentation network to detect one or more lanes on the visual-dependent driving image, and allow the recognition network to detect driving environment corresponding to the visual-dependent driving image, (I-2) a process of inputting acceleration information acquired from the acceleration sensor and gyroscope information acquired from the gyroscope sensor into a recurrent network, to thereby allow the recurrent network to generate status information on the driver corresponding to the acceleration information and the gyroscope information, and (II) (II-1) a process of notifying the driver of information on an estimated probability of a collision between a vehicle of the driver and the object via the output unit of the assistance glasses by referring to the object detected by the detection network, a process of notifying the driver of lane departure information on the vehicle of the driver via the output unit by referring to the lanes detected by the segmentation network, and a process of notifying the driver of the driving environment detected by the recognition network via the output unit, and (II-2) a process of giving a safe-driving warning to the driver via the output unit by referring to the status information on the driver detected by the recurrent network.

20. The assistance glasses of claim 19, wherein the sensors further include at least one illumination sensor installed on the assistance glasses, and (0) (1)

21. The assistance glasses of claim 19, wherein the output unit includes (i) at least one speaker to be positioned at a location corresponding to at least one ear of the driver if the assistance glasses are worn by the driver and (ii) at least one Virtual Retinal Display (VRD) to be positioned at a location corresponding to at least one eye of the driver if the assistance glasses are worn by the driver. (0)

View Abstract and Specification Size

PDF with Images and Document Face >

Full Text Publication >

Patent US10633007

Autonomous driving assistance glasses that assist in autonomous driving by recognizing humans' status and driving environment through image analysis based on deep neural network

Patent Matrix® Search