AI System Recognition physical object, traffic jams throught sensor, camera, controller by human
Combination of areas as robotics, self-driving cars, motion planning, sensor fusion, Lidar, radar, camera, IMU, GPS, Apollo, embedded system, machine learning, system architecture, connectivity, real-time system
Operation System:
- QNX: https://blackberry.qnx.com/en
- ROS/ROS2: http://www.ros.org/
-
ROS for self-driving
- Learn ROS for Self-Driving Cars
- Linux
Embedded Linux: References https://www.engineersgarage.com/tutorials/embedded-linux-basics-programming
Realtime communication (RTC)
Robottics: Là nghành robot học nghiên cứu robot có những chức năng như con người (đứa trẻ)
Hardware in loop: Là mô phỏng phát triển và kiểm thử hệ thống real time phức tạp mà không cần quá trình điều khiển và dây chuyền thực tế (bao gồm phần cứng và phần mềm) tức là chuyển dây chuyền xử lý trong phòng thí nghiệm như:
- Thử nghiệm phần cứng và phần mềm điều khiển ở điều kiện môi trường tới hạn trong phòng thí nghiệm (như nhiệt độ cao/thấp, gia tốc lớn và các cú sốc cơ học, thiết bị kích thích, tính tương thích điện từ).
- Thử nghiệm các tác động của lỗi và tình trạng không mong đợi của cơ cấu chấp hành, cảm biến và máy tính trên toàn bộ hệ thống.
- Vận hành và thử nghiệm các điều kiện vận hành tới hạn và nguy hiểm.
- Các thử nghiệm tái sinh, có thể lặp lại thường xuyên.
- Vận hành dễ dàng với các giao diện người-máy khác nhau như buồng tập lái mô phỏng máy bay
Bạn nên hiểu bản chất của những phương pháp xử lý ảnh camera để tạo sự thú vị và yêu thích môn học về xe không người lái.
Là thuật toán dò các cạnh được John F. Canny đưa ra vào năm 1986
Hough Space Transformation of the Line Detection
In image space, a line is plotted as x vs. y, but in 1962, Paul Hough devised a method for representing lines in parameter space, which we will call Hough space in his honor. Read more about Hough Transform
In Hough space, I can represent my x vs. y line as a point in m vs. b instead. The Hough Transform is just the conversion from image space to Hough space. So, the characterization of a line in image space will be a single point at the position (m, b) in Hough space.
- Color Selection
- Canny Edge Detection
- Detect the edges
- Region of Interest Selection
- Extrapolation in Land Lines
- Hough Transform Line Detection
các hình ảnh input đầu vào
import numpy as np
import cv2
def select_rgb_white_yellow(image):
# white color mask
lower = np.uint8([200, 200, 200])
upper = np.uint8([255, 255, 255])
white_mask = cv2.inRange(image, lower, upper)
# yellow color mask
lower = np.uint8([190, 190, 0])
upper = np.uint8([255, 255, 255])
yellow_mask = cv2.inRange(image, lower, upper)
# combine the mask
mask = cv2.bitwise_or(white_mask, yellow_mask)
masked = cv2.bitwise_and(image, image, mask = mask)
return masked
image = select_rgb_white_yellow("/Resource/test_image.png")
cv2.imshow('test_image', image)
import numpy as np
import cv2
# HSV Color Space
def convert_hsv(image):
return cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
image = convert_hsv("/Resource/test_image.png")
cv2.imshow('test_image', image)
import numpy as np
import cv2
# HLS Colour Space
def convert_hls(image):
return cv2.cvtColor(image, cv2.COLOR_RGB2HLS)
image = convert_hsv("/Resource/test_image.png")
cv2.imshow('test_image', image)
- cv2.inRange: Filter the white color and the yellow color seperately. The function returns 255 when the filter conditon is satisfied. Otherwise, it returns 0.
- cv2.bitwise_or to combine these two binary masks. The combined mask returns 255 when either white or yellow color is detected.
- cv2.bitwise_and to apply the combined mask onto the original RGB image
import os, glob
import numpy as np
import cv2
def select_white_yellow(image):
converted = convert_hls(image)
lower = np.uint8([ 0, 200, 0])
upper = np.uint8([255, 255, 255])
white_mask = cv2.inRange(converted, lower, upper)
lower = np.uint8([ 10, 0, 100])
upper = np.uint8([ 40, 255, 255])
yellow_mask = cv2.inRange(converted, lower, upper)
mask = cv2.bitwise_or(white_mask, yellow_mask)
return cv2.bitwise_and(image, image, mask = mask)
test_images = []
test_images = [plt.imread(path) for path in glob.glob('test_images/*.jpg')]
white_yellow_images = list(map(select_white_yellow, test_images))
- use cv2.cvtColor to convert images into gray scale
- use cv2.GaussianBlur to smooth out rough edges
- use cv2.Canny to find edges
Converted into gray scaled ones in order to detect shapes (edges) in the images. This is because the Canny edge detection measures the magnitude of pixel intensity changes or gradients. I converted the white and yellow line images from the above into gray scale for edge detection.
def convert_gray_scale(image):
return cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
gray_images = list(map(convert_gray_scale, white_yellow_images))
The above images have many rough edges which causes many noisy edges to be detected. We use cv2.GaussianBlur to smooth out edges.
def apply_smoothing(image, kernel_size=15):
# kernel_size must be postivie and odd
# kernel_size value requires more time to process
return cv2.GaussianBlur(image, (kernel_size, kernel_size), 0)
blurred_images = list(map(lambda image: apply_smoothing(image), gray_images))
- If a pixel gradient is higher than the upper threshold, the pixel is accepted as an edge
- If a pixel gradient value is below the lower threshold, then it is rejected.
- If the pixel gradient is between the two thresholds, then it will be accepted only if it is connected to a pixel that is above the upper threshold.
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import cv2
def show_images(images):
cols = 3
rows = 2
plt.figure(figsize=(15, 5))
for i in range(0, len(images)):
plt.subplot(rows, cols, i+1)
plt.imshow(images[i], cmap="gray")
plt.show()
def canny(image):
low_threshold = 50
high_threshold = 150
return cv2.Canny(image, low_threshold, high_threshold)
if __name__ == "main":
edge_images = list(map(lambda image: canny(image), blurred_images))
show_images(edge_images)
When finding lane lines, we don't need to check the sky and the hills.
def filter_region(image, vertices):
# Create the mask using the vertices and apply it to the input image
mask = np.zeros_like(image)
# Defining a 3 channel, 2 channel or 1 channel color to fill the mask with depending on the input image
if len(mask.shape) == 2:
cv2.fillPoly(mask, vertices, 255)
else:
cv2.fillPoly(mask, vertices, (255,)*mask.shape[2]) # in case, the input image has a channel dimension
return cv2.bitwise_and(image, mask)
def select_region(image):
# first, define the polygon by vertices
rows, cols = image.shape[:2]
bottom_left = [cols*0.1, rows*0.95]
top_left = [cols*0.4, rows*0.6]
bottom_right = [cols*0.9, rows*0.95]
top_right = [cols*0.6, rows*0.6]
# the vertices are an array of polygons (i.e array of arrays) and the data type must be integer
vertices = np.array([[bottom_left, top_left, top_right, bottom_right]], dtype=np.int32)
return filter_region(image, vertices)
roi_images = list(map(select_region, edge_images))
show_images(roi_images)
-
Using cv2.HoughLinesP to detect lines in the edge images
-
Several parameters of the above function:
- rho: Distance resolution of the accumulator in pixels.
- theta: Angle resolution of the accumulator in radians.
- threshold: Accumulator threshold parameter. Only those lines are returned that get enough votes (> threshold).
- minLineLength: Minimum line length. Line segments shorter than that are rejected.
- maxLineGap: Maximum allowed gap between points on the same line to link them.
import matplotlib.pyplot as plt import matplotlib.image as mpimg import numpy as np import cv2 import os, glob def draw_lines(img, lines, color = [255, 0, 0], thickness = 2): imgCopy = np.copy(img) for line in lines: for x1,y1,x2,y2 in line: cv2.line(imgCopy, (x1, y1), (x2, y2), color, thickness) return imgCopy def hough_lines(img): rho = 1 theta = np.pi/180 threshold = 20 min_line_length = 20 max_line_gap = 300 return cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_length, maxLineGap=max_line_gap) if __name__ == '__main__': list_of_lines = list(map(hough_lines, roi_images)) line_images = [] # zip() used for mapping similar index, value of multiple containers of zip() compress for image, lines in zip(test_images, list_of_lines): line_images.append(draw_lines(image, lines)) show_images(line_images)
These lines are not smooth in image. Because there are multiple lines in images. We should get them and add an average line instead of them.
We'll collect positive slope lines and negative slope lines separately and take averages.
def average_slope_intercept(lines):
left_lines = [] # (slope, intercept)
left_weights = [] # (length,)
right_lines = [] # (slope, intercept)
right_weights = [] # (length,)
for line in lines:
for x1, y1, x2, y2 in line:
if x2==x1:
continue # ignore a vertical line
# Do doc
slope = (y2-y1)/(x2-x1)
# Mat phang duong
intercept = y1 - slope*x1
# Dien tich cua line bao quanh
length = np.sqrt((y2-y1)**2+(x2-x1)**2)
if slope < 0: # y is reversed in image
left_lines.append((slope, intercept))
left_weights.append((length))
else:
right_lines.append((slope, intercept))
right_weights.append((length))
# add more weight to longer lines
left_lane = np.dot(left_weights, left_lines) /np.sum(left_weights) if len(left_weights) >0 else None
right_lane = np.dot(right_weights, right_lines)/np.sum(right_weights) if len(right_weights)>0 else None
return left_lane, right_lane # (slope, intercept), (slope, intercept)
def lane_lines(image, lines):
left_lane, right_lane = average_slope_intercept(lines)
y1 = image.shape[0] # bottom of the image
y2 = y1*0.6 # slightly lower than the middle
# Convert a line represented in slope and intercept into pixel points
if (left_lane is None) or (right_lane is None) :
raise Exception('not detecting')
slope, intercept = left_lane
x1 = int((y1 - intercept)/slope)
x2 = int((y2 - intercept)/slope)
y1 = int(y1)
y2 = int(y2)
left_line = ((x1, y1), (x2, y2))
slope, intercept = right_lane
x1 = int((y1 - intercept)/slope)
x2 = int((y2 - intercept)/slope)
y1 = int(y1)
y2 = int(y2)
right_line = ((x1, y1), (x2, y2))
return left_line, right_line
def draw_lane_lines(image, lines, color=[255, 0, 0], thickness=20):
line_image = np.zeros_like(image)
for line in lines:
if line is not None:
cv2.line(line_image, *line, color, thickness)
return cv2.addWeighted(image, 1.0, line_image, 0.95, 0.0)
if __name__ == '__main__':
lane_images = []
for image, lines in zip(test_images, list_of_lines):
lane_images.append(draw_lane_lines(image, lane_lines(image, lines)))
show_images(lane_images)
def process_image(image):
try :
white_yellow_image = select_white_yellow(image)
gray_image = convert_gray_scale(image)
blurred_image = apply_smoothing(gray_image)
edge_image = canny(blurred_image)
roi_image = select_region(edge_image)
list_of_lines = hough_lines(roi_image)
result = draw_lane_lines(image, lane_lines(image, list_of_lines))
return result
except Exception as error:
return image
# Example: Resource/*.jpg
test_images = [plt.imread(path) for path in glob.glob('Resource/*.jpg')]
lane_images = []
for image in test_images :
lane_images.append(process_image(image))
show_images(lane_images)