Cosi119a

## Logistics

* Oct 29: Team Project Proposals
* Oct 31: OS Quiz on October 31
* Oct 31: Line Follower Programming Assignment

<slide_break></slide_break>

## Computer Vision

<slide_break></slide_break>

### Seeing ==? Vision

* We focus here on cameras
* Ignore other kinds of "seeing" such as touch sensors or LIDAR.
* A lot more than taking pictures
* Can be used for:
  1. Mapping
  2. Localization
  3. Object recognition
  4. Trajectory Estimation
  5. Driver Distraction
  6. Redundancy

<slide_break></slide_break>

### Camera Types: Regular (webcam-like)

1. RGB images
1. Video is basically a stream of images
1. Frame rate: How many images sent per second
1. The higher, the more bandwidth and CPU is used
1. Usually it is treated just that way
1. Each image is analyzed individually

<slide_break></slide_break>

### Camera features and characteristics

* Field of view (wide angle, narrow angle)
* Mount position (pose)
* Stationary or movable
* Means of connection to the robot

<slide_break></slide_break>

### Types of cameras

* Basic RaspiCam (webcam-like)
* Mobile phone camera
* Depth cameras
* Infrared cameras
* ["AI Cameras"](https://www.raspberrypi.com/products/ai-camera/)

<slide_break></slide_break>

### Considerations

* Resolution of image
* Power requirements
* Fixed direction or sometimes a swivel
* ROS needs to know how the cameras position and direction relates to the overall robot base
* Another job for TFs
* What would happen if this information is incorrect?
* Where do the images have to be "sent" to?
* Compression factors
* Where is the image processed
* Wifi and other bandwidth

<slide_break></slide_break>

### Connections

* USB, direct to Raspberry Pi
* Image needs to be viewed through unix utility
* Which is not always easy
* Bandwidth and speed of connection

<slide_break></slide_break>

### Role of TFs

* What is the likely TF that would be needed?
* What would that tell us?

<slide_break></slide_break>

### Computer Vision (CV)

* Recognizing: faces, locations, fiducials, gestures
* Image processing: filtering colors, isolating shapes etc.
* Machine Learning (ML): Statistical analysis of tagged images

<slide_break></slide_break>

### Code

<div class="callout callout-small">
  <span class="callout-badge">Note</span>Burger TB3 in Gazebo does not have a camera! Must use Waffle
</div>

* `roslaunch samples linemission.launch model:=waffle`
* in `samples`, cvexample.py

<slide_break></slide_break>

### Image Properties

* Images are stored in arrays
* Top left pixel is 0,0
* Bottom right pixel is m,n
* Each element of this array is another array, with the values for that pixel
* And that depends on the image encoding.

<slide_break></slide_break>

### Image Encodings

* Usually each component is an integer from 0-255
* Zero means none of that component; 255 means the maximum of that component
* The first three components determine the color
* There can be more components, e.g. distance, or transparency, or others.
* Generally you can freely convert between encodings

<slide_break></slide_break>

### Encoding for color: RGB

* RGB - Red, Green, Blue
* Like mixing colored lights. 100% of Red + Blue + Green gives white

<slide_break></slide_break>

### Encoding for color: HSV
* HSV - Hue, Saturation, Value
* A little more subtle. You have to practice your eye
* Hue is the color where all colors are arranged in a rainbow from 0 to 255
* Saturation is how "pure" or "saturated" that color is
* Value is how "bright" the color is
* Sometimes its easier to think in terms of HSV

<pre><code class="language-python">
# Simple manipulations

tl_pixel = image[0][0] 				# top left pixel
r_channel = image[0][0][0] 			# 'R' channel of top left pixel
cropped_img = image[120:240, 0:320]	# crops the image to be only the bottom half
</code></pre>

<slide_break></slide_break>

### Image processing with OpenCV

* OpenCV is the key library used for image procesing in ROS
* It is totally distinct from ROS but great bridging functions exist
* The current version is OpenCV2

<slide_break></slide_break>

### Basics to understand

* Understand representations of images as 2+ dimensional arrays
* Understand that a given image "array" will have image information encoded
* There are different encodings which change what is stored at the innermost array
* Monochrome, Depth, RGB, HSV, and more
* Different OpenCV functions expect and generate images of specific types

<slide_break></slide_break>

### Connecting OpenCV to ROS ...

* CvBridge() provides a series of methods that convert from ROS messages to OpenCV2() formats
* And vice versa
* e.g. `CvBridge().imgmsg_to_cv2(msg, desired_encoding='bgr8')`

<slide_break></slide_break>

### How to think about algorithms

* They are a series of distinct steps (e.g. a `pipeline`)
* Often, given an image, transform it, to a different image
* The order really matters
* Sometimes, given an image, analyze it, generate data (a number, an array, a list, etc)

<slide_break></slide_break>

<h2 class="shadow p-3 bg-white rounded">Thank you. Questions?<img class="img-fluid w-100" src="https://picsum.photos/800/100.jpg" /> <small> (random Image from picsum.photos)</small></h2>