Summary of notebooks Data Processing advanced

Summary Data processing advanced

Week 1
Array = ordered and structured collection of elements.
- Arrays are structured around the number of dimensions they contain.

1-dimensional array  vector (e.g., stock price)
2-dimensional array  matrix (e.g., gray scale image)
N-dimensional array  tensor (e.g., RGB image)

How to create an array
My_list = [1, 3, 5, 7, 9]
My_array = numpy.array(my_list)
- My_array outputs  [1 3 5 7 9]

You could also create an array without specifying a python list in advance:
My_array = numpy.array([2, 4, 6, 8])

Dtype()  specifies the datatype for the elements inside the array.

Creating an array from a file
 Numpy.loadtxt()  loads a text file.
 Numpy.load()  loads a binary file.
 Numpy.savetxt()  saves an array in a file
 numpy.savetxt("sample_file.txt", a)
 it saves the array (a) to the text file specified

creating arrays based on range
 numpy.arange(a, b, s)  a is the start point, b the endpoint and s is the step size.
 Can have integers or floats as inputs.
 So, useful when you want to specify the stepsize.
 Numpy.linspace(a, b, i)  a is the start point, b the endpoint and I is the number of
items.
 Useful when you want the specify the amount of items in the range.

Creating arrays of random values
 Numpy.random.random(n)  returns n numbers between 0 and 1
 Numpy.random.uniform(x, y, z)  x is the starting point, y the endpoint and z the
number of items drawn between x and y.
 Numpy.random.randint(x, y, z)  returns z random integers between x and y.
For a normal distribution you can use:
 Numpy.random.normal(x, y, z)

,Shape and size
Examples for array a:
 A.size  shows the total number of element in the array
 A.ndim  shows the number of dimensions.

To check if two arrays are the same you could check:
 Type()  looks if the object types match.
 Numpy.allclose()  looks if the elements all match.
 Id()  looks if the location in computer memory is the same for both objects.

Immutable datatypes python:
 Boolean
 Integer
 Float
 Complex
 String
 Byte
 Tuple

Mutable datatypes:
 List
 Set
 Dictionary

You can specify the datatype of the array when you create the array with the array() function
using the dtype() argument.
- Note: not every combination is possible.
- Example:
strings = ["12", "3", "24"]
z = numpy.array(strings, dtype='int')
you can use the astype() function to convert an existing numpy array to a different type.
- Sometimes, the data types are converted but the content is slightly changed. For
example, when converting from float to integer, then the numbers are rounded down
(floor). For example, when converting an array with only zeros and ones to the data
type Boolean than the 0 is converted to False and the 1 is converted to True.
Example from something that gives an error:
- x = numpy.array(["true", "false"])
- a = x.astype('bool')

indexing a 1-dimensional array is the same as indexing a python list.
A = numpy.arange(10)  [0 1 2 3 4 5 6 7 8 9]
Print(a[0])  0
Print(a[3])  3

A function can also return a linear index. Two examples are:
- argmin
- argmax
these return the index of the minimal value and index of the maximal value.

,Boolean indexing  return all values in the array for which the index is True.
- Example:
index = a > 0.0
b = a[index]
print( b )

as mentioned before, when you want the number of elements in an array, you use the size()
function. You don’t use len() because this can sometimes be misleading
- Example:
A = [[1, 3], [2, 4]]
Len(a)  returns 2
a.size  returns 4

Week 2
Bookshelf analogy for multi-dimensional arrays:
 1d array is a single row of a bookshelf, where a book can be identified by its position
in the row.
 2d array is the whole bookshelf, where a book can be identified by its row number
and its position in the row.
 There has to be the same number of books on every shelf.
 3d array is a room full of bookshelves, where a book can be identified by the number
of the bookshelf, row, and position in the row.
 4d array is a library with rooms with bookshelves, where a book can be identified by
the room, bookshelf, row, and position in the row.
 ...and so on...

Attributes to gain insight about the dimensionality:
- X.ndim  number of dimensions
- X.shape  length of each dimension
 Can for example be (2, 2, 3)  fist 2 tells you how many lists there are in the
array; second 2 tells you how many small lists are in the bigger lists; and 3 tells
you the size of each separate list.
 Looks like:
[[[ 1 3 1]
[ 2 4 2]]

[[11 13 3]
[12 14 4]]]
- X.size  total number of elements

An empty array or an array with only zeros and ones can be created with the empty, zeros
and ones functions.
 One-dimensional empty array  numpy.empty( (3) )
 Two-dimensional array with zero’s  numpy.zeros( (2, 3) )
 Four-dimensional array with ones  numpy.ones( (2, 2, 2, 2) )

, Identity matrix
= two-dimensional square matrix in which all values are zeros, except for the ones along the
diagonal.
This can be created with the eye(n) function.
- Example: numpy.eye(4)
Output:
[[1. 0. 0. 0.]
[0. 1. 0. 0.]
[0. 0. 1. 0.]
[0. 0. 0. 1.]]

More complex arrays can be created by combining two or more one-dimensional arrays.
- This is called vector stacking.
Horizontal stack:
- Numpy.hstack([x, y, z])
Example:
x = numpy.arange(0, 5)
y = numpy.arange(5, 10)
z = numpy.arange(10, 15)

numpy.hstack([x, y, z])
output = [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14]
shape = (15, )

Vertical stack:
- Numpy.vstack([x, y, z])
Example:
numpy.vstack([x, y, z])
output =
[[ 0 1 2 3 4]
[ 5 6 7 8 9]
[10 11 12 13 14]]
Shape = (3, 5)

You could also split arrays into a list of one-dimensional arrays (vectors).
Horizontal split:
- Numpy.hsplit(x, N)
Example:
a = numpy.arange(0, 5)
b = numpy.arange(5, 10)
c = numpy.arange(10, 15)
d = numpy.vstack([a, b, c])

numpy.hsplit(d, 5)  d is the array you want to split