Pictures to Text

For a fun little weekend project, I created a program that converts a picture into text. I developed it in a Jupyter notebook, and the code is available on GitHub.

The notebook contains three parts:

1. Image — The input image is divided into character-sized blocks.
2. Font — Each character in the input font is rendered as an image.
3. Calculation — The image blocks and rendered characters are compared to find the best matches.

First, import the image. I use the finch as an example. Most of the following code converts the Pillow image into a two-dimensional NumPy array and displays it with Matplotlib.

def show_image(path):
    image = Image.open(path)
    image = image.convert('L')
    d = list(image.getdata())
    image_2D = np.reshape(d, (image.height, image.width))
    plt.imshow(image_2D, cmap='gray')

show_image('download.png')

Output:

The next step converts the imported image into character-sized blocks. The key function is view_as_blocks, which requires the image dimensions to be exactly divisible by the block dimensions. The trim_image function removes any excess rows and columns first.

def trim_image(matrix, size_x, size_y):
    height, width = matrix.shape
    trim_height = height % size_y
    trim_width = width % size_x

    if trim_height > 0:
        matrix = matrix[:-trim_height,...]

    if trim_width > 0:
        matrix = matrix[...,:-trim_width]

    return matrix

def get_image_slices(image_path, char_width, char_height):
    image = Image.open(image_path)
    image = image.convert('L') #greyscale

    data = list(image.getdata())
    image_2D = np.reshape(data, (image.height, image.width))

    image_2D = trim_image(image_2D, char_width, char_height)
    image_slices = view_as_blocks(image_2D, block_shape=(char_height,char_width))

    return image_slices

char_width = 10
char_height = 18
image_slices = get_image_slices('download.png', char_width, char_height)


images = image_slices.reshape(image_slices.shape[0]*image_slices.shape[1],
            image_slices.shape[2], image_slices.shape[3])
nb_across = image_slices.shape[0]
nb_down = image_slices.shape[1]
plt.figure()
for i, img in enumerate(images):
    plt.subplot(nb_across, nb_down, i+1)
    plt.imshow(img, cmap='gray', vmin=0, vmax=255)

Output:

Fonts

Now that we have the image blocks, we can match characters to them. First, we render the font's characters as images. This example uses the printable ASCII characters exposed by string.printable.

def font_images():
    fnt = ImageFont.truetype('fonts/SFMono-Regular.otf', 15)
    import string
    letters = string.printable
    results = []

    for letter in letters:
        img = Image.new('RGB', (10, 18), color=(255,255,255))
        d = ImageDraw.Draw(img)
        d.text((0,0),letter, font=fnt, fill=(0,0,0))
        img = img.convert('L') #greyscale

        d = list(img.getdata())
        image_2D = np.reshape(d, (img.height, img.width))

        results.append(image_2D)

    return (letters, np.stack(results, axis=0))

letters_idx, letters = font_images()

plt.figure(figsize=(60,60))
for i, img in enumerate(letters):
    plt.subplot(20, 20, i+1)
    plt.imshow(img, cmap='gray', vmin=0, vmax=255)

Output:

Calc

Now that we have the input blocks and character images, we need to pick the closest match. In this case, I used:

Here, A and B are matrices of the same size. The lowest score represents the smallest difference, so this value can be treated as a loss.

def solve_section(section, letters):
    min_score = float('inf')
    min_letter = None
    id = 0

    for letter in letters:
        difference = letter.astype(np.int16) - section.astype(np.int16)
        score = np.sum(np.abs(difference))
        if score < min_score:
            min_score = score
            min_letter = letter
            min_id = id
        id += 1

    return (min_id, min_letter)

def solve(sections, letters):
    results = []
    ids = []
    for s in sections:
        id, min_letter = solve_section(s, letters)
        results.append(min_letter)
        ids.append(id)

    return (ids, np.stack(results, axis=0))

ids, result = solve(images, letters)

fig = plt.figure(figsize=(20, 20))

for i in range(len(result)):
    sub = fig.add_subplot(image_slices.shape[0], image_slices.shape[1], i+1)
    sub.imshow(result[i], cmap='gray', vmin=0, vmax=255)

Output:

Finally, we collect the selected characters and print them:

result_string = ""
i = 1
for id in ids:
    result_string += letters_idx[id]
    if i % image_slices.shape[1] == 0:
        result_string += '\n'
    i += 1

print(result_string)

Output:

.gMBy
  WgM_
  M@DMg_
  $BMGDMg
   `MgZMMg_
    _<V _MMK_
  _1P"`    "gg