Creating a bot for EXAPUNKS's solitaire

31 Dec, 2023 | 979 words | 5 minutes

#python #gui

It’s December, so it’s again time for AOC! Same as last year, I figured I wouldn’t stress too much about completing AOC on time or at all. Based on previous years I noticed that it’s in general harder for me to wrap my head around puzzles involving grid/graph evaluations, so after completing 9 different days, I slowly stopped working on AOC.

Then again, my puzzle-solving itch still remained, so I bought EXAPUNKS from Steam’s Winter Sale. The puzzles in EXAPUNKS are great. They are relatively small, as the solutions are less than 100 lines, but they are still challenging enough. In addition to programming puzzles, there’s also minigames that you play manually. One of them is solitaire (pictured below). Now, the whole game is about finding automated solutions to problems, so I figured… why not extend this to the solitaire as well?

I’ve never worked with GUI automation and I wouldn’t consider myself to be proficient with Python either, so I figured I could tackle both issues with this mini-project. The solution consists of 3 main parts:

1. Get screen and find cards

Python is often nice choice for prototyping, because a lot of functionality can be hidden behind a single pip install. For taking screenshots and finding parts of them, there’s pyscreeze with very clear API. They recommend installing cv2 library to speed up image location. This helped a little bit, but locateAll still felt somewhat slow, so I moved simple loop for the different cards into ThreadPoolExecutor. This helped to speed up this part from around 5 sec to around 2 sec.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import pyscreeze
from concurrent.futures import ThreadPoolExecutor
from itertools import repeat

def locateSingle(screen, fpath):
    results = []
    for box in pyscreeze.locateAll(fpath, screen, grayscale=False):
        results.append([fpath, box])
    return results

def find_cards():
    screen = pyscreeze.screenshot()
    files = ... # get all card images

    with ThreadPoolExecutor(MAX_WORKERS) as pool:
        result = pool.map(locateSingle, repeat(screen), files)
        # ... collect results

    # ... return results

2. Find winning moves

Once we know how our cards are arranged, we can try to find a sequence of moves that will win the game. The game states can be considered as graph nodes, so we model node transitions and use something similar to Dijkstra’s algorithm to search through all the state space, until winning state is found.

Dijkstra’s algorithm finds shortest path by prioritizing exploration of paths with smallest sum of link weights. In this case, the priority queue is ordered by two conditions – by number of completed stacks and then by number of moves taken. This allows the search to proceed with the game, as movements from and to completed stacks are discouraged, as doing so makes the stacks not complete anymore.

The speed here depends heavily on the game situation. In some cases it’s enough to explore 2,000 game states, in others the algorithm checks 20,000 states. Thus, this part may take any time between 1-10 seconds. Anyway, at the end of this, we get a list of moves in form of (from_idx, to_idx, count) tuples.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import heapq

def find_winning_moves(initial):
    seen_states = set()

    states = [initial]
    while states:
        s1 = heapq.heappop(states)
        for move in s1.get_moves():
            s2 = s1.take_move(move)

            if s2 in seen_states:
                continue

            if s2.game_is_won():
                return s2.moves

            seen_states.add(s2)
            heapq.heappush(states, s2)

3. Play a single game

Similarly to finding the card positions, I found that controlling the GUI programmatically was also easy, as there’s pyautogui with clear APIs. The hardest thing in this part was figuring out how to compute relevant coordinates and adjusting them during multiple tests. But once those coordinates are computed, then it’s just a matter of calling pyautogui.moveTo and pyautogui.dragTo repeatedly.

I also found out that you can’t be too fast with GUI automation. Specifically, there’s a “new game” button and simply clicking it wouldn’t work. However, the button worked fine when I programmed the mouse to click and drag for 20 pixels before releasing it. Similarly, moveTo and dragTo functions would fail to connect with the application if the animation time was set to 0.10 seconds or less. Maybe it’s a good thing to keep in mind that GUIs are primarily made for humans so maybe they expect human-like input.

Checking the execution time, this part is the slowest, resulting in around 28 seconds per game. Taking into account issues mentioned above, there’s probably not much we can do here, so it’s nice that I didn’t spend too much time thinking about the other parts being “too slow”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

import pyautogui as gui

def play_single_game():
    state = find_cards()
    moves = find_winning_moves(state)

    # Reset mouse to top-left corner of play field
    (top_y, top_x) = ...
    gui.moveTo(top_x, top_y, ANIMATION_TIME, gui.easeInOutQuad)

    # Replay winning moves
    for move in moves:
        state = state.take_move(move)

        (from_y, from_x) = ...
        gui.moveTo(from_x, from_y, ANIMATION_TIME, gui.easeInOutQuad)

        (to_y, to_x) = ...
        gui.dragTo(to_x, to_y, ANIMATION_TIME, gui.easeInOutQuad)

    # Wait for congratulating message
    time.sleep(2)

    # Click on "NEW GAME"
    (new_y, new_x) = ...
    gui.moveTo(new_x, new_y, ANIMATION_TIME, gui.easeInOutQuad)
    gui.drag(20, 0, ANIMATION_TIME, gui.easeInOutQuad)

    # Wait for new cards
    time.sleep(6)

4. Combining all together

Finally, bringing all of this together is as simple as:

1
2
3

if __name__ == "__main__":
    while True:
        play_single_game()

This yields a bot that’s able to play the game continuously, taking 30-40 seconds per single game. The video below is speed up x2. Full source is available here: https://github.com/yanto77/auto-solitaire.

And with this said and done, the two shining achievements are mine! 🎉

… I guess I didn’t get that far from AOC-like puzzles. 😅