Skip to content
This repository has been archived by the owner on Jan 24, 2023. It is now read-only.

Latest commit

 

History

History
141 lines (72 loc) · 6.17 KB

README.md

File metadata and controls

141 lines (72 loc) · 6.17 KB

Gaming using Computer Vision

A python bot used to play a game using computer vision and image processing techniques. The bot is able to play the game without any human intervention. It managed to make me a millionaire in the game overnight!

play-game-with-computer-vision

Here's a screen recording of the game play:

activity.mp4

Access a comprehensive guide in this article: https://paulonteri.com/thoughts/play-game-with-computer-vision.

Disclaimer - this content is for educational purposes only!


Why?

I've been playing strategy + city building + simulation? games like TownsMen 6, Clash of the Clans, SimCity for the last 10 years.

On trying out City Island 5 I found it mildly irritating that my collectables could not accumulate while I was outside the game. I might have had the best businesses, strategy, etc but I had to be in the game to ensure I collect the cash/keys/gold overtime. For example, if my bakery makes €100 per minute I would only earn €100 after leaving the game and coming back 24 hours later.

This became especially tiresome while trying to accumulate €5,000,000 required to buy the island shown below. This would take me roughly two weeks of gameplay if I don't spend any money - it's not worth it.

game_screenshot_island


How it works

1. Capture the live game feed

I needed a way to capture the live game feed.

The easiest way to to capture an in-game screenshot and pass it to the next steps in the script.

2. Identify the valuables in the screenshot

We need a way to detect a valuable in the game's feed and then return its coordinates.

OpenCv's Template Matching algorithms are perfect for this.

They are used for searching and finding the location of a template image (like a valuable) in a larger image (like the game's feed). It simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of the input image under the template image. Several comparison methods are implemented in OpenCV. (You can check docs for more details). We use it in the method: cv2.matchTemplate(... )

To achieve this, I needed the template images. I took screenshots by hand and then cropped off the cash, star and key:

cash

key

star

3. Collect the valuables by clicking on them

Once we have the coordinates of an item we can try to click on it.

pyautogui's .click(x,y) function works like magic for this. It clicks the screen on the coordinates x and y where our valuable is lying. Learn more about it here.

click_cash.mp4
click_key.mp4
click_star.mp4

4. Close any popups that may appear

Our clicks above may result in popups when we are being given a reward, levelling up, etc.

We need to close it before attempting to collect valuables again. We use the same logic used in finding and clicking on valuables.

To achieve this, I needed the template images for the popups' close buttons so that they can be clicked. I took screenshots by hand and then cropped off the various close buttons:

close

continue_level

close_button.mp4

5. Repeat

We do the steps above repeatedly to collect the valuables while the script is running.

activity.mp4

Results after running overnight

I started the game with €316,415.

game_screenshot_start

The following morning I had €6,463,870.

game_screenshot_end

I made €6,147,455 overnight!

I then proceeded to buy the Island I wanted:

image


Regrets

  • This is cheating.
  • Why get a game if you're not the one playing it?

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.


Maintainers

Current maintainers:


License

Apache License 2.0


Conclusion

Access the full guide in this article here: https://paulonteri.com/thoughts/play-game-with-computer-vision

This was fun!