Computer Vision
Image Detection looks for one exact picture. Computer Vision looks for a kind of thing: with a model that knows cars, it finds cars of any colour, angle or size. Use it when the target looks different between runs.
It is slower than template matching, needs a model download, and reports a confidence rather than a yes or no. Try first.
Experimental, and . Results vary between devices and models, and the action may change between updates. Use Share feedback in the action to report problems.
The vision engine
The first time you open a Computer Vision action you are asked to download the vision engine. It is a few megabytes, downloads once, and is shared by all macros.
It downloads through Google Play, so it is only available in the Play Store version of MacroForge. If a macro runs without it, the run stops and tells you to download it.
Choosing a model
Model is the first thing to set, and Select model opens the catalogue. It has two tabs:
- Installed, what is already on this device.
- Browse, our preset models. Each shows how many classes it knows and its input size. Tap one for details before downloading.
Import model uses your own file: ONNX (.onnx) or mobile-optimised (.ort), up to 512 MB. Export from YOLO, PyTorch or TensorFlow to ONNX first.
Advanced swaps the picker for a text field, which means the model can be chosen while the macro runs. Put a %variable% in it and one action can run a different model depending on what the macro worked out earlier.
If the model you need is missing, open a support ticket on the Discord. If we add it, you get a free month of Premium.
Classes
A model reports a class name for every box it finds, and Class filter is how you narrow that down. List the ones you care about, comma-separated, and leave it empty to accept all of them. Matching ignores case. Each model's detail page lists the classes it knows.
Search region
Off by default, which searches the whole screen. Turn it on to scan a box instead; it is faster and avoids matches elsewhere on screen. applies as usual.
Behaviors
- Wait till appear, pause until something matching is on screen.
- Wait till vanish, pause until nothing matching is left.
- Click best match, tap the highest-confidence detection.
- Click multiple matches, tap every detection that passed. Max clicks caps how many, where
0means no limit, and the order is highest confidence first.
Matching
- Minimum confidence: the score a detection has to beat. Raise it when the action fires on the wrong thing; lower it when a real target is being missed.
- Overlap merge: removes duplicate boxes drawn over one object. Raise it when the same thing is reported twice.
- Max detections: how many boxes to keep at most.
- Timeout (ms): how long to keep scanning before giving up.
- Scan rate (per second): how often to re-run the model while waiting.
Scan rate and the model's input size are what cost battery. A smaller input size is much faster and slightly less accurate, which is usually the right trade on a phone.
Test on screen
Test on screen runs the model once on the live screen and boxes everything it finds. The summary shows how many detections match your filter and confidence (3 of 7 match), how long it took, and how many fell below your confidence bar.
If your target shows up below the bar, for example at 0.38 with the bar at 0.5, lower the bar.
Save Result To Variable
Name a variable and the action stores its best detection, with the same parts every detection writes plus three of its own:
%name.label%, the class it decided on.%name.conf%, how confident it was,0to1.%name.count%, how many detections passed in total.
Position and bounds come through as usual, so %name.x% and %name.y% are the centre of the box. Guard on %name.found% before reading any of it, exactly as described under .
Sharing a macro that uses it
When someone installs your macro, MacroForge offers to download the engine and any preset models it uses.
Imported models are not included, so installers have to pick their own model. Use a preset if you plan to share the macro.