AI Command Automation: Transforming Natural Language into Actions

ai_command_automation takes a sentence like “open Chrome, search for penguins, bookmark the page” and turns it into a list of typed actions that a local runner executes with real mouse and keyboard events. It is about 1,200 lines of Python against a local Llama through Ollama. I built it in late 2024, after Anthropic shipped computer use, as a way to find out where the LLM-OS idea actually breaks.

It breaks earlier than I expected, and not where the interesting part was supposed to be.

I assumed the hard part would be the GUI: finding the address bar, dealing with pop-ups, recovering from a mis-click. The hard part was getting a structure out of the model that I could execute at all.

The shape of the problem

The pipeline has three stages. Interpret the command; decompose it into atomic actions; execute them.

"open chrome and search for penguins"
        |
        v  NLU: intent + needs_decomposition?
        v  decompose: -> JSON array of actions
[
  {"action_type": "open_application", "parameters": {"application_name": "Chrome"}},
  {"action_type": "wait",             "parameters": {"duration": 2}},
  {"action_type": "click",            "parameters": {"target": "address bar"}},
  {"action_type": "type_text",        "parameters": {"text": "penguins"}},
  {"action_type": "press_key",        "parameters": {"key": "enter"}}
]
        |
        v  executor: pyautogui moves, clicks, types

The middle arrow is the whole problem. Everything downstream is ordinary code that works. Everything upstream is a language model being asked to stop producing language.

The prompt in the repo does what every such prompt does: it asks nicely, in bold, several times:

**Important Instructions:**
- **Respond with only a JSON array of actions.**
- **Do not include any text or explanations before or after the JSON array.**
- **Do not include code blocks, markdown, or any formatting.**
- **Ensure the JSON is properly formatted without any trailing commas.**

Every one of those bullets is scar tissue. Each was added after a specific failure. None of them work reliably, because a model that has been trained to be helpful will explain itself, and a model that has seen a million fenced code blocks will fence your JSON.

So I measured the recovery instead

When the strict parse fails, the repo does one thing: it takes everything between the first [ and the last ] and tries again. That is the entire fallback, and I had never checked what it actually buys.

try:
    actions = json.loads(response_text)
except json.JSONDecodeError:
    json_start = response_text.find("[")
    json_end = response_text.rfind("]")
    if json_start != -1 and json_end != -1 and json_start < json_end:
        actions = json.loads(response_text[json_start : json_end + 1])

So I wrote down fifteen ways model output comes back wrong (the shapes I had actually hit, plus the well-documented ones) and ran all three parsers over them.

Bracket-slicing doubles the recovery rate, from 3 to 6. That is a real return on four lines of code and I would keep it.

But look at the hollow marks.

The two failures that matter

Two inputs make the parser report success and hand back something the executor cannot run. Those are much worse than the ten it rejects, because a rejection is caught and a wrong shape is executed.

The array wrapped in a key. You ask for a list of actions, and the model returns {“actions”: [ ... ]}, helpfully labelling the thing with the word you used to describe it. That is valid JSON, so the strict parse succeeds, and the executor then iterates a dict and gets the string “actions” where it expected an action object.

NaN in a numeric field. The model does not know how long to wait, so it writes NaN. This is not valid JSON, but Python’s json module accepts it anyway, by default, and hands you a float that fails every comparison you make against it. duration > 0 is false. duration <= 0 is also false. The wait silently does not happen and the click lands on a page that has not loaded.

That second one took me an embarrassingly long time to find, because the symptom is “sometimes it clicks too early” and the cause is a default flag in the standard library.

>>> json.loads('{"duration": NaN}')
{'duration': nan}          # no error. this is the default.

>>> json.loads('{"duration": NaN}', parse_constant=reject)
ValueError: non-JSON constant: NaN

The parser I should have written first

Four layers, cheapest first, each one a named failure mode. The ordering matters: well-formed output never touches the repair code.

def parse(text):
    text = strip_code_fences(text)

    attempts = (
        lambda t: strict(t),                        # 1. it was fine
        lambda t: slice_array(t),                   # 2. prose around it
        lambda t: strict(repair(t)),                # 3. wrong dialect of JSON
        lambda t: slice_array(repair(t)),           # 4. both
        lambda t: strict(close_truncated(repair(t))),  # 5. hit the token limit
    )

    for attempt in attempts:
        try:
            result = unwrap(attempt(text))          # {"actions": [...]} -> [...]
        except Exception:
            continue
        if is_runnable(result):                     # <- the part that was missing
            return result

    return None

repair handles smart quotes, comments, trailing commas and unquoted keys. close_truncated drops the half-written last element and closes the array. unwrap takes a single-key object whose value is a list and returns the list.

But the line that actually matters is is_runnable. Parsing was never the contract. The contract is a list of objects, each with a string action_type and a dict of parameters containing no values that will poison arithmetic downstream:

def is_runnable(actions):
    if not isinstance(actions, list) or not actions:
        return False
    for a in actions:
        if not isinstance(a, dict):
            return False
        if not isinstance(a.get("action_type"), str):
            return False
        if not isinstance(a.get("parameters"), dict):
            return False
        for v in a["parameters"].values():
            if isinstance(v, float) and v != v:   # NaN
                return False
    return True

With the shape check in the loop, a layer that parses into the wrong thing falls through to the next layer instead of returning. Fourteen of fifteen come back runnable, and the fifteenth, the NaN duration, is refused rather than executed, which is the correct answer. There is no sensible number to recover there. Asking again is cheap; clicking early is not.

The honest summary is that most of what I called “prompt engineering” for two months was a parser I had not written yet.

What is still genuinely hard

Structured output is a solved problem once you stop treating it as a prompting problem. Constrained decoding and schema-enforced tool calls solve it better than any of the above, and if I were starting today I would reach for those first. The thing they do not solve is the next stage.

{“action_type”: “click”, “parameters”: {“target”: “address bar”}} is perfectly well-formed and completely unexecutable. Somebody has to turn “address bar” into a pixel coordinate on this screen, in this window, at this scroll position. That is the real problem, and no amount of JSON discipline touches it.

I pulled that piece out into its own project: mouse-point resolves a natural-language description of a UI element to coordinates. Splitting it out was the right call: targeting and planning fail for unrelated reasons and improve on unrelated schedules, and keeping them in one repo meant every targeting bug looked like a reasoning bug.

I wrote up where the broader field is on this in LLM-based agents for human-like computer control. Short version: nobody is close, and the missing piece is not model quality.

What this is not

• Not an end-to-end success rate. I have not measured how often the whole pipeline completes a task, because doing that honestly needs a task suite and a controlled desktop, and I have neither. Anyone quoting you a success rate for a GUI agent without describing the environment is quoting you a number about their environment.
• The corpus is synthetic. Fifteen hand-written shapes drawn from failures I hit and failure modes that are well documented. It measures the parser exactly; it does not predict how often any given model produces each shape. That distribution is model- and prompt-specific and it moves with every release.
• The project is an experiment, not a tool. It runs on my machine against a local model and it is bounded by that. Issues and pull requests welcome; production use is not advised.

The code for both the original parser and the layered one is in the repo. The benchmark is fifteen strings and a loop; if you have a structured-output pipeline, running your own version of it against your own logged failures will take you an afternoon and will probably tell you something unflattering.