Agent Monitor¶

Agent Monitor provides real-time visibility into autonomous background AI agents and scheduled automations (such as Alert Triage) executing across the grid.

This screen is located under the Administration menu on the left side of the Security Onion Console when agentic mode is enabled, and requires the automations/read permission, which is granted to the analyst, auditor, and superuser roles. It is also accessible directly from the Background Agent Activity pill in Onion AI.

Controls and Scheduler Status¶

The top toolbar offers quick controls for supervising active execution:

• Breadcrumbs: Navigate between the main overview and detailed execution transcripts.
• Pause / Resume: Freezes live updates so analysts can inspect in-flight work without the view changing, or resumes live polling.
• Refresh: Manually reloads current execution state.
• Options (Gear Icon): Controls whether Run History hides successful runs with no work items.
• Documentation Link: Quick access to Onion AI documentation.

If the automation scheduler is not running, an alert banner appears at the top of the page.

Agent Pool Metrics¶

Summary stat pills at the top of the screen give an immediate health check of agent pool capacity:

• Pool Running: Displays the count of currently running tasks compared against the maximum concurrency limit (e.g., 2 / 5). The value turns yellow to warn operators when the pool is full and tasks are waiting.
• Pool Queued: Displays the count of tasks currently waiting in the execution queue compared against the maximum queue depth (e.g., 1 / 20). The value turns yellow under the same condition as Pool Running.
• Peak Running: The most tasks running at once since SOC started.
• Peak Queued: The most tasks waiting at once since SOC started.
• Rejected: Count of task submissions refused by the pool because the queue was full. Displays in red if any submissions have been rejected.
• Deduplicated: Count of task submissions skipped because the same work was already queued or running.
• Agent Busy: Count of chats refused because their agent was at its Max Concurrent Sessions limit.

Understanding Rejected Items¶

The execution pool protects the grid from resource exhaustion by enforcing a maximum queue limit (maxQueuedItems). When the queue is full, new task submissions are rejected. Here is what happens to those rejected items:

• Automatic Requeueing for Automations: Work items generated by scheduled automations (such as Alert Triage) are not lost or deleted when rejected. The automation engine catches the queue-full condition and automatically returns the work item to the pending backlog. The scheduler will re-attempt the item on a subsequent cycle once queue space or running slots become available. The run that hit the full queue appears in Run History with a Failed outcome.
• Retry Limits: Work items track their attempts and failures. If an item fails in as many runs as the automation's maxFailures setting (default 3), it is given up and transitions to the failed state, preventing broken items from looping indefinitely.
• Operator Guidance: A non-zero or rising Rejected counter indicates that background workloads are outpacing worker throughput. Administrators should:
  • Review the schedule interval of active automations in Agent Studio (e.g., lengthening intervals or tuning alert query filters to reduce batch sizes).
  • Review Max Concurrent Sessions on active agents to allow sufficient concurrency.
  • Check for slow upstream LLM provider responses that may be causing running tasks to hold slots longer than usual.

Work Item Lifecycle¶

Automated tasks move through defined execution states:

• Pending: The work item is in the backlog, awaiting a run to claim it (including items being retried after a failed run).
• Queued: The work item has been claimed and is waiting in the execution pool for an available worker slot.
• Running: An agent has claimed the item and is actively analyzing data, executing model turns, or invoking tools.
• Applying: The agent has concluded its investigation and its results are being written to the alerts the item covers.
• Done: The work item finished its run successfully.
• Failed: The task encountered an unrecoverable error or exceeded its retry attempts.

In-Flight Work Items¶

The In Flight panel lists tasks that are currently queued, executing, or applying updates across the grid.

Table Columns¶

• Automation: The name of the automation driving the task.
• Group: The identifier of the workload group, made of the values the automation groups alerts by (e.g., rule.name:Suspicious PowerShell).
• Status: Visual chip indicating whether the item is pending, queued, running, or applying.
• Doing: Live phase indicator revealing what the agent is currently doing in real time, such as waiting for a model completion or invoking a specific tool (e.g., Invoking query_events).
• Agent: The active agent and delegation chain responsible for the item (e.g., Investigator or Investigator → DetectionEngineer).
• Attempt: How many times the item has been claimed for processing.
• In State: How long the item has remained in its current state.
• Elapsed: Total time elapsed since the work item was initially created.
• Actions: Click the binoculars icon to open the live execution transcript.

Expanded Work Item Inspector¶

Expanding an in-flight row opens an inspector card with four subtabs:

• Details Subtab: A key-value summary of the Work Item ID, Run ID, Automation, Group, Status, Attempt, Created and Last Changed timestamps, and any Error. Superusers also get a gear icon to open the automation in Agent Studio.
• Payload Subtab: Displays the exact input parameters and raw data supplied to the automation for this work item.
• Attempts Subtab: Lists every attempt made on this work item, showing the attempt number, session UUID, live status chip, start time, and a binoculars icon to inspect that specific attempt's transcript. Alerts if any attempt was orphaned due to unexpected worker restarts.
• Result Subtab: Displays the result the agent produced (for Alert Triage, the investigation session ID). Only shown once the item has a result, such as while it is applying.

Run History¶

The Run History panel lists finished automation runs across the grid.

Filtering and Searching¶

• Automation: Limits the list to one automation, or shows All.
• Search: Matches automation names, errors, and work item groups, or a pasted run ID.

Table Columns¶

• Automation: Name of the automation that ran.
• Started: When the run began.
• Duration: How long the run took.
• Done and Failed: Number of work items the run finished or failed.
• Outcome: Completed or Failed.

Inspecting a Run¶

Expanding a run shows its Run ID, any run error, and the work items it handled, with each item's Group, Outcome, Error, Attempt, Finished, Duration, and a binoculars icon to review its transcript.

Expanding a work item opens the same inspector card as In Flight.

Live Transcript View¶

Selecting the binoculars icon on any in-flight or Run History item opens the live, step-by-step transcript view.

Transcript Controls and Status¶

• Agent Title & Model: Displays the active agent name and the underlying LLM executing the run.
• Live Badge: A Live chip appears while the task is still executing and you are viewing its latest attempt.
• Live Updates: Model turns and tool results appear as they happen without requiring manual page reloads.

Inspecting Agent Reasoning and Tool Execution¶

• Tool Use Cards: Every tool execution is displayed in an interactive card:
  • Tool Name & Parameters: Formatted code block showing exact parameters passed to the tool (such as OQL queries, alert IDs, or detection filters).
  • Status Indicator: Visual indicator showing whether the tool is preparing, executing, completed, or returned an error.
  • Raw Results: Expandable accordion displaying the structured data returned by the tool.
• Nested Delegations: When an agent delegates sub-tasks to subordinate agents, the delegation appears as a nested panel:
  • Displays the delegated sub-agent identity.
  • Tracks credits and tokens consumed by the sub-agent.
  • Contains the sub-agent's own thoughts, in an expandable accordion labeled with a lightbulb icon, and nested tool executions.