.. _reports: Reports ======= Starting in version 2.4.180, Security Onion Pro added support for producing reports and data exports for analytical purposes. This is not a data migration or full data export feature but rather is intended for analysts to present relevant data to the broader team. Consequently, exports are limited in the amount of data that will be produced. .. note:: This is an enterprise-level feature of Security Onion. Contact Security Onion Solutions, LLC via our website at https://securityonion.com/pro for more information about purchasing a Security Onion Pro license to enable this feature. Reports ------- Reports are generated in Portable Document Format (PDF) in the background. Once a report is requested, a confirmation message will appear to notify the user that the report is being generated and will be available for download on the Reports screen. Two types of reports are available: standard and custom. Standard Reports ~~~~~~~~~~~~~~~~ Standard reports are pre-built reports provided by the Security Onion team. A *Productivity Report* provides general team productivity metrics. This includes: - Ingested event metrics - Alert metrics (total, acknowledged, escalated, by severity, etc) - Case metrics (total, by status, by assignee, comments by user, etc) - Time tracking (total case hours, case hours by user) Productivity reports can be generated by navigating to the Reports interface and clicking the ``Add Job`` button near the top of the screen. A popup will appear allowing the user to select the Productivity report. A *Case Report* provides detailed information about a specific case. This includes: - ID, Title, description - Current status, assignee, etc - Hours logged against the case - All case comments and the date and author - All detections involved in the case - All attachments (excluding actual file content) - Observable data - Related events - Case audit history Case reports can be generated by opening a case and clicking the Export icon above the Case title. .. note:: Customer feedback will help the Security Onion team determine which additional reports will be provided in future releases. Provide feedback by reaching out to our Support team. Standard reports can be lightly customized via the Configuration screen. Search for the report name, such as "Case report". Customizations include changing layout formatting, text content, and hiding or showing additional data. The ability to show additional data is limited to what the built-in report has already queried. For example, the built-in Case Report shows the TLP of each observable, but not the PAP. So if you want to see the PAP and not the TLP, then you could customize to do so. Remember to synchronize the grid after saving customizations. Custom Reports ~~~~~~~~~~~~~~ While the standard reports can be lightly customized, some customers may need more flexibility in customizing reports. For this, Security Onion Pro offers custom report types. Custom reports are generated by navigating to the Reports interface and clicking the Add Job button. The popup will show a list of available custom reports. Customers can create a limited number of custom reports by opening the Configuration screen and searching for ``custom report``. Custom reports allow users to design their own report layout and choose which data is queried and then rendered on the report. Customizing these reports requires an understanding of markdown format (https://www.markdownguide.org/) and a general understanding of OQL syntax. For best results, report designers should build queries in the :ref:`hunt` interface to ensure the data returned is what they want for their report. Then the report designer will copy that same OQL query into the custom report. A sample report is provided in the standard Security Onion Pro installation. Here is the content of this report file followed by a brief review of each section. :: {{- /* query.myDocEvents.Oql = metadata.type: _doc | groupby event.module, event.dataset | sortby @timestamp desc */ -}} {{- /* query.myDocEvents.MetricLimit = 10 */ -}} {{- /* query.myDocEvents.EventLimit = 100 */ -}} Security Onion Custom Report ============================ {{ if .Error }} **NOTE: This report encountered a problem extracting the relevant data and may not be complete.** **Error:** {{.Error}} {{ end }} Records must have been created or updated during the following time frame in order to be reflected in this report. **Report Start Date:** {{formatDateTime "Mon Jan 02 15:04:05 -0700 2006" .BeginDate}} **Report End Date:** {{formatDateTime "Mon Jan 02 15:04:05 -0700 2006" .EndDate}} ## Sample Doc Events **Total Events:** {{ formatNumber "%d" "en" .Results.myDocEvents.TotalEvents}} ### Event Counts By Module and Dataset | Count | Proportion | Module | Dataset | | ----- | ---------- | ------ | ------- | {{ range sortMetrics "Value" "desc" .Results.myDocEvents.Metrics.groupby_0_event_module_event_dataset -}} | {{ formatNumber "%.0f" "en" .Value}} | {{ formatNumber "%.1f" "en" .Percentage}}% | {{index .Keys 0}} | {{index .Keys 1}} | {{end}} ### Individual Events (Limited to first {{.Results.myDocEvents.Criteria.EventLimit}}) | Event Time | Module | Dataset | Category | | ---------- | ------ | ------- | -------- | {{ range .Results.myDocEvents.Events -}} | {{.Timestamp}} | {{.Payload.event_module}} | {{.Payload.event_dataset}} | {{.Payload.event_category}} | {{end}} Starting at the top: The first three lines are internal instructions to the report generator. They tell the report generator to run the OQL query ``metadata.type: _doc | groupby event.module, event.dataset | sortby @timestamp desc`` and store the results into the ``.Results.myDocEvents`` variable, for later reference in the report. The ``MetricLimit = 10`` parameter limits the groupby metrics to the top 10 counts, and the ``EventLimit = 100`` limits the raw event results to 100 at most. These are important parameters because without them you could end up with a PDF document hundreds of pages in length, and that much data is unlikely to be helpful when presenting security-related findings to others. Pay close attention to the traditional template ``{{ ... }}`` syntax as well as the ``{{- /* ... */ -}}`` internal report instruction syntax. A missed character will cause the report generation to fail. Next is the title of the report. Notice how the full length of the title is followed by an equally long series of equal ``=`` characters. This denotes that the previous line is the report title, and will be the name that appears in the Add Job popup list on the Reports interface. Refer to the Markdown guide reference earlier in this section for more help with defining titles, headers, sections, and other styling within your report document. Following the title is a ``{{ if .Error }} ... {{ end }}`` block. This is useful for presenting errors encountered during the rendering of the report. If there is no error then this section will not be visible in the final generated report. Next is the content of the report itself. You'll see report text content mixed with data outputs. For example, if 12 events matched the OQL query provided at the top of this report, then the line containing ``**Total Events:**`` will be rendered as: **Total Events**: 12 Finally, there are two tabular outputs included. The first is showing "Event counts by Module and Dataset" and, according to markdown format, will be rendered in a table structure including the column headers "Count, Proportion, Module, and Dataset". Note that those ``|`` characters are used to separate the column data. The first column will output the Value of the first groupby Metric (referenced as ``groupby_0_event_module_event_dataset``), which will be a "count" value. The ``%.0f`` is instructing the output to hide the decimal place values, and the ``en`` is instructing that the number be formatted in the English locale (1,220 vs 1.220). The second column will show the proportion of events that matched this row out of the full set of events, the third column will show the event module for that row, and the last column will show the event dataset for that row. The second tabular output is showing the list of events (up to 100 as previously instructed at the top of the file). This is not a metric so there is no ``.Value`` count column on this table. Instead, the column will show the actual event field values. Notice how the first column is the event timestamp, which is useful for showing chronologically-ordered lists of events. The remaining columns can be any field that is associated with that event. Use :ref:`hunt` to determine which fields are available for that query. .. note:: Customizing reports can be intimidating to those not familiar with this level of detail. Pro customers with professional service hours can utilize the experience of the Security Onion support team to help get you started. Remember to synchronize the grid after saving custom reports. Tabular Exports --------------- Also included with this feature is the ability to export data from several of the SOC screens, including: - :ref:`alerts` - :ref:`dashboards` - :ref:`hunt` - :ref:`cases` (listing) - :ref:`detections` (listing) The exported data will be saved to a CSV file and include the headers as the first line of the CSV. The CSV will be generated in the background and will then be available for download in the Reports interface. Group and Event data can be exported by clicking on the data export icon, typically found in the top-left corner of the Group metric table or graph, or the event table. While the export icon is visible on the graph mode of a group metric, it will still export the underlying data in CSV format. It does not attempt to export the graph visualization. However, starting with this same version 2.4.180, users can now use the browser's Print feature to send those graph visualizations and dashboards to a printer or PDF file. CSV exports will export all underlying data, up to a configured max. Therefore, while the :ref:`dashboards` interface could show 10 entries in a Group metric because of how the group limits are set, the CSV could contain more than the 10, if the backing data has more aggregated metrics available. .. warning:: Exporting CSV data while in the relative time mode, such as 24 hours, will often result in exported CSV data that is different from what is on the screen at the time when the export was initiated. This is because the export is re-running the same query, using the same relative time but starting at the time of the export, versus when the dashboard was queried earlier. To export the identical data shown on the SOC interface first change the time range to absolute mode and define the start and end times prior to refreshing the results and clicking the export icon. CSV exports are limited to 10,000 records by default. .. note:: This CSV export feature is not intended for Elasticsearch data migration purposes. Report Customization Reference ------------------------------ PDF Render Instructions ~~~~~~~~~~~~~~~~~~~~~~~ - Generate a table of contents at the beginning of the exported report: :: {{- /* pdf_param: -generate-toc */ -}} - Disable TLS verification when reaching out to pull external images for inclusion in the report: :: {{- /* pdf_param: -insecure */ -}} - Add a page break whenever it encounters a horizontal rule (HR) tag in the report, such as ``---``: :: {{- /* pdf_param: -new-page-on-hr */ -}} - Change the report output layaout from *portrait* (default) to *landscape*: :: {{- /* pdf_param: -orientation */ -}} {{- /* pdf_param: landscape */ -}} OQL Query Instructions ~~~~~~~~~~~~~~~~~~~~~~ - Define a new OQL query that filters by ``some.field`` and ``another.field``, and includes a single aggregated metric of counts by both ``event.modules`` and ``event.dataset`` with a max limit of no more than 10 aggregations per groupby and no more than 25 total event records returned: :: {{- /* query.mySpecialQuery.Oql = some.field: 123 AND another.field: "Jason" | groupby event.module, event.dataset */ -}} {{- /* query.mySpecialQuery.MetricLimit = 10 */ -}} {{- /* query.mySpecialQuery.EventLimit = 25 */ -}} Accessing Query Results ~~~~~~~~~~~~~~~~~~~~~~~ - Loop through a query's event results. Note that all fields containing a period will have their periods replaced with an underscore to avoid colliding with the template syntax: :: {{ range .Results.mySpecialQuery.Events -}} {{.Timestamp}} | {{.some_field}} | {{ .another_field }} {{ end }} - Loop through a query's aggregation results. Note that aggregations have special fields available: - ``.Value`` The calculated aggregation, typically a summed count - ``.Percentage`` The calculated ratio * 100 of this group's occurrences compared to the total number of groups for this aggregation - ``.Keys `` Where is the 0-based index of the groupby fields. In this example, ``.Keys 0`` will be the ``event.module`` value, and ``.Keys 1`` will be the ``event.dataset`` value for this group. :: {{ range sortMetrics "Value" "desc" .Results.mySpecialQuery.Metrics.groupby_0_event_module_event_dataset -}} {{ formatNumber "%.0f" "en" .Value }} | {{ formatNumber "%.1f" "en" .Percentage }}% | {{ index .Keys 0 }} | {{ index .Keys 1 }} | {{ end }} If there had been a second ``groupby`` clause then it would have been referenced as ``groupby_1_fieldx``. Formatters ~~~~~~~~~~ - Format a date and time into a specific format of *Mon Jan 02 15:04:05 -0700 2006*. Always reuse the same date and time as shown below for the format template: :: {{ formatDateTime "Mon Jan 02 15:04:05 -0700 2006" .SomeDataField }} - Format a numeric value to an integer, decimal, etc output, following the given locale, in this case *English*: :: {{ formatNumber "%.2f" "en" .SomeNumericField }} Sorters ~~~~~~~ - Sort the given metrics by the .Value field in descending order: :: {{ range sortMetrics .Value "desc" .Results.mySpecialQuery.Metrics.groupby_1_fieldx }} - Sort the given case data by the .CreateTime field (or event's timestamp) in ascending order (applicable to Case Report): :: {{ range sortComments .CreateTime "asc" .Comments }} {{ range sortDetections .CreateTime "asc" .Detections }} {{ range sortArtifacts .CreateTime "asc" .Attachments }} {{ range sortArtifacts .CreateTime "asc" .Observables }} {{ range sortRelatedEvents "fields:soc_timestamp" "asc" .RelatedEvents }} {{ range sortHistory .CreateTime "asc" .History }} Filters ~~~~~~~ - Convert the given user ID into the user's email address. Applies to any valid user ID event field or indexed metric field: :: {{ .SomeField | getUserDetail "email" }} - Convert the value to uppercase: :: {{ .SomeField | upper }} - Convert the value to lowercase: :: {{ .SomeField | lower }} - Join an array value into a string, separated by commas: :: {{ .SomeField | join "," }}