Skip to main content

Report Templates

Follow these steps to create a report:

  1. Create a report template.

  2. Apply the report template to your test results.

Create a New Report Template

Assign a name to the report template that allows you to easily identify and search for the report template. This name appears in the list of report templates to which you have access.

Optionally, add a description to the report template.

Optionally, add one or more tags to describe this report template. Tags are searchable in WorkloadWisdom.

2019-11-15_14-15-56.png

Select Report Type

Define the type of charts to be built by the report template. Baseline and multi-statistic charts are supported.

Baseline Chart

Use baseline charts if you want to create a report template that compares the same statistic from different test runs, where one of the test runs is the baseline.

For example, if you are creating a report that compares how 5 different firmware versions perform when handling the same workload, and you want the performance obtained with the current firmware version to be the baseline, you might use a baseline chart. Create a baseline chart and identify the test run performed on the current firmware version to be the baseline result.

Select Baseline

  1. Select a Workload Test or Suite from the drop-down

    1. If selecting a Workload Test

      1. Once you select a Workload Test, all available test runs associated with the selected workload that meet the filter criteria and that were not started by a Suite are made available.

        2019-11-15_14-19-15.png

        Note

        Prior to WorkloadWisdom 6.2, if you ran Workload Test X ad-hoc 3 times, and that Workload Test X was also used in an Iteration Suite Y, and the Iteration Suite Y ran 1,000 iterations of Workload Test X, then you would see all 1,003 Workload Test Runs when you select Workload Test X here. This made it difficult to differentiate and find the 3 runs of Workload Test X from the other 1,000 runs. Starting with 6.2, the 3 ad-hoc runs of Workload Test X will appear when you select a Workload Test, and the 1,000 runs of Workload Test X that were started from Iteration Suite Y can be found by selecting a run from Iteration Suite Y, making it easier to find both the 3 runs and the 1,000 runs.

    2. If selecting a Workload Suite or Iteration Suite

      1. Once you select a Workload Suite or an Iteration Suite, an additional drop-down is available for you to select the Suite Run, in which each Suite Run contains one or more Workload Test Runs that you can select in the next step.

        2019-11-15_14-21-11.png

    Caution

    If the number of Workload Tests and Suites is very large (over 1,000), it may take 1 to 2 minutes to load the list in the drop-down.

  2. Select the Run from the selected Workload Test / Suite Run that you want to be the baseline result.

  3. Select the Run from the selected Workload Test / Suite Run that you want to be the baseline result. The Runs are identified by their system generated timestamps in the format YYYY-MM-DD HH:MM:SS AM/PM, followed by their user entered Run Description if you have entered any. In the example below, “2018-10-05 11:34:52 AM” is generated by the system, and “Storage firmware A” is manually entered by the user when viewing the Workload Test Run’s Results Dashboard. By default, Runs that have an Aborted, Failed, or Skipped status are filtered out as these are usually not used for reporting. However, if you have a need to include one of these Runs in the report, you can deselect one or more of these by using the Do not show checkboxes.

    2019-11-15_14-24-53.png

  4. Define the test runs from which you want the report template to collect statistics. Three options are available:

    1. Most recent test run. Select “last test run” to collect statistics from the very last test run of the workload when applying this report template.

    2. Most recent N test runs. Select “last” to define the number most recent test runs of the workload when applying this report template.

    3. Test runs from recent days. Select “test runs during last” to define the maximum number of test runs of the workload from the past days when applying this report template.

  5. Define the specific baseline charts to include in the report template.

    2019-11-15_14-28-05.png

  6. For each baseline chart, open the statistics selection dialog by clicking on the … button in the screenshot below and select one statistic (for example, NFSv4.1 Commands Succeeded/sec) at a time. This is the statistic that is collected from the applicable test runs to populate this chart when the report template is applied. Optionally configure other settings on the chart. To preview the resulting chart when you create a Report out of this Report Template, click Reload chart preview.

    2019-11-15_14-30-14.png

Multi-Statistic Chart

Use multi-statistic charts if you want to create a report template that plots different statistics on the same chart.

For example, suppose that you want to plot throughput, IOPS, and latency on the same time-series chart, to enable you to ascertain how these statistics relate and correlate to each other over the course of a workload. Create a multi-statistic chart, and select the three statistics to be plotted on the same chart.

Specify Test Runs for the Report

  1. Optional: Select a Workload Test or Suite from the drop-down that will always be included in the Report when you create it. Follow the same instructions from the Baseline Report section to select a Workload Test or Workload Suite or Iteration Suite. The only difference is that you can have multiple Test Runs.

  2. Optionally, select one or more test runs by following the step above for Test Run 2 in the Report Template.

    2019-11-15_15-05-44.png

    Note

    You can select the same Workload Test / Suite or a different one. There is no requirement in Multi-Statistic Charts to use Runs from the same Workload Test or Suite Run as required by Baseline Charts. However, in most cases, you want to use at least the same protocol.

  3. Define the specific multi-statistic charts to include in the report template.

    2019-11-15_15-12-03.png

    1. Custom Chart: use custom chart to define exactly what and how many statistics to plot on the same chart, as well as the presentation of the statistics. For example, you may want a KPI chart where you plot throughput, IOPS, and response time on the same time graph.

      2019-11-15_15-19-21.png

      1. For each chart, select one or more statistics to plot on the same chart. The selected statistics are collected from the applicable test runs when you apply the report template.

        2019-11-15_15-20-17.png

      2. By default, for each statistic you add to the custom chart, a dedicated Y-axis is generated for that statistic. You might want to have only one Y-axis if multiple statistics share the same unit, for example, read throughput statistic and write throughput statistic.

        2019-11-15_15-22-10.png

      3. To make that change click the down arrow to open up the Y-axis dialog and enter the same Y-axis name for the statistics you want to plot over the same Y-axis.

        2019-11-15_15-24-27.png

    2. Scatter Plot: Use scatter plot if you want to define which two statistics to plot on a standard scatter plot. For example, you may want to see how sensitive the SUT's Response Time is to a wide range of IOPS.

      2019-11-15_15-33-30.png

      1. For each chart, select one statistic to be plotted on the x-axis, and one statistic to be plotted on the y-statistic, for each test run.

        2019-11-15_15-31-17.png

      2. Click Reload chart preview.

Creating a Multi-Statistic Report Template with Different Protocols

If you have different Runs from different protocols that you want to put in the same Report, it is recommended that you apply Test Runs of different protocols to different charts. For example, one Custom Chart for FC for Test Runs 1 and 2, and one Custom Chart for NFSv4.1 for Test Runs 3 and 4. Overall, the workflow is no different between one protocol and multiple protocols.

Chart Presentation Options

Statistics Selection

For each selected statistic, you can define the ports from which to collect the statistics.

The following statistics selections are supported:

  • Total. Aggregate from all Workload Generator test ports used in the test run. This is the most common statistics selection, as well as the default.

  • Client total. Aggregate from all Workload Generator client ports used in the test run.

  • Server total. Aggregate from all Workload Generator server ports used in the test run.

  • Port n. Where n is a specific Workload Generator test port used in the test run.

Provide a chart title (for example, Average SCSI Throughput) for the chart in the report template.

Chart Type

Define the chart type for the chart.

The following chart types are supported:

  • Line chart (value vs time). Plots the statistic from each test run in a time-series chart.

    2019-11-15_15-38-44.png

  • Bar chart (values grouped by stat). Plots the statistic from each test run in a bar chart.

    2019-11-15_15-40-29.png

  • Bar chart (values grouped by test run). Plots the statistic from each test run in a bar chart.

    2019-11-15_15-41-16.png

  • Bar chart (values grouped by action). Plots the statistic, broken down to a per command basis if applicable, from each test run in a bar chart.

    2019-11-15_15-42-11.png

  • Histogram: plots the statistic from each test run on a histogram chart

    2019-11-15_15-43-14.png

You will find more options under the Wrench Tool menu on each chart, such as changing units, or selecting Max value instead of Mean value (default), and others.

Chart Legend

You can define the content of the legend at the bottom of the charts.

The following three options are available:

  • None. No legend information displayed.

  • Test runs only. Display test runs that are applicable to the report template, based on the report type settings.

  • Summary data by test runs. Display statistics for each test run in tabular format.

Reload Chart Preview

While building the report template, you can click Reload chart previewto preview the chart to be created when you formally apply the report template.

In this section: