Insights

Get the most out of your Floods

As load testers, we know that analyzing results can sometimes be more difficult than running the load tests themselves, so we've built a new feature, Insights, to help with that. Insights complements our real-time dashboards and allows you to explore your data more thoroughly. It's currently in public beta and is available to all customers with an active Flood account.

What is it

Insights is our data analysis tool for asking and answering your own questions about your floods. From the Insights database, you can access the measurements for your floods, tagged by Project, Stream, Grid, and more. This lets you compare data across any dimension you'd like.

Insights is all about asking and answering the questions you have about your data. Maybe you want to list all passed, failed and total transactions of all Labels in a flood. Maybe you want to build a special Project dashboard, giving you important information about floods in the last 30 days. Or maybe you're trying to understand which regions the errors came from in a specific flood. You can do all this and more with Insights.

  • Get median response time for a particular label across 5 specific floods.

  • Plot concurrency for all floods in a stream by elapsed time.

  • Summarize failed transactions for all floods in two different projects.

  • Build a dashboard and share it with coworkers.

  • Whatever it is that you need to know.

It works by asking questions. You start with a source, filter it to your desired dataset, summarize the data in some way and then visualize. The basic sources are the seven measurement tables but you can source from other questions you've asked before too. This lets you combine measurements, do multi-step aggregations, or even just remix off your previous work.

Once you have some questions, you can make a dashboard. Which is a collection of questions, laid out as you wish, with the option for filters up the top. Finally, save your questions and dashboards for future use and reference.

Available Data

Data becomes available in Insights once a flood has finished. The following measurements are available in the Insights database. Each is their own table.

  • Concurrency: Each row is the maximum concurrency over the minute for the grid it represents. It is not available per label.

  • Failed Rate: Each row is the mean failed transactions per second for the label on the grid. If the label is empty, it's the rate calculated across all labels.

  • Passed Rate: Each row is the mean passed transactions per second for the label on the grid. If the label is empty, it's the rate calculated across all labels.

  • Failed Count: Each row is the total number of failed transactions over the minute for the label on the grid. If the label is empty, it's the rate calculated across all labels.

  • Passed Count: Each row is the total number of passed transactions over the minute for the label on the grid. If the label is empty, it's the rate calculated across all labels.

  • Response Time: Each row is the average response time over the minute for the label and the grid.

  • Transaction Rate: Each row is the mean total transactions per second, i.e., passed and failed, over the minute for the grid. It is not available per label but this can be derived by adding passed and failed per label.

Each row represents a minute and includes columns for:

  • Time: The minute this row represents.

  • Value: The value for this minute.

  • Project, Project Sequence ID, and Project UUID.

  • Stream and Stream UUID.

  • Flood, Flood Sequence ID, and Flood UUID.

  • Grid UUID.

  • Region: A string representing which region the grid was in. E.g. us-east-2.

  • Elapsed Time: How many seconds since the start of the flood this value is for.

You can use the Flood Official collection questions as examples of how to derive metrics from these tables.

Functionality

Under the hood, Insights is powered by Metabase. It's a business intelligence tool that powers all the question and dashboard building. It has excellent technical documentation available on how to use it, and you can do everything in there except ask SQL questions.

Asking and storing questions

Asking a question

From the Insights page, visit the Insights database:

Then select a table to investigate:

You will see the data, and can now filter, summarize and visualize it. Once saved you can share, embed, or use it as the basis of a new question. See the Metabase documentation for the full details of how.

Asking a simple question

Here we are going to ask: What is the highest concurrent users I have ever had? I would want to know the maximum of Value across all rows. Click the Summarize button.

By default it shows the row count as a single statistic. We need to change the aggregation from a count to a maximum. To change it, click Count of rows and then choose Maximum of...

Then choose Value as the column we want the maximum value of.

You should now see the value for the largest Flood you've run in the last 18 months.

If you wanted to see a quick visualization of max concurrency across all Floods, you can group by Flood to get a histogram. For this you could Group by Flood Sequence ID, and it would give you a histogram.

Finally, click Visualization at the bottom to change from a histogram to something else you would prefer.

Advanced Questions

There is a question editor which gives you access to advanced functionality. In the data table view, you can click the Show Editor button to bring it up.

This will take you to the advanced editor, where you can source your questions from previously saved questions or earlier steps, access more aggregations and build formulas.

For more information about how to build your questions, please consult the excellent Metabase technical documentation.

Storing Questions

Every user has their own Personal Collection where they can store Questions and Dashboards.

After saving you can organize in sub-collections as desired. You can access your Personal Collection and the Flood Official collections by clicking on Browse All Items from the Insights page.

Flood will periodically add new questions and dashboards to the Flood Official collection which all users have access to.

Creating and storing dashboards

To create a dashboard, first, save a question. Once saved, you are asked if you want to add it to a dashboard.

And then click Create a new dashboard.

You'll be asked where to store it. Just like questions, you can store in your personal collection and then access it and move it into organized sub-collections as desired. For more information on what to do with dashboards, see the Metabase technical documentation.