Explorers

Explorers will help you to compare 1000s of AI experiments with a few clicks. Explorers are the main tools that Aim is built around.

In this section we will go through the Aim explorers, introduce their features and how to use them.

Metrics Explorer

Overview

Use Metrics explorer to search and compare 1000s of ML training metrics.

The Metrics Explorer allows you to search, group and compare your metrics. Due to this and number of other visual features on the Metrics Explorer, you will save considerable amounts of time when comparing experiments- compared to other open-source experiment tracking tools.

The Metric Explorer has the following main sections:

  • Metrics Select: to select the metrics for exploration

  • Search bar: to query the runs for exploration

  • Charts explorer: the space where the metrics are rendered

  • Metrics modifiers: all the grouping, chart division and other metrics modifier tools

  • Context table: the full information about the selected metrics is available

There is also an advanced mode of search is available too where you can use the full Aim QL (more on this further in this section).

There are two ways you can query metrics and runs

An overview of what you can do with queried metrics - the modifiers:

metrics_explorer

Select metrics and query runs

On the Metrics Explorer, there is + Metrics button. Once pressed, a dropdown will appear with all your tracked metrics and their contexts flattened. The dropdown is searchable - so you can get to your metric of interest within a stroke!

select_form_dropdown

The Search bar is located below the + Metric button. It allows to do a pythonic query (that is eval-ed as python statement) over every param you have tracked.

Search runs with Aim QL

select_form_filter

Advanced Search mode

Once you press the Enable advanced search mode button underneath the main Search button, it will enable the full Aim QL search editor - to query the metrics, the runs via full Aim QL

Here is an example:

((metric.name == 'bleu' and metric.context.subset == 'val') or (metric.name == 'loss' and metric.context.subset == 'train')) and 1e-5 < run.hparams.learning_rate < 1e-2

select_form_advanced_search

Group by any parameter

Grouping selected metrics by any tracked params will allow you to quickly distinguish the most impactful params, decisions you have made (the preprocessing steps, the hyperparams etc).

The parameters include not only the ones you have tracked but also the native Aim objects too such as

  • metric.name

  • metric.context.[context_key]

  • run.hash

There are several ways you can group the selected metrics and runs - by color, by stroke and by chart.

Group by Color

Use this to divide the selected metrics into different clusters based on selected values of params. Each cluster gets colored differently.

There are a number of options available when grouping

  • group by values - divides into clusters as per the values of selected params)

  • reverse grouping - divides into clusters by every param except for the chosen one.

The grouping colors are picked randomly, however it is possible to fix with the advanced coloring features.

Here are the features in the advanced mode:

  • Fix the colors of the grouping

  • Control the color palette to use during the grouping

groupby_color

Group by Stroke

Groups the metrics by a stroke style. Has all the rest of the other features available on the color grouping except the advanced mode.

groupby_stroke

Group by Chart

The end result of using this feature: divides into subplots based on the value of the selected params. Why this is a grouping mechanism? It groups the metrics belonging to the same group into separate charts.

groupby_chart

Aggregate metrics

The metrics aggregation helps to quickly see the trends of each individual group of metrics. See more about metrics grouping.

There are two aspects of aggregation you can control:

  • the trend-line

  • the area that the group of metricsc take

The trend-line calculation methods:

  • Mean

  • Median

  • Min

  • Max

The area calcualtion methods:

  • None (when you’d like to remove the area)

  • Min/Max

  • Mean +/- Standard Deviation

  • Mean +/- Standard Error

  • Confidence Interval (95%)

Pls see the screenshot:

aggr_line_mean

Axes properties

Axes properties section is for aligning metrics by time, epoch or another metric and for setting axes range manually.

Alignment:

Following types of metrics alignment are available: Step, Epoch, Relative Time, Absolute Time and Custom Metric. By default, metrics aligned by Step.

Step

By setting metrics alignment to Step, x-axis values will represent the steps of tracked metrics.

alignment_step

Epoch

By setting metrics alignment to Epoch, x-axis values will represent the epochs of tracked metrics.

alignment_epoch

Relative Time

By setting metrics alignment to Relative Time, x-axis values will represent by HH:mm:ss, duration of tracking process.

alignment_relative

Absolute Time

By setting metrics alignment to Absolute Time, x-axis values will represent by date HH:mm:ss D MMM, YY, since the start date of the first run until the last run.

alignment_absolute

Custom Metric

By setting metrics alignment to Custom Metric, x-axis values will represent selected metric values, you can detect correlations between queried metrics and selected metric.

Set axes range:

To fix an axis range across all the charts, set the corresponding axis minimum and maximum bounds in the form.

set-axes-range

Axes Scale

Axes Scale section gives ability to display axes scale’s linear or logarithmic.

By default, axes scale’s are Linear.

Linear Scale

axes_scale_linear

X-axis scale: Linear, Y-axis scale: Log

y-axis-scale-log

X-axis scale: Log, Y-axis scale: Linear

x-axis-scale-log

Log Scale

axes-scale-log

Chart Smoothing

While smoothing the chart, the data points are modified so individual points higher than the adjacent points (presumably because of noise) are reduced, and points that are lower than the adjacent points are increased leading to a smoother signal. You can select curve interpolation methods: Linear or Cubic. By default, chart smoothing in Exponential moving average mode and curve interpolation method is Linear.

Exponential moving average

An exponential moving average, also known as an exponentially weighted moving average (EWMA), is a first-order infinite impulse response filter that applies weighting factors which decrease exponentially.

smoothing_ema

Centered moving average

When you center the moving averages, the data points are placed at the center of the range rather than the end of it. This is done to position the moving average values at their central positions in time.

smoothing_cma

Ignore outliers

An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. Examination of the data for unusual observations that are far removed from the mass of data. These points are often referred to as outliers.

ignore-outliers-off

Excluding outliers can cause your results to become statistically significant. By default, outliers are ignored.

ignore-outliers-on

Highlight Modes

Highlighting functionality is useful for filtering metrics and highlight only hovered metric. Following types of highlighting mode are available: Highlight Off, Highlight Metric on Hover, and Highlight Run on Hover. By default, highlighting mode is the Highlight Run on Hover.

Highlight Off

By setting Highlight mode Off, there is no highlighting functionality on hover.

highlight_off

Highlight Metric on Hover

By setting Highlight mode Metric on Hover, mouse point closest metric highlights and other metrics displays with opacity.

highlight_metric

Highlight Run on Hover

By setting Highlight mode Run on Hover, mouse point closest metric highlights and highlighted metric corresponding run also highlights other metrics displays with opacity.

highlight_run

Set tooltip parameters

You can select tooltip parameters to show params and values in tooltip Params section. You can select hide or show button to display or hide tooltip on hover.

tooltip-parameters-popover

select-tooltip-parameters

Apply zoom on charts

Zoom In

zoom-in-popover.png

zoom-in-action.png

zoom-out.png

Zoom Out

zoom_out_popover

Export chart as image

Metric explorer also, gives ability to export your chart as image. By clicking export button from control panel, will be opened chart preview modal. You can change exportable chart image width, single chart height, set image name and format.

export_preview

Following formats of chart export are available: SVG, JPEG, PNG.

export-format

Images Explorer

Overview

Track intermediate images search easily by using select form functional compare them on the Images Explorer by using reach controls panel.

Images explore overview

Features:

Query any image

Use select form to easily query any image. There are two option to query images using dropdown, by using Aim QL language and advance mode for Aim QL.

Images explore overview

  • Click on Images button

  • Select options you are want to use in query

  • Click on the Search button

Images explore overview

Images explore overview

  • Click on pencel icon in the right side of select form to show input

  • Type advance Aim QL query

  • Click on the Search button

Images explore overview

Group image by run parameters

Use select grouping dropdown which is located in the right top corner of the image explore page.

Images explore overview

  • Click on grouping button

  • Select fields by which you want to groupe images

Grouping will be apply after each field selection also you can select grouping mode (Group or Reverse)

Images explore overview

Image explorer right controls panel

Any change in controls will help to explore images better on the workspace

Images size manipulation control

  • Click on image property button

  • Select value from dropdown to align image. (by default dropdown value is Height). Use slider to configure value for scale relative to window size by default scale value is 15%.

    • By height

      Images explore overview

    • By width

      Images explore overview

    • Original size

      Images explore overview

  • Use image rendering variation by default value of this control is Pixelated

    Images explore overview

Images sorting control

  • Click on image sorting button

  • Select fields for sorting images. Selection ordering is meaningful and data will be sorting by selection order. Bellow is visible Ordered By list where contains all selected fields from dropdown. You can remove any already selected field by clicking on x icon or change sorting direction by clicking radio button Asc or Desc. Default selected direction is Asc.

  • For reset all existing sorting fields you can simply click on Reset Sorting button

Images sorting popover

Set tooltip parameters

You can select tooltip parameters to show params and values in tooltip Params section. You can select hide or show button to display or hide tooltip on hover.

tooltip-parameters-popover

Params Explorer

Overview

Params explorer helps you to represent high dimensional data as a multi-dimensional visualization. Features:

Params explore

Query any metrics and params

Select params and metrics from dropdown

Select params and metrics from dropdown

Search runs with Aim QL

Search runs

Grouping

Group by color, stroke, or chart with selected parameters

Grouping

Curve interpolation

By clicking on the Curve interpolation button in the Controls panel, it’s possible to make lines from straight to curve to show correlations between non-adjacent axes.

Curve interpolation

Color indicator

By clicking on the Color indicator button in the Controls panel, it’s possible to turn on lines gradient coloring by the last dimension.

Color indicator

Scatters Explorer

Scatter explorer gives ability to visualize correlations between metric last value data with hyper-parameter.

It represents graph where corresponding values from a set of data are placed as points on a coordinate plane. A relationship between the points is sometimes shown to be positive, negative, strong, or weak.

scatters-explore

Abilities provided by Scatter explorer

Select params and metrics from X and Y axes dropdowns to align metric last value data with hyper-parameter.

  • X axis

x axis dropdown

  • Y axis

y axis dropdown

Also, you can search runs with Aim QL

search runs

Easily group data by color and chart with selected parameters.

  • By Color

Grouping color

  • By Chart

Grouping chart line

A trend line is a straight line that best represents the points on a scatter plot. The trend line may go through some points but need not go through them all.

trend line

From trend line options popover you can change regression from Linear (by default) to LOESS(locally weighted smoothing), which creates a smooth line through a scatter plot to help you to see relationship between variables and foresee trends. Also, you can change the bandwidth with slider

trend line loess

Scatter explorer also, gives ability to export your chart as image.

By clicking export button from control panel, will be opened chart preview modal. You can change exportable chart image width, single chart height, set image name and format.

export chart

Following image formats are available export: SVG, JPEG, PNG.

export format