mighty.monitor.monitor.ParamRecord

class mighty.monitor.monitor.ParamRecord(param, monitor_level=0)

A parameter record, created by a monitor, that tracks parameter statistics like gradient variance, sign flips on update step, etc.

Parameters:

paramnn.Parameter

Model parameter.

monitor_levelMonitorLevel, optional

The extent of keeping the statistics. Default: MonitorLevel.DISABLED

Methods

__init__(param[, monitor_level])
count_sign_flips(new_data, prev_data)	Count the no.
reset()	Resets the current state and all saved variables.
update_grad_variance()	Updates the gradient variance, need for Signal-to-Noise ratio estimation.
update_signs()	Updates the number of sign flips by comparing with the previously stored tensor.

static count_sign_flips(new_data, prev_data)

Count the no. of sing flips in prev_data -> new_data.

The default implementation is

sum(new_data * prev_data)

Parameters:

new_data, prev_datatorch.Tensor

New and previous tensors.

Returns:

sign_flipsint

The no. of signs flipped.

reset()

Resets the current state and all saved variables.

update_grad_variance()

Updates the gradient variance, need for Signal-to-Noise ratio estimation.

update_signs()

Updates the number of sign flips by comparing with the previously stored tensor.

Returns:

sign_flipsfloat

Normalized number of sign flips in range [0, 1].