Correlations

baseband_data_classes

Baseband

Baseband(file_name, readlen=-1, num_overflows=0, force_cpu=False, verbose=True)

Create instance of Baseband object. Headers and spec_num always stored on host memory. Raw_data can be stored on either host/device depending on whether GPU is in use. Storing raw data on GPU enables much faster IPFB->PFB->Correlation step. Specnum can be passed to GPU when needed, no need to do so by default.

TODO: Explain what the Baseband class is and what it does??

Parameters:

Name	Type	Description	Default
file_name		Path to baseband binary file to be read.	required
readlen		The number of packets to read. If it is an integer >=1, specifies # of packets to read. If it is a float in (0,1), reads fraction of total packets. Defaults to -1, in which case all packets are read.	-1
num_overflows		Defaults to 0. Number of cycles of int32 wrap that need to be undone in spectrum numbers due to PAST overflows.	0
force_cpu		If True, raw_data will be stored on CPU (NumPy) even if GPU (CuPy) is available. Useful for background pre-fetching.	False
verbose		If False, disable printing file read times	True

Returns:

Name	Type	Description
self	Baseband

__str__

__str__()

Calls print_headers()

print_header

print_header()

Formats and prints a string displaying header info.

get_hist

get_hist(mode=-1)

Get

Parameters:

Name	Type	Description	Default
mode		mode=0 for pol0, 1 for po1, -1 for both.	-1

Returns:

Name	Type	Description
histvals	??	??

BasebandFloat

BasebandFloat(file_name, readlen=-1, rowstart=None, rowend=None, channels=None, chanstart=0, chanend=None, unpack=True, num_overflows=0, force_cpu=False, verbose=True)

Bases: Baseband

Create instance of BasebandFloat.

A child of the Baseband class. BasebandFloat... TODO: describe??

Parameters:

Name	Type	Description	Default
file_name		Path to baseband binary file to be read.	required
readlen		The number of packets to read. If it is an integer >=1, specifies # of packets to read. If it is a float in (0,1), reads fraction of total packets. Defaults to -1, in which case all packets are read.	-1
num_overflows		Defaults to 0. Number of cycles of int32 wrap that need to be undone in spectrum numbers due to PAST overflows.	0
channels		List of channel indices that must be unpacked. If passed, chanstart and chanend ignored.	None
chanstart		Index of channel at which to start selection. Default is 0. Used to instantiate channels if channels is None.	0
chanend		Index of channel at which to end selection. Default is None in which case select up to highest available frequency channel. Used to instantiate channels if channels is None.	None
force_cpu		If True, raw_data will be stored on CPU (NumPy) even if GPU (CuPy) is available.	False
verbose		If False, disable printing file read times	True

BasebandPacked

BasebandPacked(file_name, readlen=-1, rowstart=None, rowend=None, channels=None, chanstart=0, chanend=None, unpack=True, num_overflows=0, force_cpu=False, verbose=True)

Bases: Baseband

Create instance of BasebandPacked.

A child of the Baseband class. BasebandPacked... ??

Parameters:

Name	Type	Description	Default
file_name		Path to baseband binary file to be read.	required
readlen		The number of packets to read. If it is an integer >=1, specifies # of packets to read. If it is a float in (0,1), reads fraction of total packets. Defaults to -1, in which case all packets are read.	-1
num_overflows		Defaults to 0. Number of cycles of int32 wrap that need to be undone in spectrum numbers due to PAST overflows.	0
chanstart		Index of channel at which to start selection. Default is 0.	0
chanend		Index of channel at which to end selection. Default is None in which case select up to highest frequency channel.	None
force_cpu		If True, raw_data will be stored on CPU (NumPy) even if GPU (CuPy) is available.	False

BasebandFileIterator

BasebandFileIterator(file_paths, fileidx, idxstart, acclen, nchunks=None, channels=None, chanstart=0, chanend=None, type='packed', num_overflow=0)

Create an instance of BasebandFileIterator (BFI).

An iterator for iterating through an arbitrary number of baseband files passed to it, acclen spectrums (= 1 chunk) at a time. Typically, a BFI instance is associated with a particular antenna, allowing one to obtain a timestream of acclen-long chunks, which can be correlated and averaged. BFI gracefully handles missing spectra and int32 overflows.

Parameters:

Name	Type	Description	Default
file_paths		Paths to baseband binary files to be read.	required
fileidx		The index of the file in file_paths from which contains idxstart.	required
idxstart		The starting spectrum number.	required
acclen		Accumulation length, i.e. size of each chunks.	required
nchunks		Defaults to None. You need to pass nchunks if you are passing the iterator to zip(). Without nchunks, iteration will stop once BFI runs out of files.	None
chanstart		Index of channel at which to start selection. Default is 0.	0
chanend		Index of channel at which to end selection. Default is None in which case select up to highest frequency channel.	None

fill_arr

fill_arr(specnum, spec_per_packet)

Make the spectrum number array continuous. Raw spec_num has starting spectrum number of each packet. This function expands each packet. Array is required in x-corr computation for handling missing packets.

Parameters:

Name	Type	Description	Default
specnum		Raw spectrum number array. If GPU enabled, it resides on device	required
spec_per_packet		Number of spectrums in a single packet (stored in file header).	required

Returns:

Type	Description
ndarray	Expanded array. If GPU enabled, it resides on device.

add_constant

add_constant(arr, const)

Helper function to add a constant to an array. Function typically used for handling spectrum number int32 wrapping. Parallelized since spectrum number array can be O(1e7) long, and addition needs to happen for each file after a wrap is encountered.

Parameters:

Name	Type	Description	Default
arr		1d array to which constant will be added.	required
const		constant to add	required

get_header

get_header(file_name, verbose=False)

Get header dictionary from (attributes of) baseband file.

Only reads the header of by instantiating a Baseband object with readlen=0. Returns the object's public attributes which is the header information.

Parameters:

Name	Type	Description	Default
file_name		Path to baseband data file.	required
verbose		Defaults to True, in which case header data is printed.	False

Returns:

Type	Description
dict	Dictionary containing header information.

get_rows_from_specnum

get_rows_from_specnum(stidx, endidx, spec_arr)

??

??

Parameters:

Name	Type	Description	Default
stidx		Start index of... ??	required
endidx		End index of... ??	required
spec_arr		??	required

Returns:

Name	Type	Description
l	int??	Left...??
r	int??	Right...??

timing_solution_class

TimingSolution

TimingSolution(query_unix_start, root)

Interface for accessing antenna timing solutions.

The class loads a database of calibration batches and selects the batch containing a requested observation start time. Timing solutions and associated metadata for the selected batch can then be accessed through the class methods. Timing solutions can then also be interpolated as desired for data analysis.

Parameters:

Name	Type	Description	Default
query_unix_start	float or int	Unix timestamp (seconds) used to identify the calibration batch containing the desired observation.	required
root	str or Path	Path to the directory containing the timing solution database. This directory must contain an index.json file and the HDF5 files referenced by the index.	required

Attributes:

Name	Type	Description
root	Path	Root directory of the timing solution database.
batch	dict	Dictionary describing the selected calibration batch.
batch_name	str	Name of the selected batch, in form 'batch_{batch_start_unix}'.
batch_start_unix	float	Start time of the selected batch in Unix seconds.
batch_end_unix	float	End time of the selected batch in Unix seconds.
ref_ant	str	Name of the reference antenna for the selected batch.
non_ref_ants	list of str	Names of the non-reference antennas whose delays are stored in the timing solutions.
UTC_per_spec	float	Conversion rate from spectrum number to absolute UTC time
UTC_offset	float	Initial UTC time corresponding to spectrum number zero.

Raises:

Type	Description
FileNotFoundError	If the timing solution database or its index.json file cannot be found.
ValueError	If query_unix_start is not contained within any calibration batch, or if it is contained within more than one batch.

Notes

Timing solutions are organized into independent, non-overlapping batches. The class selects the unique batch whose time range contains query_unix_start. Moreover, spectra and absolute UTC are interchangable via UTC_per_spec * s + UTC_offset.

spectra_to_unix

spectra_to_unix(spectra)

Converts any desired reference antenna spectrum indices to UNIX timestamps.

Parameters:

Name	Type	Description	Default
spectra	ndarray	Array of reference antenna spectra.	required

Returns:

Type	Description
ndarray	Unix timesamps corresponding to input spectra.

interpolate_delay

interpolate_delay(interp_unix_start, interp_unix_end, dt, extrapolate=True, method='linear')

Interpolate antenna delay timing solutions for entire requested Unix interval.

The timing solutions loaded for the selected batch are interpolated onto a user-defined time grid. Each antenna delay solution is interpolated independently using the specified interpolation method.

Parameters:

Name	Type	Description	Default
interp_unix_start	float	Starting Unix timestamp for the requested interpolation range.	required
interp_unix_end	float	Ending Unix timestamp for the requested interpolation range.	required
dt	float	Spacing between requested output timestamps in seconds.	required
extrapolate	bool	Whether to allow interpolation outside the time range covered by the loaded timing solution. If False, a ValueError is raised when the requested range extends beyond the available data. Default is True.	True
method	str	Interpolation method to use. Currently only "linear" is supported. Default is "linear".	'linear'

Returns:

Name	Type	Description
unix_interp	ndarray	Requested Unix timestamps at which delay solutions have been interpolated.
taus_interp	ndarray	Interpolated antenna delay solutions. Shape is (nant - 1, ntime), where nant - 1 is the number of non-reference antennas and ntime is the number of requested output timestamps.

Raises:

Type	Description
ValueError	If extrapolation is disabled and the requested interpolation range lies outside the available timing solution range.
NotImplementedError	If an unsupported interpolation method is requested.

Notes

Delay solutions are stored internally as self.taus and their corresponding Unix timestamps as self.unix. Linear interpolation is performed independently for each antenna delay stream.

interpolate_delay2

interpolate_delay2(unix_interp, extrapolate=True, break_batch=False, method='linear')

Interpolate antenna delay timing solutions for a requested Unix time array.

The timing solutions loaded for the selected batch are interpolated onto a user-defined time array. Each antenna delay solution is interpolated independently using the specified interpolation method.

Parameters:

Name	Type	Description	Default
unix_interp	ndarray	Unix timestamp array. Must be increasing.	required
extrapolate	bool	Whether to allow interpolation outside the time range covered by the loaded timing solution. If False, a ValueError is raised when the requested range extends beyond the available data. Default is True.	True
method	str	Interpolation method to use. Currently only "linear" is supported. Default is "linear".	'linear'

Returns:

Name	Type	Description
taus_interp	ndarray	Interpolated antenna delay solutions. Shape is (nant - 1, ntime), where nant - 1 is the number of non-reference antennas and ntime is the number of requested output timestamps.

Raises:

Type	Description
ValueError	If extrapolation is disabled and the requested interpolation range lies outside the available timing solution range.
NotImplementedError	If an unsupported interpolation method is requested.

Notes

Delay solutions are stored internally as self.taus and their corresponding Unix timestamps as self.unix. Linear interpolation is performed independently for each antenna delay stream.

interpolate_existing_delays

interpolate_existing_delays(interp_unix_start, interp_unix_end, dt, border_window)

Interpolate timing solutions only around periods with existing data.

Finds all timing solution samples within the requested interval, extends each contiguous data region by border_window seconds, and interpolates delays onto a regular time grid. Regions with no timing solution data are not included.

Parameters:

Name	Type	Description	Default
interp_start	float	Start Unix timestamp of requested range.	required
interp_end	float	End Unix timestamp of requested range.	required
dt	float	Output time resolution in seconds.	required
border_window	float	Time padding applied around the edges of existing data regions.	required

Returns:

Name	Type	Description
unix_interp	ndarray	Unix timestamps where interpolated delays are available.
taus_interp	ndarray	Interpolated delay solutions corresponding to unix_interp.

get_data_mask

get_data_mask(query_unix, tolerance)

Return mask indicating where timing solution samples exist nearby.

Parameters:

Name	Type	Description	Default
query_unix	ndarray	Requested Unix timestamps.	required
tolerance	float	Maximum allowed distance (seconds) between a requested time and an available timing solution sample.	required

Returns:

Type	Description
ndarray	Boolean mask. True where a timing solution sample exists within the tolerance window.

all_blines

all_blines(taus)

For taus that are only with respect to the reference, creates all baseline combinations. By convention the reference antenna has zero delay.

Parameters:

Name	Type	Description	Default
taus	ndarray	All delays, shape (nant-1, ntimes)	required

Returns:

Type	Description
ndarray	All delays for all baselines, shape (nbl, ntimes)

Notes

Beware of convention: ref-nref, and small-large. Means that when subtracting, if A<B, have delayA - delayB = (ref - tauA) - (ref - tauB) = tauB - tauA. Therefore, counterintuitively, you subtract smaller index from larger.

get_all_batches

get_all_batches(root_dir)

Gives start and end times of all batches for which there are timing solutions.

Parameters:

Name	Type	Description	Default
root_dir	str	Directory to the data	required

Returns:

Type	Description
list	List of tuples (start, end) for all the batches

correlations

avg_xcorr_4bit_float_gpu

avg_xcorr_4bit_float_gpu(data0, data1, specnum0, specnum1)

Unified GPU-based 4bit correlation function for a SINGLE BASELINE. If specnum0 and specnum1 are the same array i.e. point to the same location in the memory function returns autocorrelation, else cross-correlation of electric fields

Parameters:

Name	Type	Description	Default
data0	ndarray	n_spectra x n_channels	required
data1	ndarray	n_spectra x n_channels	required
specnum0	list or np array int64	spectrum numbers associated with first dataset	required
specnum1	list or np array int64	spectrum numbers associated with second dataset	required

Returns:

Type	Description
np array complex64	size n_channels

autocorr_4bit

autocorr_4bit(pol)

Compute autocorrelations of 4bit data.

Parameters:

Name	Type	Description	Default
pol		Raw pol baseband data.	required

Returns:

Name	Type	Description
corr	ndarray	Autocorrelation of pol with itsself.

avg_autocorr_4bit

avg_autocorr_4bit(data, specnums)

Compute time-average of autocorrelation for each channel. (??)

Parameters:

Name	Type	Description	Default
data		??	required
specnums		??	required

Returns:

Name	Type	Description
corr	ndarray	Time-averaged autocorrelations.

xcorr_4bit

xcorr_4bit(data0, data1)

Compute cross-correlation for 4-bit data.

Parameters:

Name	Type	Description	Default
data0		2d baseband array.	required
data1		2d baseband array.	required

Returns:

Name	Type	Description
xcorr	ndarray	Cross correlation of data0, data1.

avg_xcorr_4bit

avg_xcorr_4bit(data0, data1, specnums)

Compute time-averaged cross-correlation.

Parameters:

Name	Type	Description	Default
data0		2d baseband array.	required
data1		2d baseband array.	required
specnums		??	required

Returns:

Name	Type	Description
corr	ndarray	1d array, time averaged cross correlation in each channel.

avg_xcorr_4bit_2ant

avg_xcorr_4bit_2ant(data0, data1, specnum0, specnum1, start_idx0, start_idx1)

Compute cross correlation between two antennas, for 4bit data.

??

Parameters:

Name	Type	Description	Default
data0		2d baseband data for antenna #0.	required
data1		2d baseband data for antenna #1	required
specnum0		??	required
specnum1		??	required
start_idx0		??	required
start_idx1		??	required

Returns:

Name	Type	Description
xcorr	ndarray	??

correlations_gpu

unpacking

hist

hist(data, rowstart, rowend, length_channels, bit_depth, mode)

Gets list of values ready for plotting in histogram.

Parameters:

Name	Type	Description	Default
data		?? What shape??	required
rowstart		??	required
rowend		??	required
length_channels		??	required
bit_depth		Either 1 or 4	required
mode		mode=0 for pol0, 1 for pol1, -1 for both.	required

Returns:

Name	Type	Description
histvals	array - like	Binned frequency-of-occurence for each quantization level.

unpack_4bit

unpack_4bit(data, rowstart, rowend, channels, length_channels)

Unpacks the raw baseband data file in 4-bit mode and inflates array of electric field as 64-bit complex numbers (fp32, fp32).. Returns pol0 and pol1 separately.

Parameters:

Name	Type	Description	Default
data	ndarray or ndarray	Raw data file returned from Baseband class. Normally, n_packets x n_bytes_per_packet	required
length_channels	int	Number of channels in the raw baseband file	required
rowstart	int	Row number corresponding to first spectrum (numpy index convention)	required
rowend	int	Row number corresponding to last spectrum (numpy index convention) rowstart, rowend = 10, 20 unpacks 10 spectra starting at 11th spectrum (index 10).	required
channels	list	Channel INDICES in numpy convention (not channel numbers) you'd like to unpack. E.g. channels = [0, 2, 4] corresponds to channels at index 0, 2, and 4.	required

Returns:

Type	Description
(ndarray, ndarray)	Arrays of nrows x len(channels) corresponding to pol0 and pol1. Cupy arrays on device, if GPU in use.

unpack_1bit

unpack_1bit(data, rowstart, rowend, channels, length_channels)

Unpacks the raw baseband data file in 1-bit mode and inflates array of electric field as 64-bit complex numbers (fp32, fp32). Returns pol0 and pol1 separately.

Parameters:

Name	Type	Description	Default
data	ndarray	Raw data file returned from Baseband class. Normally, n_packets x n_bytes_per_packet	required
length_channels	int	Number of channels in the raw baseband file	required
rowstart	int	Row number corresponding to first spectrum (numpy index convention)	required
rowend	int	Row number corresponding to last spectrum (numpy index convention) rowstart, rowend = 10, 20 unpacks 10 spectra starting at 11th spectrum (index 10).	required
channels	list	Channel INDICES in numpy convention (not channel numbers) you'd like to unpack. E.g. channels = [0, 2, 4] corresponds to channels at index 0, 2, and 4. In this mode, channel indices must be continuous, and must start at an even channel index. Total number of channels should also be a multiple of 2.	required

Returns:

Type	Description
(ndarray, ndarray)	Arrays of nrows x len(channels) corresponding to pol0 and pol1. Cupy arrays on device, if GPU in use.

sortpols

sortpols(data, length_channels, bit_mode, rowstart, rowend, chanstart, chanend)

??

??

Parameters:

Name	Type	Description	Default
data		??	required
length_channels		??	required
bit_mode		??	required
rowstart		??	required
rowend		??	required
chanstart		??	required
chanend		??	required

Returns:

Name	Type	Description
pol0	??	??
pol1	??	??