BFI
In [9]:
Copied!
import numpy as np
import cupy as cp
import matplotlib.pyplot as plt
import os
from os import path
import sys
sys.path.insert(0, "/home/thomasb/")
# Baseband Data Classes script
from albatros_analysis.src.correlations import baseband_data_classes as bdc
# Baseband utils script
from albatros_analysis.src.utils import baseband_utils as butils
from albatros_analysis.scripts.xcorr import helper as hxc
import numpy as np
import cupy as cp
import matplotlib.pyplot as plt
import os
from os import path
import sys
sys.path.insert(0, "/home/thomasb/")
# Baseband Data Classes script
from albatros_analysis.src.correlations import baseband_data_classes as bdc
# Baseband utils script
from albatros_analysis.src.utils import baseband_utils as butils
from albatros_analysis.scripts.xcorr import helper as hxc
This is to show people how to use the Baseband Object, in particular how to go through BFI objects.
First, baseband data is saved in spectra. These have time resolution of about 16 microseconds and frequency resolution of about 60 kHz.
In [2]:
Copied!
T_SPECTRA = 4096/250e6
dfreq = 250e6/4096
print('Time resolution:', T_SPECTRA*1e6, 'microsec')
print('Frequency resolution', dfreq/1e3, 'kHz')
T_SPECTRA = 4096/250e6
dfreq = 250e6/4096
print('Time resolution:', T_SPECTRA*1e6, 'microsec')
print('Frequency resolution', dfreq/1e3, 'kHz')
Time resolution: 16.384 microsec Frequency resolution 61.03515625 kHz
In [3]:
Copied!
start_ts = 1753210000
end_ts = 1753220000
path_data = "/scratch/mohanagr/summer_2025/baseband/mars1"
chunk_size = 1000000
nchunks = 20
chanstart = 1834
chanend = 1852
start_ts = 1753210000
end_ts = 1753220000
path_data = "/scratch/mohanagr/summer_2025/baseband/mars1"
chunk_size = 1000000
nchunks = 20
chanstart = 1834
chanend = 1852
First we have to determine what exact files your data lives in. If your interval spans several minutes or hours, this will include many different files. We also need to find out what the starting index of your data is within that first file.
In [4]:
Copied!
try:
files, idx = butils.get_init_info(start_ts, end_ts, path_data)
except Exception as e:
print(e)
print(f"WARNING: MISTAKE IN FILE CHECKER!!")
sys.exit()
print('first file', files[0])
print('idx within first file:', idx)
print('last file', files[-1])
try:
files, idx = butils.get_init_info(start_ts, end_ts, path_data)
except Exception as e:
print(e)
print(f"WARNING: MISTAKE IN FILE CHECKER!!")
sys.exit()
print('first file', files[0])
print('idx within first file:', idx)
print('last file', files[-1])
1753209977 1753219967 first file /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753209977.raw idx within first file: 1403809 last file /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753219967.raw
Next, we have to get the channel indices we want within the data. The channel numbers we use (integer multiples of the channel resolution) may not be the same channel indices within the file.
In [5]:
Copied!
# Set up the number of channels we look through
print("Setting Antenna as BFI Objects", '\n')
channels = np.asarray(bdc.get_header(files[0])["channels"],dtype='int64')
chanstart_idx = np.where(channels == chanstart)[0][0]
chanend_idx = np.where(channels == chanend)[0][0]
nchans = chanend_idx - chanstart_idx
print('channel start', chanstart, 'with file index', chanstart_idx)
print('channel end', chanend, 'with file index', chanend_idx)
# Set up the number of channels we look through
print("Setting Antenna as BFI Objects", '\n')
channels = np.asarray(bdc.get_header(files[0])["channels"],dtype='int64')
chanstart_idx = np.where(channels == chanstart)[0][0]
chanend_idx = np.where(channels == chanend)[0][0]
nchans = chanend_idx - chanstart_idx
print('channel start', chanstart, 'with file index', chanstart_idx)
print('channel end', chanend, 'with file index', chanend_idx)
Setting Antenna as BFI Objects Not reading any data channel start 1834 with file index 284 channel end 1852 with file index 302
Now we define an antenna object. This is essentially the interface that lets us interact with and parse through data effectively and efficiently
In [6]:
Copied!
nchans = chanend - chanstart
ant = bdc.BasebandFileIterator(
files,
0,
idx,
chunk_size,
nchunks,
chanstart=chanstart_idx,
chanend=chanend_idx,
type="float",
)
print('BFI object type', type(ant))
print('acclen', ant.acclen)
print('nchunks', ant.nchunks)
nchans = chanend - chanstart
ant = bdc.BasebandFileIterator(
files,
0,
idx,
chunk_size,
nchunks,
chanstart=chanstart_idx,
chanend=chanend_idx,
type="float",
)
print('BFI object type', type(ant))
print('acclen', ant.acclen)
print('nchunks', ant.nchunks)
ACCLEN RECEIVED IS 1000000 took 0.438 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753209977.raw START SPECNUM IS 602494209 obj start at 601090400 submitted next file queue job /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210031.raw BFI object type <class 'albatros_analysis.src.correlations.baseband_data_classes.BasebandFileIterator'> acclen 1000000 nchunks 20
took 0.451 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210031.raw took 0.476 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210084.raw took 0.466 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210136.raw took 0.438 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210191.raw took 0.402 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210245.raw took 0.418 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210300.raw took 0.406 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210353.raw
We want to find the starting spectrum number for our data. Spectra are a proxy for time, so we like knowing what spectrum number we start out at.
In [7]:
Copied!
start_specnum = ant.spec_num_start
print('Initial Specnum:', start_specnum)
start_specnum = ant.spec_num_start
print('Initial Specnum:', start_specnum)
Initial Specnum: 602494209
Now we can iterate through the chunks. This is the method we use to read through data when you want to look at data individually, or want to parse through lots of data in a row without demolishing your ram, and while being able to deal with holes in your data.
In [8]:
Copied!
for chunkidx, chunk in enumerate(ant):
print(f'\n----- Chunk {chunkidx}/{nchunks-1} ------')
data = cp.zeros((chunk_size, nchans), dtype="complex64") #remember that BDC returns complex64. wanna do phase-centering in 128.
print('CHUNK IDX', chunkidx)
print('NCHUNKS', nchunks)
if chunkidx == nchunks:
print('offbyone error!!')
sys.exit()
# we want to check how much of the data is present. if it's too much, we skip the chunk
perc_missing = (1 - len(chunk["specnums"]) / chunk_size) * 100
if perc_missing > 10:
print(f'big problem! plenty of data missing {perc_missing}. abort!')
if chunkidx == 0: #logic here is that maybe after a reboot the data is corrupted or something: will let chunk 1 free
continue
sys.exit() #otherwise we will need to check what's going on. may be a sign of something ystemically wrong in data
# want the spectrum indices for the entire chunk
spec_idxs = chunk['specnums']-(start_specnum + chunk_size*chunkidx)
print('ant start idx', chunk['specnums'][0])
# make the chunk continuous, and write it to the data array
bdc.make_continuous_gpu(cp.asarray(chunk['pol0']),
spec_idxs,
np.arange(nchans),
chunk_size,
nchans=nchans,
out=data)
# then you can do whatever with your data, chunk by chunk
print('data shape', data.shape, 'percentage missing', perc_missing)
for chunkidx, chunk in enumerate(ant):
print(f'\n----- Chunk {chunkidx}/{nchunks-1} ------')
data = cp.zeros((chunk_size, nchans), dtype="complex64") #remember that BDC returns complex64. wanna do phase-centering in 128.
print('CHUNK IDX', chunkidx)
print('NCHUNKS', nchunks)
if chunkidx == nchunks:
print('offbyone error!!')
sys.exit()
# we want to check how much of the data is present. if it's too much, we skip the chunk
perc_missing = (1 - len(chunk["specnums"]) / chunk_size) * 100
if perc_missing > 10:
print(f'big problem! plenty of data missing {perc_missing}. abort!')
if chunkidx == 0: #logic here is that maybe after a reboot the data is corrupted or something: will let chunk 1 free
continue
sys.exit() #otherwise we will need to check what's going on. may be a sign of something ystemically wrong in data
# want the spectrum indices for the entire chunk
spec_idxs = chunk['specnums']-(start_specnum + chunk_size*chunkidx)
print('ant start idx', chunk['specnums'][0])
# make the chunk continuous, and write it to the data array
bdc.make_continuous_gpu(cp.asarray(chunk['pol0']),
spec_idxs,
np.arange(nchans),
chunk_size,
nchans=nchans,
out=data)
# then you can do whatever with your data, chunk by chunk
print('data shape', data.shape, 'percentage missing', perc_missing)
print shape of my pols 1000000 18 ----- Chunk 0/19 ------ CHUNK IDX 0 NCHUNKS 20 ant start idx 602494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 874879 18 submitted next file queue job /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210084.raw print shape of my pols 125121 18 ----- Chunk 1/19 ------ CHUNK IDX 1 NCHUNKS 20 ant start idx 603494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 2/19 ------ CHUNK IDX 2 NCHUNKS 20 ant start idx 604494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 3/19 ------ CHUNK IDX 3 NCHUNKS 20 ant start idx 605494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 4/19 ------ CHUNK IDX 4 NCHUNKS 20 ant start idx 606494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 153567 18 submitted next file queue job /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210136.raw print shape of my pols 846433 18 ----- Chunk 5/19 ------ CHUNK IDX 5 NCHUNKS 20 ant start idx 607494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 6/19 ------ CHUNK IDX 6 NCHUNKS 20 ant start idx 608494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 7/19 ------ CHUNK IDX 7 NCHUNKS 20 ant start idx 609494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 432255 18 submitted next file queue job /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210191.raw print shape of my pols 567745 18 ----- Chunk 8/19 ------ CHUNK IDX 8 NCHUNKS 20 ant start idx 610494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 9/19 ------ CHUNK IDX 9 NCHUNKS 20 ant start idx 611494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 10/19 ------ CHUNK IDX 10 NCHUNKS 20 ant start idx 612494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 710943 18 submitted next file queue job /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210245.raw print shape of my pols 289057 18 ----- Chunk 11/19 ------ CHUNK IDX 11 NCHUNKS 20 ant start idx 613494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 12/19 ------ CHUNK IDX 12 NCHUNKS 20 ant start idx 614494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 13/19 ------ CHUNK IDX 13 NCHUNKS 20 ant start idx 615494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 989631 18 submitted next file queue job /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210300.raw print shape of my pols 10369 18 ----- Chunk 14/19 ------ CHUNK IDX 14 NCHUNKS 20 ant start idx 616494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 15/19 ------ CHUNK IDX 15 NCHUNKS 20 ant start idx 617494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 16/19 ------ CHUNK IDX 16 NCHUNKS 20 ant start idx 618494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 17/19 ------ CHUNK IDX 17 NCHUNKS 20 ant start idx 619494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 268319 18 submitted next file queue job /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753210353.raw print shape of my pols 731681 18 ----- Chunk 18/19 ------ CHUNK IDX 18 NCHUNKS 20 ant start idx 620494209 data shape (1000000, 18) percentage missing 0.0 print shape of my pols 1000000 18 ----- Chunk 19/19 ------ CHUNK IDX 19 NCHUNKS 20 ant start idx 621494209 data shape (1000000, 18) percentage missing 0.0
In the case that you want two (or more!) antenna, and have access to some spectrum number corrections, we load things up slightly differently. Spectrum number offsets are always with respect to the first antenna (usually MARS1), and it has zero offset with respect to itself. From here, it's all the same, with one object for each individual antenna data stream.
In [11]:
Copied!
specnum_offsets = [0, -1884333]
paths = ["/scratch/mohanagr/summer_2025/baseband/mars1", "/scratch/mohanagr/summer_2025/baseband/mars2"]
idxs, files = hxc.get_init_info_all_ant(start_ts, end_ts, specnum_offsets, paths)
specnum_offsets = [0, -1884333]
paths = ["/scratch/mohanagr/summer_2025/baseband/mars1", "/scratch/mohanagr/summer_2025/baseband/mars2"]
idxs, files = hxc.get_init_info_all_ant(start_ts, end_ts, specnum_offsets, paths)
1753209977 1753219967 Using delta of 64.0 seconds to check for gaps between files took 0.447 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars1/17532/1753209977.raw 1753210000 1753219991 Using delta of 64.0 seconds to check for gaps between files took 0.651 seconds to read raw data on /scratch/mohanagr/summer_2025/baseband/mars2/17532/1753210000.raw specnums [602494209, 604335912] spec offsets [ 0 -1884333] PROCESSING ANTENNA 0 initial offset wrt ref ant 0 idxs before correction 1403809 1403809 spec offset 0 init offset 0 correction 0 after correction 1403809 1403809 PROCESSING ANTENNA 1 initial offset wrt ref ant -1841703 idxs before correction 1403809 0 spec offset -1884333 init offset -1841703 correction -42630 after correction 1403809 42630