Preprocessingο
This notebook demonstrates how to preprocess astronomical images using the ezphot library. Preprocessing is a crucial step in astronomical data reduction that involves:
Bias correction: Removing electronic bias from CCD detectors
Dark current correction: Subtracting thermal noise from long exposures
Flat field correction: Correcting for pixel-to-pixel sensitivity variations
Weβll walk through the complete workflow from data loading to final corrected images.
1. Load Dataο
The first step in preprocessing is to load your astronomical data. Weβll use two main approaches:
``DataBrowser``: Query and filter images from your local data directory
``SDTDataQuerier``: Sync data from a central database (optional but recommended for large datasets)
π‘ Tip: The
SDTDataQuerierclass provides methods to sync data from a central database to your local workspace, making it easy to work with large astronomical datasets without storing everything locally.
1.1. Sync Data with SDTDataQuerier (Optional)ο
If you have access to a central database, you can sync specific observations to your local workspace. This is particularly useful when working with large datasets or when you only need specific observations.
[ ]:
from ezphot.utils import SDTDataQuerier
# If you connected the database, you can easily sync the data from the database.
sdtquery = SDTDataQuerier()
sdtquery.help()
Help for SDTDataQuerier
Public methods:
- show_obsdestdata(foldername: str, show_only_numbers: bool = False, pattern: str = '*.fits')
- show_obsdestfolder(folder_key: str = '*')
- show_obssourcedata(foldername: str, show_only_numbers: bool = False, pattern: str = '*.fits')
- show_obssourcefolder(folder_key: str = '*')
- show_scidestdata(targetname: str, show_only_numbers: bool = False, key: str = 'filter', pattern: str = '*.fits')
- show_scisourcedata(targetname: str, show_only_numbers: bool = False, key: str = 'filter', file_pattern: str = '*.fits')
- show_scisourcefolder(folder_key: str = '*')
- sync_obsdata(foldername: str, file_pattern: str = '*.fits', ignore_exists: bool = True)
- sync_scidata(targetname: str, file_pattern: str = '*.fits', ignore_exists: bool = True)
Now letβs sync the data for our specific observation date. Weβll use the sync_obsdata method to copy FITS files matching our pattern:
[8]:
# sync_scidata method will run multiple rsync between Science Source directory and destination directory.
print('Source directory:', sdtquery.helper.config['SDTDATA_OBSSOURCEDIR'])
print('Target directory:', sdtquery.helper.config['SDTDATA_OBSDESTDIR'])
obsdate_name = '2025-09-05_gain2750'
sdtquery.sync_obsdata(foldername = obsdate_name, file_pattern = '*T00290*.fits')
Source directory: /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata
Target directory: /home/hhchoi1022/ezphot/data/obsdata/7DT
Running:Running:Running:Running:Running:Running:Running:Running:Running:Running:Running: rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT07/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750/ rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT10/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750/rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT01/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750/rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT14/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT12/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750/rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT08/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT16/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750/
rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT06/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750/rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT11/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/
rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT03/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/rsync -av --info=progress2 --no-inc-recursive --prune-empty-dirs --ignore-existing --include */ --include *T00290*.fits --exclude * /home/hhchoi1022/ezphot/data/connecteddata/7DT/obsdata/7DT13/2025-09-05_gain2750/ /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750/
building file list ... building file list ... building file list ... building file list ... building file list ... building file list ... building file list ... building file list ... building file list ... building file list ... building file list ... done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)
sent 350 bytes received 12 bytes 241.33 bytes/sec
total size is 734,123,520 speedup is 2,027,965.52
done
sent 341 bytes received 12 bytes 235.33 bytes/sec
total size is 734,123,520 speedup is 2,079,670.03
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)
sent 358 bytes received 12 bytes 246.67 bytes/sec
total size is 734,123,520 speedup is 1,984,117.62
done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)done
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/7)
sent 361 bytes received 12 bytes 248.67 bytes/sec
total size is 734,123,520 speedup is 1,968,159.57
sent 360 bytes received 12 bytes 248.00 bytes/sec
total size is 734,123,520 speedup is 1,973,450.32
sent 359 bytes received 12 bytes 247.33 bytes/sec
total size is 734,123,520 speedup is 1,978,769.60
sent 359 bytes received 12 bytes 247.33 bytes/sec
total size is 734,123,520 speedup is 1,978,769.60
sent 358 bytes received 12 bytes 246.67 bytes/sec
total size is 734,123,520 speedup is 1,984,117.62
sent 358 bytes received 12 bytes 246.67 bytes/sec
total size is 734,123,520 speedup is 1,984,117.62
sent 361 bytes received 12 bytes 248.67 bytes/sec
total size is 734,123,520 speedup is 1,968,159.57
sent 357 bytes received 12 bytes 246.00 bytes/sec
total size is 734,123,520 speedup is 1,989,494.63
π Sync Progress: The rsync process copies FITS files from the source directory to your local destination directory. This ensures you have the latest data while maintaining efficient storage usage.
π 1.2. Query Data with DataBrowserο
The DataBrowser class allows us to query and filter our synced data. Here weβll explore the available keys and their values to understand what data we have access to.
Key Features:
Filter by observatory, telescope, object name, filter, image type, and observation date
Load data as
ScienceImageobjects or metadata tablesFlexible search patterns for file matching
[11]:
from ezphot.utils import DataBrowser
dbrowser = DataBrowser('obsdata')
dbrowser.obsdate = obsdate_name
dbrowser.search('*.fits', return_type = 'path')
[INFO] Found 66 files matching '/home/hhchoi1022/ezphot/data/obsdata/*/*/2025-09-05_gain2750/*.fits'
[11]:
['/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_233955_T00290_m700_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234653_T00290_m725_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234508_T00290_m725_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234838_T00290_m725_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234140_T00290_m700_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_233809_T00290_m700_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_233808_T00290_m450_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_234653_T00290_m475_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_234508_T00290_m475_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_233954_T00290_m450_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_234839_T00290_m475_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_234139_T00290_m450_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750/7DT06_20250905_234549_T00290_m575_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750/7DT06_20250905_234923_T00290_m575_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750/7DT06_20250905_233828_T00290_m550_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750/7DT06_20250905_234736_T00290_m575_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750/7DT06_20250905_234015_T00290_m550_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750/7DT06_20250905_234202_T00290_m550_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750/7DT13_20250905_233955_T00290_m875_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750/7DT13_20250905_234653_T00290_m400_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750/7DT13_20250905_234508_T00290_m400_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750/7DT13_20250905_234838_T00290_m400_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750/7DT13_20250905_234140_T00290_m875_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750/7DT13_20250905_233810_T00290_m875_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750/7DT12_20250905_234837_T00290_m875_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750/7DT12_20250905_233955_T00290_m850_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750/7DT12_20250905_234140_T00290_m850_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750/7DT12_20250905_234652_T00290_m875_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750/7DT12_20250905_233810_T00290_m850_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750/7DT12_20250905_234506_T00290_m875_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750/7DT10_20250905_234158_T00290_m800_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750/7DT10_20250905_234724_T00290_m825_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750/7DT10_20250905_233824_T00290_m800_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750/7DT10_20250905_234011_T00290_m800_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750/7DT10_20250905_234911_T00290_m825_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750/7DT10_20250905_234537_T00290_m825_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750/7DT07_20250905_234511_T00290_m525_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750/7DT07_20250905_234656_T00290_m525_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750/7DT07_20250905_233957_T00290_m650_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750/7DT07_20250905_233811_T00290_m650_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750/7DT07_20250905_234142_T00290_m650_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750/7DT07_20250905_234841_T00290_m525_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750/7DT16_20250905_234516_T00290_r_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750/7DT16_20250905_233815_T00290_g_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750/7DT16_20250905_234145_T00290_g_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750/7DT16_20250905_234702_T00290_r_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750/7DT16_20250905_234000_T00290_g_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750/7DT16_20250905_234846_T00290_r_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750/7DT01_20250905_234157_T00290_i_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750/7DT01_20250905_234730_T00290_m675_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750/7DT01_20250905_234918_T00290_m675_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750/7DT01_20250905_234010_T00290_i_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750/7DT01_20250905_233822_T00290_i_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750/7DT01_20250905_234542_T00290_m675_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234014_T00290_m600_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234551_T00290_m500_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234202_T00290_m600_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234928_T00290_m500_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_233825_T00290_m600_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234739_T00290_m500_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234002_T00290_m625_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_233817_T00290_m625_1x1_100.0s_0000.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234659_T00290_m425_1x1_100.0s_0001.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234147_T00290_m625_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234843_T00290_m425_1x1_100.0s_0002.fits',
'/home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234513_T00290_m425_1x1_100.0s_0000.fits']
[12]:
# If you wish to get the ScienceImage object,
target_imgset = dbrowser.search('*.fits', return_type = 'science')
target_imglist = target_imgset.target_images
# If you wish to get the metadata table from the keys,
metadata_table = dbrowser.search('*.fits', return_type = 'imginfo')
[INFO] Found 66 files matching '/home/hhchoi1022/ezphot/data/obsdata/*/*/2025-09-05_gain2750/*.fits'
Loading Science Images: 100%|ββββββββββ| 66/66 [00:01<00:00, 51.40it/s]
[INFO] Found 66 files matching '/home/hhchoi1022/ezphot/data/obsdata/*/*/2025-09-05_gain2750/*.fits'
Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT07/2025-09-05_gain2750Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750
Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT01/2025-09-05_gain2750Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT12/2025-09-05_gain2750
Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT10/2025-09-05_gain2750
Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT06/2025-09-05_gain2750Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750
Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT16/2025-09-05_gain2750
Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750
Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750Loaded 6 FITS files from /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT13/2025-09-05_gain2750
Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6Total FITS files combined: 6
Collecting ImgInfo: 100%|ββββββββββ| 11/11 [00:00<00:00, 33240.16it/s]
[13]:
metadata_table
[13]:
Table length=66
| file | altitude | azimuth | ccd-temp | gain | filter | imgtype | exptime | obsdate | locdate | jd | mjd | binning | object | objctid | obsmode | specmode | note | telescop |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| str121 | str32 | str32 | str32 | str21 | str4 | str5 | str32 | str23 | str23 | str32 | str32 | str21 | str6 | str32 | str4 | str7 | str4 | str5 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_233955_T00290_m700_1x1_100.0s_0001.fits | 39.837596 | 179.72268 | -10.0 | 2750 | m700 | LIGHT | 100.0 | 2025-09-05T23:39:55.000 | 2025-09-05 19:39:55.000 | 2460924.486053241 | 60923.98605324074 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT08 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234653_T00290_m725_1x1_100.0s_0001.fits | 39.840043 | 180.091172 | -10.0 | 2750 | m725 | LIGHT | 100.0 | 2025-09-05T23:46:53.000 | 2025-09-05 19:46:53.000 | 2460924.4908912037 | 60923.990891203706 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT08 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234508_T00290_m725_1x1_100.0s_0000.fits | 39.840357 | 179.998559 | -9.8 | 2750 | m725 | LIGHT | 100.0 | 2025-09-05T23:45:08.000 | 2025-09-05 19:45:08.000 | 2460924.489675926 | 60923.98967592593 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT08 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234838_T00290_m725_1x1_100.0s_0002.fits | 39.83912 | 180.183768 | -10.0 | 2750 | m725 | LIGHT | 100.0 | 2025-09-05T23:48:38.000 | 2025-09-05 19:48:38.000 | 2460924.4921064815 | 60923.992106481484 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT08 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_234140_T00290_m700_1x1_100.0s_0002.fits | 39.839148 | 179.815416 | -10.1 | 2750 | m700 | LIGHT | 100.0 | 2025-09-05T23:41:40.000 | 2025-09-05 19:41:40.000 | 2460924.4872685187 | 60923.98726851852 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT08 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT08/2025-09-05_gain2750/7DT08_20250905_233809_T00290_m700_1x1_100.0s_0000.fits | 39.83547 | 179.629828 | -9.8 | 2750 | m700 | LIGHT | 100.0 | 2025-09-05T23:38:09.000 | 2025-09-05 19:38:09.000 | 2460924.484826389 | 60923.984826388885 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT08 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_233808_T00290_m450_1x1_100.0s_0000.fits | 39.820706 | 179.644698 | -10.1 | 2750 | m450 | LIGHT | 100.0 | 2025-09-05T23:38:08.000 | 2025-09-05 19:38:08.000 | 2460924.4848148148 | 60923.984814814816 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT03 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_234653_T00290_m475_1x1_100.0s_0001.fits | 39.82478 | 180.107488 | -10.0 | 2750 | m475 | LIGHT | 100.0 | 2025-09-05T23:46:53.000 | 2025-09-05 19:46:53.000 | 2460924.4908912037 | 60923.990891203706 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT03 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT03/2025-09-05_gain2750/7DT03_20250905_234508_T00290_m475_1x1_100.0s_0000.fits | 39.825207 | 180.014533 | -10.0 | 2750 | m475 | LIGHT | 100.0 | 2025-09-05T23:45:08.000 | 2025-09-05 19:45:08.000 | 2460924.489675926 | 60923.98967592593 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT03 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234202_T00290_m600_1x1_100.0s_0002.fits | 39.823601 | 179.853116 | -10.0 | 2750 | m600 | LIGHT | 100.0 | 2025-09-05T23:42:02.000 | 2025-09-05 19:42:02.000 | 2460924.4875231483 | 60923.98752314815 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT14 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234928_T00290_m500_1x1_100.0s_0002.fits | 39.822232 | 180.246067 | -10.1 | 2750 | m500 | LIGHT | 100.0 | 2025-09-05T23:49:28.000 | 2025-09-05 19:49:28.000 | 2460924.4926851853 | 60923.992685185185 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT14 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_233825_T00290_m600_1x1_100.0s_0000.fits | 39.820335 | 179.662379 | -9.9 | 2750 | m600 | LIGHT | 100.0 | 2025-09-05T23:38:25.000 | 2025-09-05 19:38:25.000 | 2460924.485011574 | 60923.98501157408 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT14 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT14/2025-09-05_gain2750/7DT14_20250905_234739_T00290_m500_1x1_100.0s_0001.fits | 39.823586 | 180.150701 | -9.9 | 2750 | m500 | LIGHT | 100.0 | 2025-09-05T23:47:39.000 | 2025-09-05 19:47:39.000 | 2460924.491423611 | 60923.991423611114 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT14 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234002_T00290_m625_1x1_100.0s_0001.fits | 39.82206 | 179.744821 | -9.9 | 2750 | m625 | LIGHT | 100.0 | 2025-09-05T23:40:02.000 | 2025-09-05 19:40:02.000 | 2460924.486134259 | 60923.986134259256 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT11 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_233817_T00290_m625_1x1_100.0s_0000.fits | 39.820108 | 179.652273 | -10.1 | 2750 | m625 | LIGHT | 100.0 | 2025-09-05T23:38:17.000 | 2025-09-05 19:38:17.000 | 2460924.4849189813 | 60923.984918981485 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT11 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234659_T00290_m425_1x1_100.0s_0001.fits | 39.823944 | 180.11177 | -10.1 | 2750 | m425 | LIGHT | 100.0 | 2025-09-05T23:46:59.000 | 2025-09-05 19:46:59.000 | 2460924.4909606483 | 60923.990960648145 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT11 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234147_T00290_m625_1x1_100.0s_0002.fits | 39.823463 | 179.837513 | -10.1 | 2750 | m625 | LIGHT | 100.0 | 2025-09-05T23:41:47.000 | 2025-09-05 19:41:47.000 | 2460924.487349537 | 60923.987349537034 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT11 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234843_T00290_m425_1x1_100.0s_0002.fits | 39.822916 | 180.20439 | -9.9 | 2750 | m425 | LIGHT | 100.0 | 2025-09-05T23:48:43.000 | 2025-09-05 19:48:43.000 | 2460924.492164352 | 60923.992164351854 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT11 |
| /home/hhchoi1022/ezphot/data/obsdata/7DT/7DT11/2025-09-05_gain2750/7DT11_20250905_234513_T00290_m425_1x1_100.0s_0000.fits | 39.824364 | 180.019297 | -9.9 | 2750 | m425 | LIGHT | 100.0 | 2025-09-05T23:45:13.000 | 2025-09-05 19:45:13.000 | 2460924.489733796 | 60923.9897337963 | 1 | T00290 | 9473692bf1874a48a7492f0f1ad94624 | Spec | specall | None | 7DT11 |
2. Preprocessingο
2.1. Load master framesο
Master frames are combined calibration images that provide the best signal-to-noise ratio for correction. We need three types:
Master Bias: Electronic bias pattern (zero exposure time)
Master Dark: Thermal noise pattern (same exposure time as science images)
Master Flat: Pixel sensitivity variations (illumination pattern)
β οΈ Important: Master frames should be taken close in time to your science observations (typically within a few days) to ensure similar detector conditions.
[ ]:
from ezphot.methods import Preprocess
preprocess = Preprocess()
preprocess.help()
Help for Preprocess
Public methods:
- correct_bd(target_img: ezphot.imageobjects.scienceimage.ScienceImage, bias_image: ezphot.imageobjects.calibrationimage.CalibrationImage, dark_image: ezphot.imageobjects.calibrationimage.CalibrationImage, save: bool = False, verbose: bool = True, **kwargs)
- correct_bdf(target_img: ezphot.imageobjects.scienceimage.ScienceImage, bias_image: ezphot.imageobjects.calibrationimage.CalibrationImage, dark_image: ezphot.imageobjects.calibrationimage.CalibrationImage, flat_image: ezphot.imageobjects.calibrationimage.CalibrationImage, save: bool = False, verbose: bool = True, **kwargs)
- correct_bias(target_img: ezphot.imageobjects.scienceimage.ScienceImage, bias_image: ezphot.imageobjects.calibrationimage.CalibrationImage, save: bool = False, verbose: bool = True, **kwargs)
- correct_dark(target_img: ezphot.imageobjects.scienceimage.ScienceImage, dark_image: ezphot.imageobjects.calibrationimage.CalibrationImage, save: bool = False, verbose: bool = True, **kwargs)
- correct_flat(target_img: ezphot.imageobjects.scienceimage.ScienceImage, flat_image: ezphot.imageobjects.calibrationimage.CalibrationImage, save: bool = False, verbose: bool = True, **kwargs)
- generate_masterframe(calib_imagelist: List[ezphot.imageobjects.calibrationimage.CalibrationImage], mbias: ezphot.imageobjects.calibrationimage.CalibrationImage = None, mdark: ezphot.imageobjects.calibrationimage.CalibrationImage = None, combine_type: str = 'median', n_proc: int = 4, clip_type: str = 'extrema', sigma: float = 3.0, nlow: int = 1, nhigh: int = 1, verbose: bool = True, save: bool = True, **kwargs)
- get_masterframe(observatory: str = '7DT', telkey: str = '7DT_C361K_HIGH_1x1', telname: str = '7DT02', imagetyp: str = 'BIAS', exptime: float = None, filter_: str = None, obsdate: str = None, max_days: float = 5, verbose: bool = True)
- get_masterframe_from_image(target_img: ezphot.imageobjects.scienceimage.ScienceImage, imagetyp: str = 'BIAS', max_days: float = 5)
[ ]:
mbias = preprocess.get_masterframe_from_image(target_imglist[0], 'BIAS', max_days = 60)[0]
mdark = preprocess.get_masterframe_from_image(target_imglist[0], 'DARK', max_days = 60)[0]
mflat = preprocess.get_masterframe_from_image(target_imglist[0], 'FLAT', max_days = 60)[0] # For flat frames, master frame is needed to specify by the filter
mbias.show()
mdark.show()
mflat.show()
(<Figure size 800x600 with 2 Axes>,
<Axes: title={'center': '20250723-nm700.fits'}, xlabel='X pixel', ylabel='Y pixel'>)
2.2. Apply Correctionο
Now weβll apply the master calibration frames to correct our science images. The preprocessing follows this sequence:
Bias correction:
corrected = (raw - bias)Dark correction:
corrected = (bias_corrected - dark)Flat correction:
corrected = dark_corrected / flat
This removes systematic errors and normalizes pixel responses across the detector.
[ ]:
target_img = target_imglist[0]
corrected_img = preprocess.correct_bdf(
target_img = target_img,
bias_image = mbias,
dark_image = mdark,
flat_image = mflat,
save = False
)
[26]:
target_img.show()
corrected_img.show()
[26]:
(<Figure size 800x600 with 2 Axes>,
<Axes: title={'center': '7DT08_20250905_233955_T00290_m700_1x1_100.0s_0001.fits'}, xlabel='X pixel', ylabel='Y pixel'>)
[ ]:
# Save the corrected image. (You can also save the image with save = True)
corrected_img.write()
Saved: /home/hhchoi1022/ezphot/data/scidata/7DT/7DT_C361K_HIGH_1x1/T00290/7DT08/m700/7DT08_20250905_233955_T00290_m700_1x1_100.0s_0001.fits
2.3. Batch Process All Target Imagesο
For production workflows, youβll want to process all your target images automatically. This section demonstrates how to:
Loop through all target images
Automatically find appropriate master frames for each image
Apply corrections and save the results
π‘ Tip: The preprocessing methods automatically handle different exposure times and filters by finding the most appropriate master frames for each science image.
[ ]:
for target_img in target_imglist:
mbias = preprocess.get_masterframe_from_image(target_img, 'BIAS', max_days = 60)[0]
mdark = preprocess.get_masterframe_from_image(target_img, 'DARK', max_days = 60)[0]
mflat = preprocess.get_masterframe_from_image(target_img, 'FLAT', max_days = 60)[0]
preprocess.correct_bdf(
target_img = target_img,
bias_image = mbias,
dark_image = mdark,
flat_image = mflat,
save = True)
# You can also run in multiprocessing
# Run for all target_imglist with multiprocessing
# from multiprocessing import Pool
# from tqdm import tqdm
# def preprocessing(target_img):
# mbias = preprocess.get_masterframe_from_image(target_img, 'BIAS', max_days = 60)[0]
# mdark = preprocess.get_masterframe_from_image(target_img, 'DARK', max_days = 60)[0]
# mflat = preprocess.get_masterframe_from_image(target_img, 'FLAT', max_days = 60)[0]
# preprocess.correct_bdf(
# target_img = target_img,
# bias_image = mbias,
# dark_image = mdark,
# flat_image = mflat,
# save = True
# )
# # for target_img in tqdm(target_imglist, desc="Photometric Calibration..."):
# # photometriccalibration(target_img)
# with Pool(processes = 16) as pool:
# # Use tqdm with imap to show progress
# list(tqdm(pool.imap(preprocessing, target_imglist),
# total=len(target_imglist),
# desc="Preprocessing..."))