Ancillary Data Sets Used for ADL and CSPP Product Generation ------------------------------------------------------------ Kathleen Strabala kathy.strabala@ssec.wisc.edu Space Science and Engineering Center University of Wisconsin-Madison 13 March 2015 The entire set of leading edge and archived ancillary data sets used by the CSPP and some ADL software is available from this web site: https://jpssdb.ssec.wisc.edu/cspp_v_2_0/ These files are selected and downloaded automatically if required by any of the JPSS CSPP SDR or EDR algorithms. Descriptions of files in daily directories follows: ------------------------------------------------------------------------------- Name: Daily Two Line Element Files (TLE Files) Blob and Ascii Files Example File Names: 4fe9f2c6-a2a28-6880dc6d-f929ca0b.TLE-AUX_npp_20120626140339Z_20120626000000Z_ee00000000000000Z_-_c3s-_ops_all-_ops 4fe9f2c6-a2a28-6880dc6d-f929ca0b.asc Required for VIIRS, ATMS and CrIS SDR creation File Description: The Blob file is a binary file that contains two lines of information about the Suomi-NPP Keplerian orbital elements. The accompanying .asc file contains metadata including the Effectivity date range of the TLE binary file. These files are available once per day. These files are converted from the .h5 files that are distributed through NOAA CLASS - http://www.class.ngdc.noaa.gov/saa/products/welcome Example of HDF5 filename: TLE-AUX_npp_20150309142745Z_20150309000000Z_ee00000000000000Z_-_c3s-_ops_all-_ops.h ------------------------------------------------------------------------------- Name: Weekly Polar Wander Blob and Ascii Files Example File Names: 4fe4cc2e-a2c1f-6880dc6d-8988aaea.off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120622_201206221200Z_20120622000005Z_ee20120629120000Z_np.bin_Internal 4fe4cc2e-a2c1f-6880dc6d-8988aaea.asc Required for VIIRS, ATMS and CrIS SDR creation File Description: The Polar Wander file describes the relationship of changes in the spin axis of the earth to the hemispherical asymmetry. The Blob file is a binary data file and the .asc file contains the metadata describing the binary file including the Effectivity date range. These files are available once per week (Friday). These files are converted from the .h5 files that are distributed through NOAA CLASS - http://www.class.ngdc.noaa.gov/saa/products/welcome Example of HDF5 filename: off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20150313_201503130000Z_20150313002503Z_ee20150320120000Z_np.h5 ------------------------------------------------------------------------------- Name: The Near Real-Time SSM/I EASE-Grid Daily Global Ice Concentration and Snow Extent (NISE) Example File Name: NISE_SSMISF17_20120613.HDFEOS Required for VIIRS EDR Products File Structure Description: Each file contains NISE snow/ice extent on 721x721 northern and southern hemisphere azimuthal grids. The file is in HDF-EOS format. Record/Line Format: Snow/ice extent for the northern and southern hemispheres stored in separate HDF-EOS Grid objects. Product Description: The Near Real-Time SSM/I EASE-Grid Daily Global Ice Concentration and Snow Extent (NISE) product is generated by the National Snow and Ice Data Center (http://nsidc.org). This data set was originally designed to provide NASA EOS researchers with near real-time, daily, global snow extent and sea ice concentration data. The MODIS Instrument Team uses the NISE data as input to their products. Data access is unrestricted but users are strongly encouraged to register with NSIDC as users of the data. Registered users automatically receive notification of changes in the product-generation algorithm and of modifications in data manipulation and formatting. Registered users automatically receive E-mail notification of any data production problems. NSIDC requests that prospective data users explain how the data will be used and agree to contribute comments regarding the usefulness, relevance, and accuracy of the data. Such feedback is essential to ensure ongoing product improvement. A link to register for the data and read more about the product can be found at: http://nsidc.org/data/nise1.html ------------------------------------------------------------------------------ Name: 6 hourly Global Data Assimilation System T126 resolution analysis from NCEP at 1 degree resolution Example File Name: gdas1.pgrb00.1p0deg.20120613_00_000.grib2 gdas1.pgrb00.1p0deg.YYYYMMDD_HH_000.grib2 Where: YYYY is the year, MM is the month, DD is the day, HH is the model run time (00,06,12 or 18) Required for VIIRS EDR Products For more information about the models and files, please see: http://www.nco.ncep.noaa.gov/pmb/products/gfs/#GDAS These files are created 4 times per day and staged at these sites: ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.YYYYMMDD http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gdas.YYYYMMDD These are the files that are used in anything other than real-time processing. In real time, the data will be processed using the GFS forecast model files, since these analysis files are not available until 6-9 hours after the model run time. File naming conventions at the above sites: eg: gdas1.t06z.pgrb2.1p00.f000 File Type: Binary GRIB2 File Content Description: 6 hourly Global Data Assimilation System T126 resolution analysis from NCEP. File Structure Description: Each file contains fields of z, u, v, temp, vert vel, rh, and absolute vorticity, most of which are on 26 mandatory pressure levels (1000 - 10mb). vv is on 1000-100mb and rh is on 1000-300mb. There are additional single level fields as well, for example, boundary level winds, temps, and rh, trop temp, pressure and winds. The grid is a 1 x 1 deg global grid (GRIB grid #3). ------------------------------------------------------------------------------ Name: 3 hourly Global Forecast System (GFS) Numerical Weather Prediction Fields for NPP at half degree resolution Example File Name: gfs.press_gr.0p5deg_pt.20120613_00_018.npoess.grib2 gfs.press.gr.0p5deg_pt.YYYYMMDD_HH_0XX.npoess.grib2 Where: YYYY is the year, MM is the month, DD is the day, HH is the model run time (00,06,12 or 18), XX is the model run time step (3 hourly, 03-24) Required for VIIRS EDR processing In real time data processing, these GFS model files will be used since the GDAS analysis files are not available until 6-9 hours after the model run time. For more information about these files, please see: JPSS Common Interfaces and Services Interface Control Document Volume 1 JPSS CDFCB - External - Volume VI - Ancillary Data, Auxiliary Data, Messages, and Reports both can be found at: http://npp.gsfc.nasa.gov/documents.html These files are grabbed from this public ftp site: ftp://tgftp.nws.noaa.gov/ An example of the directory and file naming structure is: ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.gfs_CY.12/RD.20150313/PT.grid_DF.gr2/fh.0003_tl.press_gr.0p5deg_pt.npoess ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.gfs_CY.HH/RD.YYYYMMDD/PT.grid_DF.gr2/fh.00XX_tl.press_gr.0p5deg_pt.npoess Where: YYYY is the year, MM is the month, DD is the day, HH is the model run time (00,06,12 or 18), XX is the model run time step (3 hourly, 03-24) These files are also distributed through this site: https://msds.npoess.noaa.gov/MSDS/ANC/ODAD/ with file names like: off_NCEP-GFS-03HR-ANC_GFS_NCEP_003f_20120607_201206071800Z_20120607213027Z_ee20120608000000Z_np.grib2 ---------------------------------------------------------------------------- Name: Daily Global Modeled Sea Ice Concentration from NCEP at 0.5 degree resolution Example File Name: eng.YYYYMMDD Where: YYYY is the year, MM is the month, DD is the day, Required for VIIRS Sea Surface Temperature EDR Product For more information about the models and files, please see: http://polar.ncep.noaa.gov/seaice/index.html These files are created once a per day and staged at these sites: ftp://polar.ncep.noaa.gov/pub/cdas/ During a month, each day's GRIB file will be placed here. After the end of the month, the entire month's data will be concatenated to a single file and put in the archive subdirectory ftp://polar.ncep.noaa.gov/pub/cdas/archive/ File naming conventions at the above sites: eg: eng.YYYYMMDD File Type: Binary GRIB1 File Content Description: Daily Global Modeled Sea Ice Concentration from NCEP at 0.5 degree resolution File Structure Description: Each file contains a single grib message... 1:Ice cover (1=land, 0=sea):(0 - 1) (instant):regular_ll:meanSea:level 0:fcst time 0 :from 201303020000 Values are stored as fractional concentration -- 0.00 to 1.00 for ice cover. In the analyst's grids, values of up to 1.28 are possible, and seem to relate to physical processes. Most users will want to reset all values between 1.00 and 1.28 to 1.00. In the modeler's grid, this is already done. Flag values which may be encountered: 1.57 -- Land 1.95 -- Coast 1.66 -- Weather 2.24 -- No data The modeler's grid sets ice concentrations to zero over land currently. This has not always been the case, and the archive has not yet been corrected to reflect this current practice. Ice concentrations over 'coastal' points are and always have been given. In ideal circumstances, these values would not be used. ---------------------------------------------------------------------------- Name: 3 hourly Navy Aerosol Analysis and Prediction System (NAAPS) Fields for NPP at half degree resolution Example File Name: off_NAAPS-03HR-ANC_NAAPS_FNMOC_003f_20150306_201503061200Z_20150306151500Z_ee20150306180000Z_np.grib2 off_NAAPS-XXH-ANC_NAAPS_FNMOC_0XXf_YYYYMMDD_YYYYMMDDHH00Z*np.grib2 Where: YYYY is the year, MM is the month, DD is the day, HH is the model run time (00,06,12 or 18), XX is the model run time step (3 hourly, 03-24) Required for VIIRS Aerosol Product EDR processing These files contain grid (half degree - 720x361) Total Column Aerosol Optical Thickness (AOT) at 550 nm For more information about these files, please see: JPSS Common Interfaces and Services Interface Control Document Volume 1 JPSS CDFCB - External - Volume VI - Ancillary Data, Auxiliary Data, Messages, and Reports both can be found at: http://npp.gsfc.nasa.gov/documents.html These files are grabbed from this public ftp site: https://msds.npoess.noaa.gov/MSDS/ANC/ODAD/ ---------------------------------------------------------------------------- Name: GHRSST Level 4 CMC0.2deg Global Foundation Sea Surface Temperature Analysis Produced by the Canadian Meteorological Center Example File Names: 20150312120000-CMC-L4_GHRSST-SSTfnd-CMC0.2deg-GLOB-v02.0-fv02.0.nc These files are issued once per day. Required for ASCPO Sea Surface Temperature software product generation. File Description: NetCDF file that contains a number of parameters including latitude, longitude and analyzed sea surface temperature. The grid is a 0.2 degree global grid (901 x 1800). These files are distributed through this website: ftp://podaac-ftp.jpl.nasa.gov/allData/ghrsst/data/GDS2/L4/GLOB/CMC/CMC0.2deg/v2/YYYY/jdy Where: YYYY is the year, and jdy is the 3 digit Julian day of the year (1-366) A Group for High Resolution Sea Surface Temperature (GHRSST) Level 4 sea surface temperature (SST) analysis produced daily on an operational basis at the Canadian Meteorological Center. This dataset merges infrared satellite SST at varying points in the time series from the (A)TSR series of radiometers from ERS-1, ERS-2 and Envisat, AVHRR from NOAA-16,17,18,19 and METOP-A, and microwave data from TMI, AMSR-E and Windsat in conjunction with in situ observations of SST from drifting buoys and ships from the ICOADS program. It uses the previous days analysis as the background field for the statistical interpolation used to assimilate the satellite and in situ observations. This dataset adheres to the GHRSST Data Processing Specification (GDS) version 2 format specifications. For more information about the files, please see this website: http://www.nodc.noaa.gov/SatelliteData/ghrsst/ ---------------------------------------------------------------------------- Name: NOAA Optimum Interpolation (OI) Sea Surface Temperature (SST) V2 Example File Names: avhrr-only-v2.20150310_preliminary.gz avhrr-only-v2.20150210.gz Required for the CSPP The Clouds from AVHRR Extended (CLAVR-x) Software Package File Description: Binary files on a 1.0 degree latitude x 1.0 degree longitude global grid (180x360). These files are issued once per day on a 100km grid Information on reading these binary files can be found here: ftp://eclipse.ncdc.noaa.gov/pub/OI-daily-v2/IEEE/README These files are distributed through this website: ftp://eclipse.ncdc.noaa.gov/pub/OI-daily-v2/IEEE/YYYY/AVHRR/ Where: YYYY is the year The preliminary files are updated when all information is available about 4-5 days after real-time. These files represent a weekly interpolated data set. ---------------------------------------------------------------------------- Name: 6 hourly Global Forecast System (GFS) Numerical Weather Prediction Fields Example File Name: gfs.tHH.051201.pgrbfXX (gfs.t06.150311.pgrbf12) Where: HH is the model run time (00,06,12 or 18), XX is the model run time step (3 hourly, 03-12) Used by CSPP Products: UW Dual Regression HSRTV Retrievals NOAA Unique CrIS/ATMS Processing System (NUCAPS) NOAA ASCPO Sea Surface Temperature Software The Clouds from AVHRR Extended (CLAVR-x) For more information about these models and files, please see: http://www.nco.ncep.noaa.gov/pmb/products/gfs File Type: Binary GRIB2 File Content Description: 6 hourly Global Data Assimilation System T126 resolution analysis from NCEP. 1 degree global resolution These files are created in real time and staged at these realtime sites: ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.YYYYMMDD http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/YYYYMMDD using this naming convention: gfs.tYYyz.pgrb2.1p00.XX Where: HH is the model run time (00,06,12 or 18), XX is the model run time step (3 hourly, 03-12) ---------------------------------------------------------------------------- VIIRS Instrument Look-Up_Tables Distributed from: https://jpssdb.ssec.wisc.edu/cspp_v_2_0/ancillary/LUTS_V_2_0/ Example File Names: 54fe37ce-db0ee-6880cc6e-9a80361a.VIIRS-SDR-F-PREDICTED-LUT_npp_20150301061500Z_20150305232300Z_ee00000000000000Z_PS-1-O-CCR-15-2306-JPSS-DPA-132-PE_noaa_all_all-_all.bin 54fe37ce-db0ee-6880cc6e-9a80361a.asc Required for VIIRS SDR Calibrated Data. File Description: The Blob files is a binary file that contains the calibration information. The accompanying .asc file contains metadata including the Effectivity date range. The files come in at various times, but there is usually at least one per week. The source of these files is the Land PEATE. ftp://lpeate1.nascom.nasa.gov/LUTs/Mx8/SDRS_GEO/IDPS/ This is a password protected ftp site.