Validation / intercomparison of daily satellite precipitation
estimates -- An IPWG project
1. Why validate satellite rainfall estimates?
There has been a great deal of research in the last two decades on
methods
for estimating rainfall from infrared (IR) and microwave satellite
observations.
As a result there are now several satellite
algorithms running operationally and semi-operationally from
national
centers and universities to produce rainfall estimates for time periods
ranging from half-hourly to monthly. The great advantage of space-based
precipitation estimates is their global coverage, providing information
on rainfall frequency and intensity in regions that are inaccessible to
other observing systems such as rain gauges and radar. The disadvantage
is that they are indirect estimates of rainfall, depending on the
properties
of the cloud top (in the case of IR algorithms) and cloud liquid and
ice
content (in the case of passive microwave algorithms). Active radar
observations
from the TRMM Precipitation Radar
provide the most accurate high resolution satellite-based rainfall
estimates
to date, but the sampling characteristics of this instrument are quite
limited.
Rainfall products from the operational algorithms are easily
obtainable
via the web or FTP, and are being used for many diverse meteorological,
climate, hydrological, agricultural, and other applications. It is
therefore important to have an idea of their accuracy and expected
error
characteristics.
This is done by validating the satellite precipitation estimates
against "ground truth" from rain gauge and/or radar observations. A
thorough
verification of satellite-based precipitation products should
quantify their accuracy in a wide range of weather and climate regimes,
give users information on the expected errors in the estimates, help
algorithm developers understand the strengths and weaknesses of the
satellite
rainfall algorithms, including which aspects are in greatest need of
improvement, monitor the performance of existing algorithms, and assist
with
evaluating algorithm upgrades. To get good estimates of absolute
accuracy
satellite products are verified against very high quality radar and
gauge data.
However, these sites are only
few in number. To get estimates of regional and spatial accuracy it is
necessary to use a much larger quantity of data, for example, from
national rain gauge networks. While these verification data are less
reliable
than those from high-quality sites, their errors are usually much
smaller
than those associated with the satellite estimates, at least on short
time
scales.
In 2003 the
International Precipitation Working Group
(IPWG) began a project to validate and intercompare operational and
semi-operational
satellite rainfall estimates over Australia and the US in near real
time.
A European verification was added in 2004, and other regions may be
included in the future. This study focuses on the large-scale
validation of daily rainfall estimates, for two reasons. First, the
large number of rainfall observations from rain gauges at the 24-hour
time scale provides good quality verification data on a large scale.
Second, daily rainfall estimates are required as input to a large
number of climate and other applications. For comparison, 1-day
forecasts from a limited number numerical weather prediction models,
namely the ECMWF, the US (NCEP), and US Navy global models, and the
Australian regional model, are also verified.
This site describes the IPWG algorithm validation / intercomparison
project and presents some of the results obtained thus far.
2. Reference data
a. Australia
Australia is unique among developed nations in having extensive
tropical, subtropical, and mid-latitude climate regimes that are well observed by
a high quality national rain gauge network. The Australian rain gauge
network consists of up to 6000 sites that measure 24-hour accumulated rainfall
at approximately 9 a.m. local time (approximately 00 UTC). Between 1000
and 1500 of these sites report daily rainfall in near-real time,
providing data for the Australian Bureau of Meteorology's operational daily
rainfall analysis. The remaining observations are made primarily by cooperative
observers and are reported at the end of each month. These additional
data are entered into the Bureau's climate database and combined with the
original gauge data to produce a more accurate reanalysis of daily rainfall. The
objective analysis scheme uses a multi-pass inverse distance-weighting
scheme to map the rainfall observations onto a 0.25-degree grid over
Australia
(Weymouth et al., 1999).
b. United States
The Climate Prediction Center (CPC) daily (1200 - 1200 U.T.C.) rain
gauge analysis (Higgins et al., 2000)
which contains rain gauge information from over 7000 stations across
the U.S. each day is used
as the reference data set for the U.S. validation activities. The
gauge data are quality controlled by removing duplicates, ensuring that
clouds were present for observations of non-zero precipitation amounts,
and "buddy checks" are performed. The data are then objectively
analyzed to a 0.25-degree latitude/longitude grid (Cressman, 1959).
Because the data have been objectively analyzed, the spatial coverage
of very light intensity observations is inflated and the intensity of
intense rainfall events is damped. This is an artifact of all
objective analysis techniques but is particularly inherent in the
Cressman scheme. It is for this reason that the spatial coverage
statistics use a threshold of 1 mm d-1 instead of zero.
c. Western Europe
The validation data for western Europe is obtained via the UK
Meteorological Office (UKMO) from a
network of radars across the United Kingdom, France, Germany, Belgium,
and the Netherlands. The UKMO gathers the data from
participating nations and combines the data into a single pan-European
product on a (equal area) polar-stereographic projection. The data is collected
at a nominal 5 km / 15 minute resolution and is subsequently compiled into
daily accumulations at the University of Birmingham. At present no additional
quality control is
performed on the data, although future comparisons / recomparisons will account
for range dependency, data availability etc.
d. South America
The CPC Realtime Analysis is prepared using GTS daily reports and additional
reports provided by CPTEC, INMET, FUNCEME and SIMEPAR of Brazil and INAMHI of Ecuador.
Its spatial resolution is 1 degree, and the 24 hour period extends from 12 UTC
to 12 UTC. As with the U.S. analysis, quality control is performed prior to analysis
using a modified Cressman scheme. Grid boxes with no gauge observations are
not used in the verification.
e. Japan
The Radar-AMeDAS data (Makihara 1996)
is produced by the Japan Meteorological Agency, and provided
by Japan Weather Association in accordance with joint study between Japan Aerospace
Exploration Agency (JAXA) and Kyoto University.
We acknowledge that the gauge and radar rainfall analyses contain
error related to imperfect sampling, measurement
error, and artifacts of the objective analysis scheme. This error in
the
"truth" data causes the calculated errors for the satellite
precipitation
estimates to be greater than their true values. In general, we expect
the errors
in the analyses to be much smaller than those of the satellite
estimates,
so the verification results can be considered as valid. The inflation
of
the calculated algorithm error due to errors in the "truth" data is a
topic
of research (e.g., Krajewski
et
al., 2000).
3. Operational and semi-operational satellite rainfall products
Several global operational and semi-operational satellite rainfall
estimates
are routinely verified on daily and longer time scales.
Estimates provided on sub-daily time scales are summed for a 24-hour
period (00-24 UTC for Australia and western Europe, 12-12 UTC for
the US) to correspond to the verification data. Details of the
algorithms and estimates can be found on their respective web sites.
They include:
GSFC
TRMM-based 1- and 3-hourly, 0.25-degree resolution (3B40RT, 3B41RT,
3B42RT, MPA)
(Huffman et al., 2002)
Convective Stratiform Technique (CST)
TRMM-calibrated IR, daily, 0.25-degree resolution (Negri et al., 2002)
NRL hourly,
0.10-degree
resolution geostationary-microwave blend and microwave-only merge (Turk et al., 2002)
NESDIS merged AMSU-B estimates (Weng et al., 2003)
NESDIS
Hydro-Estimator, hourly, 0.25-degree resolution (Vicente et al., 1998)
NOAA
CPC morphed IR/microwave, half-hourly,
0.25-degree resolution, also merged microwave (Joyce
et al., 2004)
GOES
Multispectral Rainfall Algorithm (GMSRA) (Ba and
Gruber, 2001)
U.
Birmingham,
0.50-degree resolution merged microwave (Kidd
et al., 2003)
U.
Birmingham, 0.25-degree resolution SSM/I from 85-19 GHz frequency
difference algorithm
U.
Birmingham, 0.25-degree resolution GPI (Arkin and Meisner, 1987)
U. California Irvine,
PERSIANN 6-hourly, 0.25-degree resolution (Sorooshian
et al., 2000)
STAR Microwave Integrated Retrieval System (MIRS) and
Microwave Surface and Precipitation Products System (MSPPS)
JAXA Global Satellite
Mapping of Precipitation (GSMaP) near real time product
(Aonashi
et al. 2009)
GPCP 1-degree daily (Huffman et al., 2001)
All products except for the GPCP 1-degree daily product
are available within 2 days
of the satellite observations, and often within a few hours. Scientists
wishing to obtain data used in this validation study may wish to visit
the IPWG
Precipitation Validation / Intercomparsion Study Data Archive.
4. Verification methodology
The study is primarily concerned with the daily verification of
satellite spatial rainfall fields. This is accomplished using maps,
time series, and statistics. The map
display allows the satellite
rainfall estimates to be easily viewed and evaluated on a case by case
basis, which
is essential for understanding the characteristics of the satellite
algorithms.
The quantitative scores measure the accuracy of the spatial estimates,
and are archived for summary and monitoring purposes. Each statistic
gives only one piece of information about the error, and so it is
necessary
to examine a number of statistics in combination in order to get a more
complete picture. The statistics used in the IPWG validation /
intercomparison project are:
Estimated and observed rain area
Estimated and observed rain volume
Estimated and observed conditional rain rate
Estimated and observed maximum rain rate
Mean absolute error
Root mean square (RMS) error
Correlation coefficient
|
Hits, misses, false alarms, correct rejections (2x2
contingency table)
Bias score (ratio of estimated/observed rain frequency or area)
Probability of detection
False alarm ratio
Hanssen and Kuipers score
Equitable threat score
Heidke skill score
|
For descriptions of these scores please refer to the Forecast
Verification - Issues, Methods and FAQ page (WWRP/WGNE, 2005) or the textbook of Wilks (1995). Time series plots
allow the temporal variation in algorithm
performance to be easily evaluated and intercompared on daily, monthly,
and seasonal time scales. Other diagnostic plots enable various aspects
of algorithm performance to be evaluated.
Download the
validation code
5. Results
Daily spatial validation results are currently generated and updated
daily for rainfall estimates over Australia, the United States,
western Europe, and South America. These results can be viewed at
http://cawcr.gov.au/projects/SatRainVal/sat_val_aus.html
(Australia)
http://www.cpc.ncep.noaa.gov/products/janowiak/us_web.shtml
(United States)
http://meso-a.gsfc.nasa.gov/ipwg/ipwgeu_home.html
(Western Europe)
http://cics.umd.edu/~dvila/web/SatRainVal/dailyval.html
(South America)
http://www-ipwg.kugi.kyoto-u.ac.jp/IPWG/dailyval.html
A more detailed analysis and synthesis of these results is ongoing. Some preliminary results were
presented at the 2nd
IPWG Workshop in Monterey in October 2004:
Monitoring
the quality of operational and semi-operational satellite precipitation
estimates: the IPWG validation / intercomparison study (E.
Ebert)
- extended abstract
talk
Validation of satellite-derived rainfall estimates and numerical
model forecasts
of precipitation over the US (J. Janowiak) - extended abstract talk
Validation of satellite rainfall estimates over the mid-latitudes
(C. Kidd) - extended
abstract talk
In early 2007 a paper on the IPWG validation results was published in the
Bulletin of the American
Meteorological Society - click here
to download the paper. Some of
the most important findings are:
1. Merging passive microwave and IR estimates provides more accurate
estimates of precipitation than do the separate components
2. The satellite algorithms performed best in the warm season
(convective rainfall), and worst during winter
3. NWP forecasts generally outperformed the blended satellite estimates
during the winter season over all regions examined
4. The satellite algorithms tended to underestimate the amount of light
rainfall but overestimate the amount of heavy rainfall
5. Two major systematic biases were apparent in the satellite estimates
over the United States:
Over-estimation
over snow-covered regions
Over-estimation
in semi-arid regions during the warm season
6. Over the Australian tropics the microwave, IR, and microwave-IR
schemes showed similarly good performance, and outperformed NWP models
for heavy rain
Additional regions for which validation results will soon be available
include South Africa and India. All of the regional studies
will contribute their results to the Program for the
Evaluation of High Resolution Precipitation Products (PEHRPP).
6. Caveats
1. The IPWG validation results apply only to satellite precipitation
estimates over land. In so far as the rain
processes and also many of the rain algorithms are different over water
than over land, the error measures derived over land cannot be assumed
to apply equally well to ocean-based estimates.
2. The project is only validating daily rainfall estimates, not
those at shorter time scales. This is because there are many 24-hour
gauge observations
from which to produce national scale rainfall analyses. The observation
error of the 24-hour data (gauge or radar) is also
smaller, in a relative sense, than it would be for shorter accumulation
periods.
3. The spatial scales of the estimates are typically ~25 km. Since
validation
results tend to improve with spatial averaging of the estimates, the
quantitative results reported here would differ for finer or
coarser scales.
4. The reported errors in the satellite precipitation estimates are
larger than their true values because observation error in the
reference data contributes to the total error. We don't have any good
way to know the true magnitudes of the satellite errors given perfect
observations. RMS errors would be a bit smaller, correlations a bit
higher, and mean biases probably not noticeably different. The
categorical statistics like POD, FAR, ETS would probably not
change a great deal since they are less sensitive to the magnitude of
the rainfall.
7. References
Aonashi, K., J. Awaka, M. Hirose, T. Kozu, T. Kubota, G. Liu, S. Shige,
S. Kida, S. Seto, N. Takahashi, and Y. N. Takayabu, 2009: GSMaP passive
microwave precipitation retrieval algorithm: Algorithm description and
validation. J. Meteor. Soc. Japan, in press.
Arkin, P.A. and B.N. Meisner,
1987: The
relationship between large-scale convective rainfall and cold cloud
over the
western hemisphere during 1982-84. Mon. Wea. Rev., 115,
51-74.
Ba, M.B. and A. Gruber, 2001: GOES
Multispectral Rainfall Algorithm (GMSRA). J. Appl. Meteorol., 40,
1500-1514.
Cressman, G. F., 1959: An operational
objective analysis system. Mon. Wea. Rev., 87, 367-374.
Ebert, E.E., J.E. Janowiak, and C. Kidd, 2007:
Comparison of near real time precipitation estimates from satellite
observations and numerical models. Bull. Amer. Met. Soc., 88, 47-64.
Higgins, R.W., W. Shi, E. Yarosh and
R. Joyce, 2000: Improved United States Precipitation Quality Control
System and Analysis. NCEP/Climate Prediction Center ATLAS No. 7,
40 pp., Camp Springs, MD 20746, USA.
Huffman, G.J., R.F. Adler, M.M.
Morrissey,
D.T. Bolvin, S. Curtis, R. Joyce, B. McGavock, J. Susskind, 2001:
Global
precipitation at one-degree daily resolution from multisatellite
observations.
J. Hydromet., 2, 36-50.
Huffman, G.J., R.F. Adler, E.F.
Stocker,
D.T. Bolvin, and E.J. Nelkin, 2002: A TRMM-based system for real-time
quasi-global merged precipitation estimates. TRMM International
Science Conference, Honolulu, 22-26 July 2002.
Joyce, R.J., J.E. Janowiak, P.A. Arkin and
P. Xie, 2004: CMORPH: A method that produces global precipitation
estimates from passive
microwave and infrared data ad high spatial and temporal resolution.
J. Hydromet., 5, 487-503.
Kidd, C., D.R. Kniveton, M.C. Todd, and T.J.
Bellerby, 2003: Satellite rainfall estimation using combined passive
microwave and infrared algorithms. J. Hydrometeor.,
4, 1088-1104.
Krajewski, W.F., G.J. Ciach, J.R.
McCollum, and C. Bacotiu, 2000: Initial verification of the Global
Precipitation
Climatology Project monthly rainfall over the United States. J.
Appl. Meteor., 39, 1071-1086.
Makihara, Y., 1996: A method for improving
radar estimates of precipitation by comparing data from radars and
raingauges. J. Met. Soc. Japan, 74, 459-480.
Negri, A.J., L. Xu, and R.F. Adler,
2002: A TRMM-calibrated infrared rainfall
algorithm applied over Brazil. J. Geophys. Res., 107 (D20),
8048-8062.
Sorooshian, S., K.-L. Hsu, X. Gao,
H.V.
Gupta, B. Imam, and D. Braithwaite, 2000: Evaluation of PERSIANN system
satellite-based estimates of tropical rainfall.
Bull. Amer. Met. Soc., 81, 2035-2046.
Tapiador, F.J., 2002: A new algorithm to
generate global rainfall rates from satellite infrared imagery. Spanish
J. Remote Sensing, in press. Click here
to get a HTML version.
Turk, J., E. Ebert, H.-J. Oh, B.-J.
Sohn,
V. Levizzani, E. Smith and R. Ferraro, 2002: Verification of an
operational
global precipitation analysis at short time scales. 1st Intl.
Precipitation
Working Group (IPWG) Workshop, Madrid, Spain, 23-27 September 2002.
Vicente, G.A., R.A. Scofield, and
W.P.
Menzel, 1998: The operational GOES infrared rainfall estimation
technique.
Bull. Amer. Met. Soc., 79, 1883-1898.
Weng, F.W., L. Zhao, R. Ferraro, G.
Pre, X. Li and N.C. Grody, Advanced
Microwave Sounding Unit (AMSU) cloud and precipitation algorithms, In
Press, Radio
Sci., 2003.
Weymouth, G., G.A. Mills, D.
Jones,
E.E. Ebert, and M.J. Manton, 1999: A continental-scale daily rainfall
analysis
system. Aust. Met. Mag., 48, 169-179.
Wilks, D.S., 1995: Statistical Methods
in the
Atmospheric Sciences. An Introduction. Academic Press, San Diego,
467 pp.
WWRP/WGNE Joint Working Group on
Verification, 2005: Forecast Verification - Issues, Methods and FAQ,
http://cawcr.gov.au/projects/verification/
Related links:
Program for the
Evaluation of High Resolution Precipitation Products (PEHRPP) -
Project to evaluate high resolution satellite-derived precipitation
products
NOAA/NESDIS
Satellite Rainfall Verification Page - 6-hourly and daily
verification
of several satellite precipitation estimates over the US
Climate Rainfall Data
Center at CSU - Information on and comparison of monthly rainfall
estimates from
satellite and surface observations
LDAS
Observed Precipitation Validation - Comparison of satellite derived
precipitation products with Higgins gauge analysis and NEXRAD radar
analyses
International Precipitation
Working Group (IPWG) - Operational and research satellite- based
quantitative precipitation measurement - issues, challenges, and
algorithms
NASA
TRMM Online Visualization and Analysis System (TOVAS) - comparison
of TMPA-RT, radar, and gauge precipitation over the US during 2005
Environmental Verification and
Analysis
Center (EVAC) - Satellite precipitation verification at the GPCP
Surface
Reference Data Center at OU
Beth Ebert
Centre for Australian Weather and Climate Research (CAWCR), Bureau of Meteorology, Melbourne, Australia
Last updated October 2011
Please send all comments and questions to e.ebert@bom.gov.au
