Oklahoma Dept. of Human Services, DAB No. 1223 (1991)

Department of Health and Human Services

DEPARTMENTAL APPEALS BOARD

Appellate Division

SUBJECT: Oklahoma Dept. of Human Services
Docket No. 90-70
Decision No. 1223

DATE: February 4, 1991

DECISION

The Oklahoma Department of Human Services (State) appealed a penalty
disallowance imposed under section 403(h) of the Social Security Act by
the Office of Child Support Enforcement (OCSE). OCSE determined that
the State's child support enforcement and paternity establishment
program did not substantially comply with the requirements of the Act
and imposed a one percent reduction of the amount of money otherwise
payable to the State (estimated to be $1,007,498) for Aid to Families
with Dependent Children (AFDC) during the period July 1, 1988 through
June 30, 1989.

OCSE made its noncompliance determination based on two audits. A
program results audit of the State's child support enforcement program
for fiscal year 1985 concluded that the State was not in substantial
compliance with four Title IV-D service-related criteria, including the
criterion for withholding of unemployment compensation benefits (45
C.F.R. 305.39). State Exhibit (Ex.) A. A follow-up review conducted in
1989, which covered the period January 1, 1988 to December 31, 1988,
showed that, although the State had achieved substantial compliance with
three other criteria cited in the earlier audit, the State still had not
achieved substantial compliance with the criterion for withholding of
unemployment compensation. The penalty disallowance at issue therefore
hinges on the State's alleged failure to achieve compliance in a single
criterion, withholding of unemployment compensation, in both the program
results and follow-up audits.

On appeal, the State argued, among other things, that OCSE had
improperly applied the substantial compliance standard in its audits and
that improper procedures were used in the program review audit. Since
we find that .procedures were used in the program results audit that
make its findings unreliable, we reverse the disallowance.

Background

The Board recently provided an overview of the Title IV-D audit review
system in Ohio Dept. of Human Services, DAB No. 1202 (1990). The IV-D
regulations at 45 C.F.R. 305.20(b) provide that, with respect to fiscal
year 1985, procedures required by 13 audit criteria related to basic
program services "must be used in 75 percent of the cases reviewed for
each criterion . . . ." In both the program review and the follow-up
audits, OCSE found that the State had not applied appropriate procedures
in the "withholding of unemployment compensation" criterion in 75
percent of the cases where unemployment compensation might have been
available as a collection source.

During the program review audit of the State's performance of program
services, the auditors selected an initial "probe sample" of 500 cases
with potential for enforcement action. The sampled cases were then
examined according to the kinds of procedures that would have been
appropriate in each case, in order to determine whether each procedure
was being followed in at least 75 percent of the cases where that
procedure was appropriate. The auditors computed an efficiency rate to
reflect the percentage of cases in which an appropriate enforcement
action had been initiated for each criterion. St. Ex. A, pp. 12-22.
For some audit criteria, the probe sample did not yield a sufficient
number of cases to allow the estimate of an efficiency rate; in those
situations, the auditors used additional cases to expand the sample and
to estimate efficiency rates.

The auditors did not rely on the probe sample at all for the withholding
of unemployment compensation criterion. The auditors determined that
the actual number of cases where withholding of unemployment
compensation was necessary in the Title IV-D universe was so small that
it would be highly unlikely that any cases of that type would be
included in the sample. OCSE Ex. A, p. 3. The auditors initially
proposed to draw a separate sample solely for this criterion from a much
narrower list of cases, namely the list of all unemployment cases
referred to the State's Child Support Enforcement Division (henceforth
referred to as the "matched" list). The State had not retained this
list for the period in question, however, and it therefore was not
available for this purpose. 1/ The auditors consequently based their
assessment of this criterion on an ad-hoc review of 20 cases taken from
a file of worksheets prepared from cases active during the audit period.
Id. at 20.

In eight of these 20 cases, or 40 percent, the auditors found that
proper action had been taken. The auditors then concluded:

Based on our limited sample results, we estimate that the
"Withholding of Unemployment Compensation" efficiency rate is
between 19 and 61 percent which is below the 75 percent standard
required for substantial compliance with 45 CFR 305.39. Id.
(Emphasis supplied)


Analysis

The State argued on appeal that the sample used in connection with the
program results audit findings on withholding of unemployment
compensation cannot be used to make statistically defensible judgments
about Oklahoma's compliance with this audit criterion during the initial
audit period. State Supp. Resp., pp. 1-2. We agree, and on this basis
reverse the disallowance. 2/

The issue raised by this appeal is a rather elementary statistical
sampling issue -- whether an ad-hoc sample of the sort relied upon here
provides a statistically reliable assessment of the State's program-wide
performance for a particular audit criterion. OCSE has unquestionably
committed itself to using statistically reliable sampling methodologies.
See, e.g., the preamble to the final regulations implementing the Child
Support Enforcement Amendments of 1984, 50 Fed. Reg. 40128, Comment 3
(Oct. 1, 1985). For purposes of the program results audit here,
however, the auditors did not take a .random sample from the universe of
cases requiring services under the withholding of unemployment
compensation criterion. The auditors merely relied upon 20 cases
requiring services under the criterion that happened to be readily
available. We have no idea of the relationship between these 20 cases
and the universe of cases for this criterion for the period covered by
the audit. Since the sample was indisputably not a random sample
designed to be representational of the universe of cases in the State's
program for this criterion, we cannot make statistically defensible
judgments about the State's performance for the universe of cases based
on the sampling results.

This very point was substantiated by OCSE's own expert. After
describing various useful features of certain types of judgment samples,
he ultimately acknowledged that the results from the "ad-hoc sample"
used in the program results audit may not be used to represent the
universe for the withholding of unemployment compensation criterion. He
stated as follows:

In the foregoing presentation on the role of judgment sampling in
statistical inference, I continually referred to their selection
from a known universe. I would be willing to defend my conclusion
that Oklahoma failed to meet the Federal substantial compliance
standard for the audit criterion "Withholding Unemployment
Compensation" based on my analysis, if the ad-hoc sample were
selected from a known universe. However, this was not the case.

The judgment sample of 20 cases was selected from a readily
available file of worksheets of unemployment compensation cases.
Therefore, the resulting 40 percent efficiency rate can only
represent the sample and not the universe of these cases for the
audit period (October 1, 1984 through September 30, 1985).

Mandel Declaration, pp. 9-10.

The OCSE response brief makes a similar concession when it concedes the
"inability to project [the] results to the universe." OCSE Br., p. 36.

OCSE tried to overcome the lack of reliability of the results by arguing
that the sample selection interjected a bias favoring the State. See
Response Br., p. 35 and .Bassett Declaration, OCSE Ex. A, p. 5. OCSE
asserted that the cases reviewed were selected from worksheets
representing cases on which the State was supposed to have initiated
some unemployment compensation withholding action, and that the sampling
approach used by OCSE therefore was advantageous to the State. Even if
the worksheets contained cases generally more advantageous to the State,
we could not accept the sample results here. There is still the
possibility of a bias against the State introduced by the way the sample
was drawn from the worksheet files. Moreover, OCSE has conceded that it
is unable to recalculate the efficiency rate, standard error and
efficiency range using the two-stage evaluation technique used by OCSE
in several cases that have reached the Board (see, e.g., Ohio, supra),
because the sample here was judgmental and because OCSE was unable to
determine the sampling frame for the State's unemployment compensation
withholding cases. Bassett Declaration, OCSE Ex. A, p. 6.

Thus, based on OCSE's own concessions, we are unable to rely on the
ad-hoc sample as a basis for concluding that the State has failed to
substantially comply with this criterion on a program-wide basis for the
period covered by the program results audit.

OCSE implied at several points in its submissions that other factors,
when considered in conjunction with the ad-hoc sample results, supported
its conclusion that the State failed to substantially comply with the
criterion at issue. OCSE implied that it was the State's fault that the
auditors did not have better records to make their assessment and that
the State should have retained the matched lists that the auditors had
wanted to use. OCSE also argued that a finding of noncompliance was
justified because State officials had acknowledged in the course of the
program results audit that they were not pursuing legal means to
withhold unemployment compensation. Finally, OCSE argued that the
State's poor performance in the follow-up review supports the conclusion
that it failed the program results audit.

We find that none of these factors would be sufficient to overcome the
lack of a statistically reliable sample for the withholding of
unemployment compensation criterion.

At the outset, OCSE pointed to no authority requiring a state to retain
the specific matched list OCSE requested here (as opposed to actual case
records, which the State had retained) or requiring that the State must
fail the .criterion at issue for failure to retain the matched list.
OCSE did not dispute that the State destroyed the list well before
receiving the request from the auditors for the list. Moreover, the
State asserted that it provided a matched list, as requested, for the
follow-up review because it had prior notice of the need for the list.
Further yet, the program results audit for FY 1985 was the first audit
where the withholding of unemployment compensation criterion was subject
to assessment, and, therefore, neither party had had prior experience in
documenting performance under that criterion. The difficulty here
apparently arose because the auditors had concluded that few, if any, of
the cases selected for the probe sample would have required this service
to be taken, and thus the probe sample was viewed by the auditors as an
ineffective vehicle for measuring the State's performance under this
criterion. See Bassett Declaration, OCSE Ex. A, p. 3. The State,
however, can hardly be held responsible for OCSE's inability or
unwillingness to use or supplement the probe sample for this criterion.

Also, we find no basis in the record to conclude that the State had
failed to meet the 75 percent standard based on an acknowledgment made
by unnamed State officials during the course of the 1985 audit. The
audit report indicated that "State officials acknowledged that the legal
process for withholding unemployment benefits was not pursued" because
garnishment was not cost effective. (State Ex. A, p. 20) This
acknowledgment was made in the context of the program results findings,
and as such, explains specifically why the State had not pursued the
legal process in particular cases in the ad-hoc sample relied upon by
OCSE. It obviously could not have been an admission that the State
never followed the proper procedures for the criterion as a whole, since
the State passed 8 of the 20 cases reviewed. In any event, it clearly
was not an admission by the State that it had failed to meet the 75
percent standard for this criterion for the period covered by the audit,
and cannot be relied upon as such.

Finally, the regulations (45 C.F.R. 305.20(b)(2)) specify that the
program results audit at issue here must be based on the State's
performance for FY 1985. Contrary to what OCSE has implied, there is no
basis in the regulations for using evidence of the State's performance
during the follow-up review period (covering January 1 through December
31, 1988) to reinforce unreliable evidence relating to the period
covered by the program .results audit. The program results audit
findings must be based on evidence related to the State's performance
for the FY 1985 period alone.

Conclusion

For the reasons discussed above, we reverse the disallowance.

_____________________________
Judith A. Ballard


_____________________________
Alexander G. Teitz


_____________________________
Donald F. Garrett Presiding
Board Member

1. For the follow-up review, the State was able to furnish a list of
all absent parents who received unemployment benefits during the
relevant period, and the audit assessment of performance for this
criterion was based on a review of cases on that list.

2. Our findings in this respect make it unnecessary for us to discuss
the State's other arguments, or indeed, other issues that might have
required further development, such as the specific action required of
the State under the criterion at