Comments on the Community Survey

NCT.com story: Encinitas survey criticized; results show most respondents satisfied with services

Encinitas City Council Meeting, August 22, 2007

Comments on the Community Survey

Jean-Bernard Minster

The survey conducted by Moore Information, Inc. and reported by staff appears to be a very simple kind of survey. Nonetheless, the following points need to be made:

1. The raw data do not appear to be available. These data have been paid for and belong to the citizens of Encinitas. They should be made available publicly.
2. The methodology is not described in the very short four-page analysis offered by the contractor ($2.5K /page). There is a minimal description in the staff report that raises very serious questions about the sampling procedure.
3. It appears that the survey was conducted by taking Encinitas telephone prefixes and random 4-digit sequences. The large number of non-working numbers supports that.
4. Each working phone number reaches a household. Clearly, wealthier households with multiple phone lines have a greater chance of being sampled.
5. In order to achieve a representative sample of the population, a general practice is to select at random a member of the household, either from a table of random numbers, or through some proxy variable such as the earliest or latest birthday (Gallup uses that at times)
6. However, there have been reports that the interviewers have specifically requested adults in the age bracket 18-34 “in order to meet the quota.” This is not a valid criterion to generate a representative sample. In fact, this is a sure recipe for a biased sample. This action also reveals that the contractor was fully aware of the sampling flaws in this survey.
7. One likely reason why the sample is impoverished in persons with ages between 18 and 34 is that this age group is more likely to use wireless “cell” phones instead of so-called “ land lines.” The report attempted to disguise this flaw by “lumping” all younger responders in a 15-year age range (18-34) while differentiating older responders into much narrower 5-year ranges (e.g. 55-59, and 60-64), so it appears that all classes are more-or-less equally represented. However, as a “sanity” check, we can compare the percentages found in the sample with those obtained from the City of Encinitas / SANDAG web pages:

For instance, among the eligible responders of ages greater than 18, the SANDAG data indicate that 46% should be of age 18-45. ]The survey sample has only 28%. The SANDAG data show 54% of valid responders to be older than 45, and the survey has 72%. It is very clear that the sampling is severely biased towards older citizens and against younger ones.

8. This bias carries over to the sampling of citizens as a function of income. After correction to 2007 dollars, the SANDAG database shows 79.3% of households with income below $100K, and 20.7% with income greater than $100K. The survey sample has 38% or responding households with less than $100K income and 39% with greater income, and 22% of no-response. Any reasonable assumption made about the motivations of non-respondents leaves us with a sample that is substantially biased towards higher-income households.
9. Finally, the staff report indicates sampling error “of ±6% at the 95% confidence level”. In other words, out of 100 identical surveys, 95 would show results within ±6% of the value reported. This is clearly a large sample approximation —a rule of thumb would be a sample larger than 60— which does not apply to many of the results reported. As a simple example, consider the response to question 23. From the description, one would assume that the fraction of Hispanic origins in the sample is 3%, plus or minus 6%, or anywhere between minus 3% and plus 9%, which is clearly nonsensical. Considering that 3% means that 9 respondents —somewhat fewer than 60— out of 300 identified themselves as Hispanics, the correct answer is that in 100 unbiased surveys, 95 would show a percentage of Hispanics between 1% and 6%.

Now, considering that the 2000 Census shows a percentage of 16% of Hispanic residents in Encinitas, should we conclude that nearly 8,000 Hispanic residents have left our city between 2000 and 2007? Even after correcting for the fact that the Hispanic population is heavily weighted toward youngsters, it is practically impossible to reconcile the survey with the Census data or the SANDAG data.

10. Finally, 5 out of 6 households called in the survey declined to answer. This is a very important consideration that deserves careful analysis and discussion. Such analysis is not provided, either in the staff report or in the superficial contractor report.

Given my concerns, and the concerns expressed by others at this meeting, I recommend that the Council should decline to receive the survey report in its present form, and decline to pay for it until the raw data are provided and the methodology is suitably described. The citizens of Encinitas are not served well by this flawed survey conducted under a no-bid contract.

Respectfully:

Jean Bernard Minster

Encinitas

Coast News article about the survey click here.

Leucadia Blog: Killing 2 birds with 3 stones