Conference on Applied Statistics in Agriculture

ANALYSIS OF NUCLEI FLUORESCENCE HISTOGRAMS USING NON-LINEAR FUNCTIONS OR WAVELETS

Histograms based on 5,000 nuclei from cells (Chinese hamster ovary cells, bone marrow cells) are used to determine the coefficient of variation (CV) of observations surrounding the highest peak. The cells are subjected to various treatments, for example exposure to herbicides. By eyeballing the histogram, an interval under the highest peak is determined. The CV calculated from the histogram on the eyeballed interval is the response variable in an ANOVA. To avoid the subjectivity of eyeballing the histogram, non-linear functions such as the Gaussian density function can be used to model the histogram. The CV may then be determined from the parameter estimates. In many experiments nonlinear functions modeling the histograms smooth away differences in CV s obtained this way, though visually the histograms appear to be different. Then nonlinear functions or wavelets can be used to obtain intervals for calculating CV s of the histograms restricted to these intervals. The nonlinear models require close initial values for each histogram, while the wavelets just require choice of wavelet and level of decomposition.