JVER Article

The "Significance" of Research in Vocational Education
1992 AVERA Presidential Address
by Gary E. Moore¹

Journal of Vocational Education Research
Vol. 17, No. 4, 1992

Research is coming under increasing scrutiny. It is critical that research in vocational education be meaningful and of value. The first step in accomplishing this is to select problems to research that are truly problems that need to be researched. We must not become obsessed with the idea that statistical significance makes research significant. In order for vocational education research to become more significant, we must insist that graduate student research be more than an academic exercise and must not let the ease of conducting a particular study and the article potential of the problem be the primary determinant in selecting the research problem. We also need to broaden our research horizon to include qualitative research and we need to spend more time in constructing and reporting conclusions and the implications of our research. By doing these things, research in vocational education will become more significant.

Historically, vocational education at the secondary and postsecondary levels has suffered from a "second-class citizen" image. This image has carried over into higher education. Departments of vocational teacher education at the university level have not always been held in the highest esteem. Whether merited or not, this stigma has been attached to research in vocational education. To put it bluntly, research conducted in vocational education at the university often has been viewed as less than first-rate. To paraphrase Rodney Dangerfield, vocational education research can't "get no respect."

It is imperative, now more than ever, that we, the researchers in vocational education, take a long hard look at the research we are doing.

Today, university research, as a whole, is under scrutiny. Clifton Conrad, past president of the Association for the Study of Higher Education, wrote in the Review of Higher Education (1989) that research in higher education was "lifeless and pedestrian, inward-looking and parochial, the product of assembly-line research that has generated few new findings or challenging ideas" (p. 206). The question we must ask ourselves is "Does Clifton's comment apply to research in vocational education?"

In a recent column in a North Carolina newspaper, a journalist (Jenkins, 1992) wrote, "Once a professor at a North Carolina public university-and most everywhere else-publishes some papers, writes a book about the chemical composition of Botswanan goat cheese and kisses up to the dean for awhile, he or she is on what they call the 'tenure track', likely to be knighted after a few years" (p. 12A). Jenkins' cynicism regarding academic research is very evident, and not uncommon. The question we must as ourselves is "Does Jenkins' comment apply to researchers in vocational education?"

A perception of the general public is that university faculty members teach one or two classes a semester, have a few office hours, and spend the rest of the time on frivolous, arcane research activities. At least four states have conducted performance audits of university faculty during the past year to see what professors do. After one such audit in North Carolina, the auditor still wasn't sure what professors did. The question we must ask ourselves is "What are vocational education researchers doing?"

University governing boards are starting to look seriously at the research coming from within the university. After Paul Ferguson, a University of North Carolina professor who had won three teaching awards, was denied tenure because his "creative research"-the adaptation of literature to the stage-did not meet the "traditional articles and published works" standards of his department, the University Board of Governors decided to investigate. Over the objections of the University Chancellor who maintained that only people within the university had the expertise to evaluate the quality of Ferguson's research, the Board's Personnel and Tenure Committee decided to look into the issue anyway (Wilson & Parsons, 1992).

Budget problems have many universities eliminating programs. In addition to looking at program duplication within the state, legislators and higher education officials are questioning the quantity and quality of faculty research in making decisions. To put it succinctly, the research enterprise is under stress. At the university level, we no longer can go about business as usual, especially in regard to research. We must critically examine our research. Is the research being conducted in vocational education of value? Is it meaningful? Or, to put is simply, is our research significant?

The Significance of Research in Vocational Education

The word significance in The American Heritage Dictionary is defined as 'The state or quality of being significant," which leads us to look at the definition of significant. The word significant has three definitions. They are:

1. Having or expressing a covert meaning; suggestive: a significant glance.

2. Having or expressing a meaning; meaningful.

3. Notable; valuable.

Does research in vocational education meet any of the definitions given for the word significant? My position is that we, as vocational education researchers, overemphasize the first definition (suggestive) and don't pay enough attention to the other two definitions (meaningful and valuable).

The "Suggestive" Definition of the Word Significant

The American Heritage Dictionary uses "a significant glance" as an example of how the word significant can mean "Having or expressing a covert meaning." In vocational education research, statistics are used to give research a covert or implied meaning. How often have you seen a researcher change his or her tone of voice and declare proudly that this or that finding was "statistically significant"? It's as if having statistically significant findings makes the research good. Bracey (1988) reminded us that "Statistical significance has nothing to do with meaningfulness" (p. 257). What does statistical significance really mean? Kupfersmid (1988) observed that "A . . . problem related to the meaningfulness of 'statistically significant' findings is that what is statistically significant and what is 'significant' in a meaningful sense may be contradictory" (p. 636). Tests of statistical significance are overused and misused in an attempt to make a poor or mediocre study appear good.

Kupfersmid (1988) proposed that research be submitted to editors without the results section because editors are known to favor papers with statistically significant results. Wouldn't that be an interesting phenomenon? Reviewers would be forced to place more emphasis on the significance of the problem being studied instead of the statistical significance of the findings.

A respected professor of vocational education at one of the Big Ten universities never published his dissertation. It wasn't because he didn't try. His quasiexperimental study comparing contract grading with conventional grading was well done but achieved no statistically significant findings. When the research was submitted for publication, the reviewers rejected it because his findings weren't significant.

The research done by Peters at Purdue comparing reflective teaching with microteaching hasn't yielded many statistically significant results. What the nonstatistically significant findings really mean is that reflective teaching is comparable to microteaching, a fact that may be valuable to know as budget dollars shrink, since reflective teaching is inexpensive to implement.

Two recent doctoral dissertations at North Carolina State were virtually void of statistical significance, but this fact was, in effect, highly significant. One study (Monegain, 1992) compared the competencies needed by trainers in Japan and America. Even though we believe the two cultures to be very different, the research found competencies needed by trainers in Japan to be very similar to those needed by their counterparts in America. The second study ( Baker, 1992) found there were virtually no differences between teaching methods that promote critical thinking skills in baccalaureate nursing programs and associate degree nursing programs. While nurse educators in certain programs like to think they are dramatically different, the research didn't support this view.

Why do educational researchers place such emphasis on statistical significance? Soltis (1984) provided a clue:

Much of the social and behavioral sciences have developed in this century into their present forms by consciously seeking to imitate the methods and forms of the natural sciences, many educational researchers have tried to travel the same royal road to knowledge, legitimacy and status. (p. 6)

Shaver (1992) maintained that educational researchers insist on tests of statistical significance because they "provide a facade of scientism in research. For many in educational research, being quantitative is equated with being scientific ... despite the fact that some scientists and many psychologists ... have managed very well without inferential statistics" (p. 2).

One of the problems with canonizing tests of statistical significance in inferential statistics is that one basic assumption for their use is often violated: that random sampling be used-or at least random assignment to groups. This is often not the case in educational research.

The research methodology and statistical treatment used in vocational education research has improved greatly over the past 2 decades (Warmbrod, 1986). We, as vocational education researchers, should be proud of our improving prowess as "research technicians." I am not advocating we do away with statistical testing. But I am cautioning that we must not get caught up in the misguided belief that having statistically significant findings makes our research significant.

The "Meaningful" and "Valuable" Definition of the Word Significant

For vocational education research to be truly significant, the problems we study and the resulting recommendations and implications must be both "meaningful and valuable." (These are the American Heritage Dictionary second and third definitions of the word significant.) The logical place to start in making sure vocational research is meaningful and valuable is in the selection of the problems we research. If we select trivial, insignificant problems to research, then it is exceedingly difficult for our research to be of any value or significance. Key (1982) reminded vocational educators that the first basic in research ". . . is the consideration and thorough definition of the problem which gave the rise to the need for the research in the first place" (p. 1).

Are vocational education researchers doing a good job in selecting research problems? I'm not sure, but my inclination is to say we could be doing better. In the past I have criticized graduate research in agricultural education for being a day late and a dollar short (Moore, 1987). However, as I look at the professional literature in vocational education, I find virtually no comment about the topics we are researching. But I have found several articles that looked critically at the statistical procedures and research methodologies used. Other fields of scholarly inquiry appear to be more critical of the topics being researched.

Layzell (1990) contended that the problems being researched by higher education researchers are not significant (meaningful or valuable) to the field of higher education. In a recent Chronicle of Higher Education essay titled "Most Research on Higher Education Is Stale, Irrelevant, and of Little Use to Policy Makers" Layzell, who is Assistant Director for Fiscal Affairs at the Illinois Board of Higher Education, asserted that "Policy makers in higher education, including governors, legislators, members of state higher education boards, college trustees, and college administrators, have held the academic treatment of higher education in relative contempt for some time" (p. B3).

In psychology, Kupfersmid (1988) maintained that "Much of the published research focuses on irrelevant issues" (p. 641). Conrad (1989) contended that "Most research by faculty in higher education is oriented to scholarly peers rather than other stakeholders" (p. 200). Weiner (1986) accused higher education faculty of conducting "small research on small issues" (p. 160). After reviewing numerous research studies in education Eisner (1984) concluded: "I am skeptical about the current impact of educational research on educational practice" (p. 451). Could the same type of charges be made about vocational education research?

How Are Research Problems Selected in Vocational Education?

In vocational education, problems to research often are selected for a variety of reasons-sometimes the wrong reasons. I recently attended a graduate committee meeting for the express purpose of helping the doctoral student select a research topic. Instead of the student's saying that there were two or three problems in the field she would like to study, she said that she wanted to use the MBTI in her dissertation, period. She had identified a tool she wanted to use but hadn't identified any problems to research. At times I suspect vocational researchers do the same thing. They became enamored with a statistical tool such as canonical correlations or log linear analysis and go in search of something on which to use the tool. I have attended research reporting sessions where I came away with the distinct impression that the researcher was more concerned with trying to dazzle and impress the audience with his or her research and statistical acumen than in trying to analyze real problems facing vocational education. Hamlin (1966) believed that "Another current obsession is with elaborate statistical analysis used as much to obscure as to illuminate" (p. 15).

A second criterion that vocational education researchers may use in selecting problems to research is the "quick, easy, and publishable" criterion. On more than one occasion I have heard a colleague say something to the effect that "this study would be quick and easy to do and we could get an article out of it." It is not too difficult to put together some type of survey instrument, have a few colleagues "validate" it, collect the data, send it through the computer, then write it up. For some of the more prolific writers in our profession, the process is nearly automatic. This is the assembly line research Conrad (1989) referred to.

Much of the revered experimental research aptly fits the "quick" category. Eisner (1984) reviewed the experimental studies reported in the American Educational Research Journal in 1981 and found the median treatment time was 72 minutes. He observed, "We conduct educational commando raids to get the data and get out" .(p. 451). Yet 72 minutes was almost a 60% increase over the average amount of treatment time received by students in studies reported in the 1978 issue of AERJ. Eisner concluded, "How can we hope to achieve educationally significant results when the models of inquiry we employ virtually preclude achieving them?" (p. 451).

A third criterion sometimes used for selecting problems to research is the article potential of the research. How many different journal articles can be generated from one study? Vocational educational researchers have been known to do research studies because the study has the potential to generate three, four, or more journal articles.

Not all of the problems in vocational education are amenable to the "quick and easy" research methodology nor will they generate volumes of articles. Many of the important issues in vocational education will require considerable time and effort to research. Some of the important research issues have to do with integration of academic and vocational education, tech prep, and the long-term impact of increasing high school graduation requirements. It could take years to properly research some of the questions arising from these educational practices. The university system, specifically in the field of education, may need to change its expectations of research production and reporting (promotion and tenure) in order for long-term, significant vocational research to become possible and valued.

Barriers to Conducting Significant Research

There are at least two barriers in vocational education that tend to prevent us from conducting research that is truly significant. Both of these barriers are mental and both have to do with the way we think about research.

The first barrier is the idea that the process of research is the most crucial component of research. We tend to place far greater emphasis on the research process than on the product. In a typical journal article, about two thirds of the article, if not more, is devoted to the process. Even in the rating scales we use to evaluate research manuscripts, the great majority of the weight is assigned to procedure. The significance of the problem is one small item. Perhaps we should change our rating scales. Before we evaluate the methodology used, let's first determine if the problem has " above average" significance to the profession. It it doesn't, the review process stops; if it does, the process continues.

The idea that process is the most important aspect of research starts in graduate school. How many of us have made statements such as "It's only a master's thesis"; "Don't try to save the world with your dissertation"; or "What's important is that you learn the research process." Everything we say to graduate students emphasizes that process is the important end result of research. There is no law or rule that says graduate student research has to be this way. We should encourage graduate students to select problems that are of significance.

During my graduate days I ran across a mini essay titled "Little Piles of Dust." The author and source were unknown. The little essay speaks volumes about graduate student research.

Little Piles of Dust

Every year about a thousand young men and women go off into justly neglected corners of knowledge and assemble tiny scraps of more or less useless information into a little pile of dust, which, adorned with comparative tables, correlative graphs, and other forms of academic parsley, is served up as a thesis. The reward is the title Doctor of Philosophy, which enables its recipient to ascend the educational ladder and in time, teach other young men and women how to scrape together their own heaps of dust or doctoral dissertations, upon such themes as "The Measurement of Typical Pessimistic Attitudes Among the Ducks of Lake Cayuga Region," or "The Use of the Personal Pronoun in the Posthumous Works of Miss Atlantic City of 1936."

Another example of how we over-emphasize the importance of the research process is the numerous statistics courses and courses in research methods we require our budding young researchers to take. Such courses generally spend relatively little time discussing the importance of selecting a significant problem, and spend a "significant" amount of time emphasizing the process of various research designs.

In writing dissertations, the least amount of time is typically spent on the final chapter that contains the conclusions, discussion, and implications. Time and time again, students find frustration and confusion in writing their final chapter, often with life-altering deadlines pushing them. Yet, it is in the final chapter where valuable, new knowledge could emerge. The final chapter needs to be changing and guiding the direction of vocational education, now and in the future. Our treatment of the final chapter reinforces the concept that process is more important than the product.

The emphasis on technique instead of problem reminds me of The Saber Tooth Curriculum by Harold Benjamin (1939). In this book, the children of the cave dwellers went to school where they were taught fish grabbing, tiger scaring, and horse clubbing. These were important skills for the survival of the tribe, and they became very specialized. As time went on, however, changes in climate and an advancing glacier caused the three subjects to become obsolete (streams were now muddy so they couldn't grab fish, tigers had been replaced by bears, and horses were replaced by antelopes). Yet, the elders of the tribe resisted the call to change the curriculum. They were more concerned with teaching the techniques of the past than with solving the problems at hand.

Are we as vocational researchers more concerned with technique than with problems? Back in 1966, Hamlin, the chairman of the Department of Vocational and Technical Education at the University of Illinois observed:

It is frustrating to try to get anyone recently trained in research methods in the universities or in research seminars to attempt some of the most needed kinds of research. Their instinct is for the narrow, insignificant problem which lends itself to research and statistical treatments which have recently become hallowed. Are we training researchers or research technicians? (p. 15)

One of AERA's past presidents, Bob Warmbrod, in a speech to the Southern Agricultural Education Research Conference in 1986, suggested that researchers, in reporting research, should consider making a brief statement that the methodology and statistics were done correctly, and then spend the majority of the time discussing the importance and implications of the research. This idea has merit. What would happen if we turned our journal articles and research reports around and started with our conclusions and recommendations, then described the methodology that leads us to those conclusions and recommendations'?

The second barrier to improving the significance of research in vocational education is a mind set that the only good research is quantitative research. We tend to believe that it isn't quality research unless there is some number-crunching going on. Recently, one of our bright, new qualitative researchers sent me an article to read. It was an excellent article! Sadly though, it had to be published outside our field because our professional journals wouldn't publish it' Some of the major issues that need to be addressed in vocational education are more amenable to qualitative inquiry. After reviewing the problems and research in the field of higher education, Conrad (1989) posited a number of recommendations for improving the research, including a call to ". . . broaden our paradigm for inquiry" (p. 211). If vocational education research is to become more meaningful and valuable, a variety of research methodologies, including qualitative inquiries, also will be required.

Conclusions and Recommendations

Research in vocational education is currently not as significant as it could be. We place too much emphasis on statistical significance and not enough emphasis on the practical or applied significance of the research. We need to pay more attention to selecting problems for study. We should be conducting research on issues that are of true importance to the profession. While doing studies that are quick, easy, and can be turned into journal articles is nice, it should not be the primary criterion for selecting research problems.

In the preparation of future vocational education researchers, more emphasis should be placed on helping them select significant problems for study. The writing of a thesis or dissertation should be more than an academic exercise. Selecting a significant problem and arriving at conclusions that result in recommendations for improving practice are just as important, if not more so, than the process of conducting research.

In our own research reporting we need to spend more time and effort on implications and interpreting what the research really means. Often by the time we get through writing about the methodology and statistically significant findings, we have either run out of steam or space to do much with conclusions and recommendations. Oh, we do put a few things down but don't really spend a lot of time or effort on them. We know we have to write something. It is a sad commentary that we can generate many more "Suggestions for Further Research" than we can generate "Suggestions for Practice." Our research will really start becoming significant when we can generate more suggestions for practice and implications from the research.

We need to remember why we do research in the first place-to find answers to problems or to discover the truth. By emphasizing this focus, vocational education research can become more significant, that is, meaningful and valuable. It must.

References

Baker, P.C. (1992). Perceptions of nursing faculty concerning factors that promote critical thinking skills in nursing students. Unpublished doctoral dissertation, North Carolina State University, Raleigh.

Benjamin, H. (1939). Saber tooth curriculum. New York: McGraw-Hill.

Bracey, G.W. (1988). Tips for readers of research. Phi Delta Kappan, 70(3), 257-258.

Conrad, C.F. (1989). Meditations on the ideology of inquiry in higher education: Exposition, critique and conjecture. Review of Higher Education, 12 (3), 199-220.

Eisner, E.W. (1984). Can educational research inform educational practice? Phi Delta Kappan, 65 (7), 451-458.

Hamlin, H. M. (1966). What is research? American Vocational Journal, 41(6), 14-16.

Jenkins, J. (1992, November 26). Tenure track a smooth road for the gravy train. The News and Observer, 12A.

Key, J. P. (1982, July). Back to basics in educational research. Paper presented at the meeting of the Southern Research Conference in Agricultural Education, Athens, GA.

Kupfersmid, J. (1988). Improving what is published. American Psychologist, 43(8), 635-642.

Layzell, D. T. (1990, October 24). Most higher education research is stale, irrelevant, and of little use to policy makers. The Chronicle of Higher Education.

Monegain, A. 1. (1992). Perceptions of Japanese training and development professionals in Japan toward the job roles and competencies identified by American training and development professionals in the United States. Unpublished doctoral dissertation, North Carolina State University, Raleigh.

Moore, G. E. (1987, December). A day late and a dollar short. Doctoral research in agricultural education. Paper presented to the National Agricultural Education Research Meeting, Las Vegas, NV.

Shaver, J. (1992, April). What significance testing is, and what it isn't. Paper presented at the meeting of the American Educational Research Association, San Francisco, CA.

Soltis, J. F. (1984). On the nature of educational research. Educational Research, 13(10), 5-10.

Warmbrod, J. F. (1986, March). Priorities for continuing progress in research in agricultural education. Paper presented at the meeting of the Southern Regional Research Conference in Agricultural Education, Little Rock, AR.

Wiener, S. S. (1986). Shipyards in the desert. Review of Higher Education, 10, 159-164.

Wilson, T., & Parson, G. (1992, November 23). Bitter debate over tenure renewed. The News and Observer, 1A.

¹Gary Moore, 1992 American Vocational Education Research Association President is Professor in the Department of Occupational Education, North Carolina State University, Raleigh, NC.