Like an Olympic athlete training for games still three years away, Cathy Anderegg is focused. But she's not eyeing a gold medal. Rather, she'll log thousands of work hours with the goal of meeting the impending science standards and assessment requirements of the No Child Left Behind act that go into effect in 2005 and 2007.
As assessment director for Alaska, Anderegg recently helped guide the state as it scrapped its old science standards and adopted the national standards. Now Alaska must hash out grade level expectations and then borrow, beg or create three science assessments to be given annually beginning in 2007.
"It's kind of become my mantra, 'I need an assessment!' " says Anderegg, who knows creating a test from scratch, even when working with a testing company, takes 18 months to two years.
NCLB calls for states to establish science standards for each grade level by 2005; staff highly qualified teachers by the end of the 2005-6 school year; and to test students once a year in each of three grade spans: 3-5, 6-9 and 10-12 in 2007-8.
The Education Commission of the States, in its NCLB report card, has a map that rates each state on its progress toward meeting the science requirements. ECS bases its ratings on whether the state laws, board of education rules or directives have been approved relating to each issue. Orange means the state is off track, white means partially on track, and purple means the state appears to be on track. When it comes to science standards, no states are orange, three are white and the rest are purple. But in science assessments, five states and the District of Columbia are orange and seven others are white.
Some educators disagree with ECS's assessment of their state's progress. Maine receives an orange rating from ECS in both standards and assessments, but state assessment director Brud Maxcy says Maine actually has little work to do because its been giving science assessments for several years. "I would guess we're one of the few states that's really ready since we've been doing it," Maxcy says.
But even in states that ECS has deemed 'purple,' assessment directors aren't resting on their ratings. The reality is most state departments of education still have a long way to go before they meet the NCLB requirements of high quality science assessments aligned with challenging state standards.
Why Science is Different From Reading
Most states adopted science standards in the 1990s, and the national standards were approved in 1995-96. But many states, like Alaska, are choosing to revise their standards because they weren't written to serve as guides for science assessments. Hashing out science standards and assessments is in some ways much more difficult from similar tasks in reading and math, in part because teaching and learning science is fundamentally different from tackling reading and math, and because many of the existing standards and assessments are just plain bad.
The biggest challenge in teaching and assessing science knowledge is the way in which it fundamentally differs from reading and math, says Assistant Secretary of Education Susan Sclafani, quoting former astronaut George Nelson. Sclafani, counselor to Secretary of Education Rod Paige, made this observation in a recent presentation to the National Science Teachers Association.
"It's not a skill in the same way reading is, it's a conceptual framework that you have to help students understand," Sclafani says. "Because of that it's more difficult to teach as well as to learn."
Nelson goes even further, saying that most science teaching is ineffective in K-14, in part because there are only about 1,000 hours of science instruction during a student's K-12 career. On top of this, there isn't nearly as much research available about how kids learn science, when compared with the body of research relating to reading and math. The National Research Council is hoping to remedy this discrepancy with a recently launched two-year study that will synthesize the research on the subject and prepare a white paper to guide curriculum developers and science teachers.
"We're going to look at the cognitive, developmental and educational studies," says Jean Moon, director of the board on science education at the National Research Council and director of the study. "To date there's never been an attempt to put all of that together and come up with a series of recommendations."
Creating Challenging Standards
So, science is different and educators may not have a good understanding of how kids learn the 'big ideas' in science that administrators hope they'll grasp. That goes a long way in explaining why some experts say the science standards and assessments in use today are ineffective.
"There hasn't been much consideration up to this point of what makes a good science assessment," says Meryl Bertenthal, who's hoping to remedy this problem as study director for a National Science Foundation-funded study titled, Building from the Research: Test Design for K-12 Science Achievement. "Science assessments focus too much on discrete facts; because it's easy to measure content, people tend to disconnect it from context."
Another assessment expert, who examined science assessments given by 40 states, goes even further in his criticism of the state of science education and assessment.
"The problem with assessments now is most states do not challenge students to learn science deeply at any grade level," says Stan Metzenberg, a biology professor at California State University and a member of the California state curriculum commission. "It is unusual to find test items that measure pre-existing knowledge or meaningful skills."
Metzenberg points out a litany of problems relating to science standards and assessments, including non-specific state standards, psychometric dilemmas and what he calls 'just-in-time teaching,' the phenomenon where tests give most of the knowledge needed to answer the question in the stem of the question.
In his paper Improving State Science Assessments, Metzenberg gives many examples of poor questions. Here's one from an eighth-grade science assessment given in Maine:
Standard: Students will understand how living things depend on one another and on non-living aspects of the environment. Students will be able to describe in general terms the chemical processes of photosynthesis and respiration.
Test item: Plants get the energy they need to live and grow from:
In this case, the standard to which the test item is linked seems to be challenging, but "the test item would be more suitable for a student in early elementary school. ... It is a superficial measure of performance against the standard," Metzenberg writes.
This question would most likely fail what Metzenberg and his colleagues on the California commission call the "L.A. Times test."
"We realize these test items will eventually be released to the public, and we ask ourselves, 'How will it look if it appears in the L.A. Times?'" he says. "What the public wants are really thoughtful questions that make students think."
Officials at the Federal Department of Education are aware of these issues, and recently invited state assessment directors, psychometricians, test vendors and others to get together to create a white paper on best practices in science assessment. Sclafani is urging states to use the NCLB mandates as an opportunity to reevaluate science standards and assessments.
"The creation of standards and assessments is one way to say, 'These are the things we think are important,' " Sclafani says. "We are asking states to rethink their standards, and is it possible to teach all [the material contained in the standards] in the time that is available."
Creating a Way to Test for Hands-on Learning
There are a variety of resources available to states seeking to improve science standards and assessments in preparation for the NCLB requirements. In addition to the white paper expected from the Department of Education this fall, the National Research Council is working on two new studies in this area. The Council of Chief States School Officers also has a committee working on the issue.
Here are some additional recommendations:
Many states have existing tests that aren't professionally written. Test item writers need to consider not only the standards being tested, but also to work with psychometricians to create valid and challenging questions;
Rather than using a single, large-scale assessment, NCLB allows states to incorporate multiple sources of data. In Nebraska each district creates its own assessments, often embracing a portfolio approach called the School-based, Teacher-led Assessment Reporting System;
Sclafani and the National Science Teachers Association advocate using assessments that rely on computer-based simulations of science experiments. This new approach has the advantages of a performance-based measurement that allows students to demonstrate what they know without the expense of traditional hands-on testing.
Educators from the Secretary of Education on down say inquiry-based learning is probably the best way to teach science, and it follows that hands-on experiments may also be the best way to assess students understanding of concepts and the scientific process. And while hands-on science testing can be done on a small scale, it's cost prohibitive for an entire state.
"You can do [hands-on testing] with a random selection of students, but you can't do that when you want to test every student. There are just too many kids," Sclafani says. "It's not cost effective to try to set up a performance-based measure. But by using simulation you can simulate those experiences and give students the chance to demonstrate what they know."
Aside from testing students on a deeper level of science understanding, computer-based simulations will generate zettabytes of data about how students solve problems, another boon to the science education community.
"From an assessment perspective, anything you do on a computer you can keep track of--how people solve problems, how many times they rotate an object, how they collect data," says David Kumar, professor of science education at Florida Atlantic University. "You can collect a lot of that kind of data, which would be useful psychometrics."
There are many pilots of simulation-style science assessments, and organizations like Educational Testing Service and SRI International are beginning to work on simulation-based assessments. But the applications are nowhere near ready for prime time, experts say, in part because most of the money for research in this area comes from the National Science Foundation rather than from industry.
"State-of-the-art simulation right now is still not at the enterprise level, it's not ready for large-scale implementations for making large-scale decisions about kids," says Edys Quellmalz, associate director of the center for technology in learning at SRI International. "Like anything else, it's a matter of resources."
Some science educators have a darker view of the reason why more simulations aren't available. "Paper-based testing has become a big, multi-billion-dollar industry," says Kumar. "The companies are not much open to entertaining new ideas; they're interested in multiple-choice tests."
But a lack of resources isn't stopping some states from moving toward computer-based testing, with an eye on simulation down the road. Indiana already conducts large-scale assessments online in 11th grade English and algebra. State educators have chosen to make an end-of-year biology exam mandatory to meet the NCLB science assessment requirements, and will begin piloting computer-based test items at the end of 2005.
"We're doing a small pilot this year that is exclusively online," says Wes Bruce, assistant superintendent for assessment, research and information technology. And before science assessments are mandatory, "We certainly are going to look at what might be available [in the way of simulations], or what it might cost to design something."
Like death and taxes, the science requirements of NCLB will be here sooner than most people would like. Or, maybe not. Some educators speculate that if President Bush isn't re-elected some of the more stringent demands of NCLB may be relaxed. That may be good news for many districts, but perhaps not science teachers.
"There's a lot of enthusiasm in the science community for their content area, way more than the other content areas," says Cathy Anderegg of Alaska. "They're glad they're getting assessments so they can show their stuff."
Rebecca Sausner is a freelance writer based in Brooklyn, N.Y.