<p>
<p>Independent Study of Washington State K-8 Curriculum Review October 2008 Prepared by: Curriculum Review 1 Table of Contents Introduction Executive Summary Charge to Strategic Teaching Review of OSPI Process Differences in Scoring Methods Overview of Scoring Approaches Review Bias Confidence Intervals Elementary School Findings Comparison of Content Match for Elementary Programs Mathematical Soundness in Elementary School Middle School Findings Comparison of Content Match for Middle School Mathematical Soundness Individual Mathematics Programs Bridges in Mathematics Holt Investigations Math Connects Math Expressions Math Thematics Prentice Hall Appendices A. Descriptions of Mathematical Concepts B. Comparison of OSPI and ST Rubrics C. Relationship Between OSPI and ST Scores 2 2 5 7 9 10 12 12 13 13 21 25 25 28 31 31 33 35 36 37 39 41 43 43 48 49 Curriculum Review 2 Introduction With the Washington State K – 12 Mathematics Standards firmly in place, Washington has turned its attention to finding instructional programs and materials that align with those standards. To a large degree, curriculum controls what is learned and how it is learned. Sound programs can help guide new teachers and can shore up the skills of teachers who lack a strong background in mathematics. Good curricula also provide top-notch teachers the materials they need to do their best work. As the legislature requires, the Office of the Superintendent of Public Instruction, OSPI, identified programs for elementary and middle school. Based on a comprehensive curriculum review, OSPI recommends Math Connects (K–5) and Bridges in Mathematic for elementary schools and Holt Mathematics and Math Connects (6–8) for middle schools. The Washington State Board of Education, SBE, recognizes the importance of choosing wisely. The mathematics programs that school districts implement determine what teachers use and what students see every day. SBE has requested that Strategic Teaching, ST, work with the SBE Math Panel to study the OSPI review. The purpose is to ensure that OSPI’s highest ranked programs do meet the state’s mathematics standards. Executive Summary During June of 2008, OSPI reviewed twelve elementary and thirteen middle school sets of comprehensive instructional materials for mathematics. The process involved four independent reviews for each program and a statistical analysis of the reviews. This produced a rank ordering of all elementary school programs and another rank ordering for all middle school programs. The final report also included detailed information about every program, down to the standard level. Of the four top ranked program, ST agrees with OSPI that Math Expressions and Math Connects (K-5) at the elementary school and Math Connects (6-8), Holt Mathematics and Prentice Hall at the middle school align to the content in the standards. These programs are also mathematically sound and prepare students to move forward in mathematics. Curriculum Review 3 At the request of the SBE, with guidance from the SBE Math Panel, ST reviewed OSPI’s work. ST examined selected grade levels of the top ranked programs to be sure that the programs do indeed meet Washington’s standards. Specifically, ST: Reviewed the approach used by OSPI; Included an analysis by a nationally renowned mathematician of core mathematics topics in each program to determine the mathematical soundness of each program; Evaluated the amount of content in each program and compared its findings to those of OSPI; and Summarized the key characteristics of each program. ST found the approach OSPI used to review curriculum rigorous and similar in many ways to those used in most jurisdictions. OSPI’s contractor, Relevant Strategies, was professional and thorough at each stage of the process. The curriculum review was designed to be fair to all publishers. There are other methods of statistical analysis that might have been employed, but there is no reason to believe the outcomes would have been different. ST included a mathematician’s review of core topics—whole number multiplication, area of triangle, addition, subtraction, multiplication, and division with fractions and proportionality—to see how each is developed across grade levels in the programs under review. This is an examination of whether or not the instructional materials accurately represent the field of mathematics and whether or not errors related to the topics exist. To be clear, a mathematician is someone who studies the field of mathematics and is not an expert on teaching mathematics to students or on preparing teachers to do so. At the elementary level ST determined Math Expressions to be a rare find that completely includes the conceptual underpinnings and the procedural knowledge, with clear explanations between the two for the important content in Washington’s standards. Math Connects offers an acceptable choice, because its few shortcomings are easily remedied. Bridges and Investigations, which were judged to be relatively low on content alignment by both OSPI and ST, were also found to be mathematically problematic. At the middle school level, Math Connects is the strongest program, needing only a small amount of supplementation that can be easily done. Holt Mathematics and Prentice Hall Mathematics are acceptable choices, yet share the characteristic of depending on knowledgeable teachers Curriculum Review 4 to make connections among mathematical topics and to explain why procedures work. Math Thematics is mathematically problematic in its treatment of proportionality. The comparison between OSPI’s results and ST’s results regarding how well particular programs match to Washington’s standards is complex. ST found as much or more content as did OSPI in all but two programs. However, there was a great deal of variation at the standard level and even at the topic level. There are a lot of reasons for the variation including different rubrics, different number of reviewers, and the fact that ST didn’t give credit for off-grade level content. It may be that reviewers looked at the same materials and came to different conclusions. It is also possible, because ST looked at every lesson in every program rather than the lessons identified in the publisher’s alignment, that ST looked at more material and that resulted in slightly higher scores. A combination of both is likely. Of the top-four ranked programs, ST agrees with OSPI that Math Expressions and Math Connects (K–5) at the elementary school and Math Connects (6–8) Holt, and Prentice Hall at the middle school align to the content in the standards. These programs are also mathematically sound and prepare students to move forward in mathematics. Curriculum Review SBE asked ST to do three tasks. SBE asked SBE to evaluate the process used by OSPI. This is discussed in detail in the next section, but simply put, OSPI professionally applied practices commonly used to review curriculum. The process was designed to give every program an equal opportunity to be selected. SBE asked ST to review the highest-ranking programs at the elementary school and middle levels at grades two, four, and seven. At the elementary school level, Bridges in Mathematics, Investigations, Math Connects, and Math Expressions were examined. At the middle school level Holt Mathematics, Math Connects, Math Thematics, and Prentice Hall were selected. The relationship between the amount of matching content OSPI found in a program and the amount of matching content ST found in a matching program is complex. To summarize the finding of the twelve programs and grade levels that are reviewed: In two programs OSPI found more total content alignment between the program and the standards than did ST. In six programs OSPI and ST found about the same amount of total content in the program and in the standards. In four programs ST found more total content alignment between the program and the standards than did OSPI. Even when the total content was similar, the content was often found in different standards. More details about the comparison can be found in the sections for “Elementary School Findings” and “Middle School Findings.” SBE also requested that a mathematician review the programs to determine whether or not each is mathematically sound. To do this, ST identified core topics and examined their development across grade levels, from introduction to consolidation. ST looked at both the conceptual and the procedural sides of the topics, as well as how thoroughly procedures are explained. The topics of whole number multiplication, area of a triangle, arithmetic of fractions, and proportionality were examined. Results can be found in the sections for “Elementary School Findings,” “Middle School Findings,” and “Individual Mathematics Programs.” 5 Charge to Strategic Teaching Curriculum Review Additionally, the SBE Math Panel had questions about OSPI scores and analysis. One concern was that OSPI’s approach eliminated two programs at the elementary school level and two programs at the middle school level that are statistically tied to programs included in the top four ranked programs. ST appreciates this concern, noting that ST would also have taken a different approach to the analysis. However, OSPI’s methods are both sound and common, so only four programs will be reviewed at each level. There was discussion about some standards being more important than others and whether important standards should be weighted in the scoring process. Given the difficulty of finding consensus around which standards are most important, ST believes that reporting information by strand will at least provide information about where strong content matches exist. The Panel also had questions about how OSPI and ST score standards that receive middle scores. A detailed discussion can be found in Appendix C, but broadly speaking, standards that receive a low total score from OSPI tend to receive a low total score from ST. Standards that receive a high total score from OSPI, tend to receive a high total score from ST. 6 Curriculum Review 7 Review of OSPI Process OSPI contracted with Relevant Strategies to conduct a review of all comprehensive elementary and middle school mathematics programs. Relevant Strategies applied the highest standards of professionalism to this task. It is clear that the review was designed to answer an open question, “Which sets of instructional materials meet the Washington State standards?” and not to support a foregone conclusion. Meticulous attention to details throughout the process minimized the possibility that unconsidered factors would influence outcomes. For example, each publisher had the same amount of square footage to display its materials. Serious effort was made to eliminate or at least reduce bias from usual sources. In particular, each reviewer was asked if he or she could review a specific program fairly—without bias—before he or she was given that program to evaluate. OSPI convened a group of forty-two people between June 22 and June 27, 2008, to review the instructional materials. During that time, the reviewers received training on the standards, training related to scoring, on-site support, and daily meetings to discuss issues related to scoring. Reviewers who scored the same standard zero and two—a difference of two points—were given the opportunity to discuss their rationale and change their scores. The reviewers were systematically assigned and reassigned in ways that reduced the possibility of “scorer drift,” the tendency of a group that works together for a long time to start to think and score alike in ways that may be different than the main group. Four independent reviewers matched and scored every standard to every program at every grade level using a three-point rubric for every program at every grade level, K through eight. Time constraints meant that one program at one grade level was reviewed by three, rather than four, reviewers. The points on OSPI’s rubric ranged from zero, meaning the standard is not met, to two, meaning the standard is fully met. OSPI scored and ranked the programs on multiple components. Specifically it examined the degree to which each program aligns with the content in Washington’s K – 8 Mathematics Standards and the 1) quality of the program’s organization and design, 2) balance of student experience, 3) assessment, 4) instructional planning and professional support, and 5) equity and access. It is clear that the review was designed to answer an open question, “Which sets of instructional materials meet Washington’s standards?” and not to support a foregone conclusion. Curriculum Review 8 The programs are ranked using an average of the scaled composite scores1 of all of the standards across all of the grade levels. Content is weighted at 70 percent of the overall score. There is some variability in the weights given to the other attributes, but the total contribution of the non-content aspects is 30 percent. A total of 12 elementary programs and 13 middle school programs were evaluated. A wide range of approaches is used for curriculum reviews by various jurisdictions and there is probably no best way. It is important that the process be inclusive and fairly considers all viable options. OSPI seems to have done this by including representation from all interested groups and publishers. OSPI applied a particularly formal and systematic approach to its process—the type often associated with assessment events. Given the attention and tension around curriculum in Washington, the exceptional attention to detail was probably wise. Another strength of OSPI’s curriculum review is that information is presented for every reviewed program down to the level of specific standards, meaning the report is useful to districts ready to adopt new programs and for districts that will need to supplement current adoptions that were reviewed. There are multiple of ways to conduct curriculum reviews and OSPI’s approach has many strengths. Particularly commendable are 1) the amount of input and feedback from stakeholder groups, 2) the rigorous attention to detail in inventory, training, and scoring, and 3) the specificity of the results and their usefulness to school districts. All this said there are three areas that ST would have approached differently. First, mathematicians should have been involved in reviewing the instructional materials. Content can be included but presented in ways that build long-term misunderstandings among students. While mathematicians do not necessarily understand the pedagogical issues related to teaching mathematics, they bring a knowledge of the discipline of mathematics that most mathematics educators lack. Second, a four-point rubric has inherent advantages over a three-point rubric. The additional score point forces more refined decisions and the even 1 The score that is given divided by the maximum possible score, which converts the score to a scale of “1”. This means a standard judged fully met all the points gets “1” no matter how many actual points are awarded and allows comparisons to made because the scale is the same. To take the scaled average of these professional judgments, opinions really, out beyond one or two decimal places gives a false impression about the precision of the data Curriculum Review 9 number of levels forces the reviewer to make a decision about whether the standard is closer to “completely met” or “completely missing.” Last, the middle level in odd-numbered rubric tends to become a catchall and to receive a disproportionate amount of scores.2 Third, there are questions about the statistical analysis. To be clear, ST is not suggesting that the results would be different from those produced by OSPI because of the data analysis. ST questions how well the data set stands up to the amount of analysis that has been done. The simplest example can be seen in the scaled score. This is the average of four scores that range from zero to two translated to a one-point scale. To take the scaled average of these professional judgments, opinions really, out beyond one or two decimal places gives a false impression about precision of the data. ST also questions the method employed by OSPI to look at reviewer bias. OSPI chose to use t-test and adjust for multiple comparisons, which is a sound educational statistics method. ST believes that a more appropriate approach would have been to apply the generalizability theory, a methodology designed for situations exactly like curriculum review. That being said, OSPI’s approach is able to identify programs that align to Washington’s standards and other methods most likely would not have improved on these results. Differences in Scoring Methods There are differences—important differences—between the approaches of OPSI and ST that are largely driven by the differences in project goals. OSPI had the task of reviewing all viable comprehensive mathematics programs in order to recommend up to three programs for the elementary schools and up to three programs for middle schools. ST has the task of reviewing OSPI’s work and so has only to sample and verify OSPI’s results. These differences are summarized in the following chart. 2 Wiggins, Grant P. Educative Assessment: Designing Assessment to Inform and Improve Student Performance (San Francisco: Jossey-Bass, 1998), VI-5:1. Retrieved from Knowledge Quest on the Web on October 4, 2008 http://www.ala.org/ala/mgrps/divs/aasl/aaslpubsandjournals/kqweb/kqarchives/volume31/312philip.cfm#Ref6 Curriculum Review 10 OSPI ST Reviewed Programs 12 Elementary 13 Middle 4 top-ranked elementary 4 top-ranked middle Content Identification Publisher’s alignment is used to locate matching lessons. Reviewers examine lessons to determine if the content matches the standard. After content is scored, reviewers make overall judgments about the other attributes that are evaluated. Every lesson is reviewed and the content in the lesson mapped to the standards. After the scoring is complete, the publisher’s alignment is used to double check that all identified lessons are considered. Notes about the program are taken throughout the process and then used to discuss the non-scored attributes at the end. Scored Attributes Content Program organization and design Balance of student experience Assessment Instructional planning and professional support Equity and access Content Mathematician Review ST mathematician reviews key topics and their development across grade levels. For this review, grade placement is not important. Rather, how topics are introduced, developed, and brought to closure are analyzed as indications of a program’s mathematical soundness. Non-Scored Analysis Program organization and design Instructional approaches Presence of extra content in the program and not in the standards General program usability Number of Reviews 4 independent reviews 2 independent reviews Number of Reviewers 42 4 Time Spent on Review An average of 3.5 hours is spent per review. This time was not carefully monitored, but reviewers report spending between 6 and 18 hours on a program at a grade level. The majority of the reviews, including discussion time between reviewers, range from 9 to 12 hours. Non-matching Scores Reviewers who have scores with 2-point spreads, discuss those differences and make changes based on their own discretion. All non-matching scores (1-point spread) are discussed and changes made at reviewers discretion. Grade Levels All grades, K – 8 2, 4, 7 Training Extensive training on standards; training on rubric and review process; daily calibration meetings Reviewers had intimate knowledge of standards; collaborative rubric development; calibration training using curriculum materials from project Rubric 3-point rubric 4-point rubric Overview of Scoring Approaches Curriculum Review 11 There are some differences that cannot be mitigated. While there is the additional detail that naturally travels with additional score points in ST’s rubric, it is missing one element that should be noted. OSPI considers content that is above or below grade level by a year to be partially meeting the standard and deserving of a “one” score. Since ST only reviewed selected grade levels and reviewers did not have access to off-grade materials, it did not do this. Such standards would receive a zero from ST. Appendix B offers a side-by-side comparison of the score points that match to each other on the OSPI and ST rubrics. The effects of some of the differences in scoring approaches can be minimized. For example, as previously discussed, using scaled scores allows comparisons between three-point rubrics and fouru-point rubrics. Focusing ST’s work to look at OSPI’s scores related to the content match also helps. This defines the work as an apples-to-apples comparison by creating a common core—content—that both groups examine. Fortunately, OSPI maintained separate data for its content scores. The effects of some of the differences, such as the additional time spent by ST, are hard to determine. It seems reasonable there will be an effect, and that this effect will vary. Publishers should have identified the important content matches for OSPI reviewers to examine, but this is not the same as ST having unlimited time to examine every lesson in detail. Some of the differences, such as the mathematician’s review, will have no effect on the numerical scores. This is a separate analysis and the results are not co-mingled. Despite these differences, the reviews are more similar than not. In both reviews trained and skilled adults apply a rubric to determine the degree to which the content in a standard is covered in a particular mathematics program. Both OSPI and ST are interested in identifying programs that make it likely the typical student in an ordinary classroom will meet the standard. While it is not reasonable to expect the numbers to match exactly, it is reasonable to expect ST’s process to verify OSPI’s results. In both reviews trained and skilled adults apply a rubric to determine the degree to which the content in a standard is covered in a particular mathematics program. Curriculum Review 12 Review Bias (NEW TEXT TO BE PLACED HERE.) Confidence Intervals (NEW TEXT TO BE PLACED HERE.) Curriculum Review 13 Reasonable people, making every attempt to make fair decisions, can draw rational conclusions that differ. Elementary School Findings Comparison of Content Match for Elementary Programs The major purpose of examining this curriculum review is to determine whether or not the programs ranked highest by OSPI contain the necessary content and so, meet Washington’s standards. In order to compare ST’s results to those of OSPI, the average scaled score for each standard is computed. This simply means that all of the scores are added together and divided by the total possible points. For example, given that four OSPI reviewers score a particular standard 0-1-1-1, the scaled score would be 3 ÷ 8 = 0.38, since 8 is the maximum number of points possible based on the rubric used. And given that two ST reviewers grant this hypothetical standard the scores of 1-2, the corresponding scaled score would be 3 ÷ 6 = 0.50, since ST uses a rubric with three at the highest score. This allows the results to be compared on the same scale and shows a difference of 0.12. There is danger in reading too much into these numbers, particularly at the standard level. In the example above, OSPI and ST agreed that the hypothetical standard is missing content and/or key teaching strategies. At first glance, one might think that ST found more content in the particular standard than did OSPI. In this case, the difference is due to the combination of differences in the number of reviewers and points on the rubric. The point is, there will frequently be differences in scores, even when reviewers agree. In an attempt to make sense of the differences, ST is using the percent form. It is easier to talk and think about a 12 percent difference, than one of 0.12. Another word of caution is in order. Even with a literal interpretation, a difference of 12 percent means that ST found 12 percent more of the total content than OSPI. It does not mean ST found 12 percent more content than did OSPI, which is actually an increase of 0.12 over 0.38 or 32% more content. As scores are aggregated, the artificial differences related to scoring methodology should disappear. For that reason, ST is comparing results at the mathematical topic and at the grade level, rather than at the individual standard level. Another danger associated with over analysis of this data is that rubric-based evaluation is not a precise science. Reasonable people, making every attempt to make fair decisions, can draw rational conclusions that differ. Curriculum Review 14 A false sense of precision can be implied when averages are calculated to multiple decimal places. For this reason, ST will use only two decimal places in its tables. All that said, quantifying the numbers offers precious insight into reviewer thinking and allows systematic evaluation and comparison that wouldn’t otherwise be possible. With the limits of the methodology clearly in mind, ST will assume that differences at the mathematical topic and grade level represent real differences in content, even if those differences are not perfectly captured by the numbers. Grade 2 OSPI Scores by Section ST Scores by Section Difference Bridges in Mathematics 2.1. Place value and the base ten system 0.88 0.67 -21% 2.2. Addition and subtraction 0.83 0.72 -11% 2.3. Measurement 0.90 0.83 -7% 2.4. Additional key content 0.78 0.30 -48% 2.5. Reasoning, problem solving, and communication 0.93 0.69 -24% Bridges Grade 2 Average Scale Score 0.86 0.66 -21% Investigations 2.1. Place value and the base ten system 0.31 0.33 2% 2.2. Addition and subtraction 0.74 0.57 -11% 2.3. Measurement 0.68 0.57 -4% 2.4. Additional key content 0.65 0.70 5% 2.5. Reasoning, problem solving, and communication 0.63 0.69 7% Investigations Grade 2 Average Scale Score 0.61 0.59 -2% Math Connects 2.1. Place value and the base ten system 0.77 0.75 -2% 2.2. Addition and subtraction 0.74 0.70 -3% 2.3. Measurement 0.55 0.73 18% 2.4. Additional key content 0.80 0.93 13% 2.5. Reasoning, problem solving, and communication 0.89 0.83 -6% Math Connects Grade 2 Average Scale Score 0.76 0.78 2% Math Expressions 2.1. Place value and the base ten system 0.56 0.72 16% 2.2. Addition and subtraction 0.79 0.85 6% 2.3. Measurement 0.58 0.77 19% 2.4. Additional key content 0.63 0.73 11% 2.5. Reasoning, problem solving, and communication 0.80 0.81 1% Math Expressions Grade 2 Average Scale Score 0.69 0.80 11% Curriculum Review 15 Grade 4 OSPI Scores by Section ST Scores by Section Difference Bridges in Mathematics 4.1 Multi-digit Multiplication 0.58 0.88 31% 4.2 Fractions, decimals, and mixed numbers 0.57 0.74 17% 4.3 Concept of area 0.65 0.94 30% 4.4 Additional key content 0.72 0.79 7% 4.5 Reasoning, problem solving, and communication 0.61 0.88 27% Bridges Grade 4 Average Scale Score 0.62 0.85 23% Investigations 4.1 Multi-digit Multiplication 0.68 0.55 -13% 4.2 Fractions, decimals, and mixed numbers 0.34 0.30 -5% 4.3 Concept of area 0.44 0.31 -13% 4.4 Additional key content 0.41 0.48 7% 4.5 Reasoning, problem solving, and communication 0.60 0.47 -13% Investigations Grade 4 Average Scale Score 0.50 0.43 -7% Math Connects 4.1 Multi-digit Multiplication 0.83 0.85 3% 4.2 Fractions, decimals, and mixed numbers 0.81 0.74 -6% 4.3 Concept of area 0.63 0.58 -4% 4.4 Additional key content 0.81 0.81 0% 4.5 Reasoning, problem solving, and communication 0.69 0.77 8% Math Connects Grade 4 Average Scale Score 0.76 0.76 0% Math Expressions 4.1 Multi-digit Multiplication 0.69 0.92 23% 4.2 Fractions, decimals, and mixed numbers 0.64 0.87 23% 4.3 Concept of area 0.63 0.58 37% 4.4 Additional key content 0.81 0.79 -2% 4.5 Reasoning, problem solving, and communication 0.38 0.77 8% Math Expressions Grade 4 Average Scale Score 0.53 0.89 36% There is a lot to say about how ST’s results compare to OSPI’s results. In a perfect world, there would have been a more consistent relationship between ST’s results and OSPI’s results. There are reasons—unlimited time, examination of every lesson—that ST’s scores might have been consistently higher. There are other reasons—not considering off-grade level content, reviewer discussions about every different score point—that might cause consistently lower scores. Apparently the effects of these variables fluctuate. Curriculum Review 16 For example, in Grade two, ST identified 21 percent less of the possible content than did OSPI did for Bridges, virtually matched OSPI with Investigations and Math Connects, and found 11 percent more of the possible content for Math Expressions. There are even differences for the same programs. Bridges offers the strongest example. In Grade two, OSPI finds that 86 percent of the possible content is present, but that only 62 percent of the possible content is present in grade four. ST has almost opposite findings, but if one thinks about the aggregate of both grade levels, the differences would almost disappear. To shed light on the inconsistencies, ST looked first at grade two Bridges, where ST found less content. ST looked at score points with large differences and selected three examples. OSPI’s score ranged from 0.68 to 1.00 for the following standards, while ST scores ranged from 0.00 to 0.33. Here are ST’s scores and comments: 2.4.B Collect, organize, represent, and interpret data in bar graphs and picture graphs. (OSPI score = 1.00; ST score = 0.33) ST Comments: The program does a good job with collecting and organizing data in bar graphs. All of the lessons about glyphs were cut to make room for the extra lessons that meet WA standards. In lessons 5-19 and 5-20, students make and interpret bar graphs (TE 579 to 586). Lesson 6-8, about the speed of marbles on ramps, has students collect, organize, and average data and includes homework where they graph data (TE 696 to 699). 6-10 is about bar graphs that represent linear relationships (TE 705 to 709). 6-11 to 6-13 asks students to apply information from bar graphs to solve problems related to rolling marbles (TE710 to 720). Workplace 9 includes bar graphs (TE 796). In unit 7 lessons 4, 6, and 28 include significant work on bar graphs. This reviewer did not see picture graphs, unless the “eyes” and “snowmen” growing charts are counted. This score seems harsh, but is the correct application of the rubric since content is missing. Bridges Grade 2 Average Scale Score 0.86 0.65 -21% Bridges Grade 4 Average Scale Score 0.62 0.85 23% Curriculum Review 17 While there is an emphasis on collecting data (Volume 2, Unit 5 and Volume 3, Unit 7), the other performances are also amply represented as they pertain to bar graphs although more emphasis on interpretation is needed. Probability is considered along with data analysis – at this age would be better to separate the two. Pie graphs are also introduced which may be too much at this age since they have little understanding of fractions. Did not see pictographs. 2.2.C Add and subtract two-digit numbers efficiently and accurately using a procedure that works with all two-digit numbers and explain why the procedure works. (OSPI score 0.75; ST score = 0.17) ST Comments: The standard algorithm for addition and subtraction is shown in Supplement A5 with the caveat to “be sure they understand that it is not a replacement for all the strategies they have already invented; it is just another option,” (Sup A5.16 to A5.19). While mastery of the standard algorithm is not expected in second grade, neither should it be a black sheep. Students share their mental strategies (counting on, making tens, etc.) but strategies for working with numerals on paper seems to be missing. There is no reason to believe that students are being led to use “efficient and accurate methods that work every time” beyond manipulatives. Work Place 9 “Key Chain Charms” has students doing 2-digit subtract with renaming “using methods that make sense to them,” (TE 631 to 632). A.5-4 is about place value as applied in addition and subtraction. There are a few pages of practice, which are often homework (TE 753) but it is not clear what students are practicing. Publisher cites 7-9, which is a class discussion of mental strategies and 5-23 to 5-25, which depends on models. While there is a good amount of practice for adding and subtracting two-digit numbers (teachers told to include every week) not clear that a procedure that “works with all two-digit numbers” is singled out or privileged nor that students are asked to explain why that procedure works. Standard algorithm that involves place value is included although student innovations are encouraged from beginning to end. In May, e.g., student-invented strategies are still being solicited in Activity 4, Set A5. All in all, there is too much emphasis on game playing and too much emphasis on methods other than the standard algorithm. Curriculum Review 18 2.4.D Model and describe division situations in which sets are separated into equal parts. (OSPI score = 0.63; ST score = 0.17) ST Comments A score of “1” is given because of the lesson at the very beginning of the year about ants dividing into columns. It is unusual placement and by itself is not enough. The publisher’s reference to Workplace 10B is not a good fit because that lesson is about dividing a whole into halves. There is an occasional reference to division but it pops up without a lot of direct instruction. It is briefly mentioned but never developed. Volume 3, Unit 7 does get to dividing candy bars between and among students but the focus is on fractions. Since ST does not have access to OSPI rationale, it is not possible to know whether reviewers looked at the same material and came to different conclusions or did not see the same material. Standard 2.2.C illustrates the point of people looking at the same material and drawing different conclusions. OSPI and ST reviewers looked at the same material and awarded different scores. While ST reviewers saw a very weak content match—0.17—between Bridges and the standards, OSPI reviewers saw a strong—0.75—content match. The rationale for OSPI’s score isn’t available, but we do know that reasonable people can look at the same evidence and draw different conclusions. In some instances it is also possible that ST looked at more material than did OSPI. Both groups looked at the materials in the publisher’s alignment. Additionally, ST reviewed every lesson, including those in the supplementary materials, which was not possible for OSPI reviewers. Also, in the particular case of Bridges, materials that should have been excluded, because they were replaced by supplemental units, might inadvertently have been included by OSPI reviewers. To understand better what happens when ST identifies more content than OSPI, disparate scores for grade four Expressions are examined more carefully. This program and grade level were chosen because that is where the largest discrepancy lies. Curriculum Review 19 ST judged Expressions to have 88 percent of the content related to multi-digit multiplication versus the 58 percent of content OSPI scored present. While generally the difference for this topic is an accumulation of ST seeing slighting more content in every standard, there are some glaring exceptions. 4.1.E Compare the values represented by digits in whole numbers using place value. (OSPI score = 0.000; ST score = 1.000) ST Comments: “Understand the value of the digit” is emphasized when decimals are taught in unit 3. Beginning on page 1047 to 1086 decimals are thoroughly taught. This is a place value based unit, that includes modeling, comparing, impact of zeros, and lots of practice via decimal secret code cards. There is optional extension into thousandths. In addition to inclusion in the spiral review, there is a follow-up lesson on pg. 1139. Using place value with numbers is a focus of Unit 3 and lays a strong foundation for comparisons. Understanding the value of the digits and how they compare in later units (Unit 11) as they relate to decimals along with the decimal Secret Code Cards makes this content fully covered. 4.3.E Demonstrate that rectangles with the same area can have different perimeters, and that rectangles with the same perimeter can have different areas. (OSPI score = 0.125; ST score = 1.000) ST Comments: This is really how the program introduces area, but the non-relationship is explicitly taught on 245 &amp;246 This content is explicitly addressed on p. 245-246. 4.4.F Describe and compare the likelihood of events. (OSPI score = 0.000; ST score = 0.833) Curriculum Review 20 ST Comments: Unit 9 focuses on probability and the likelihood of events. There are two pages of practice for students—more is needed as this can be a difficult concept for students. Unit 9 is about probability and covers equally likely, probability, sample space, and outcomes with typical activities like penny tosses and drawing marbles. It includes area probability using spinners. It ends with students designing a fair game. Although no single lesson focuses on comparing the likelihood of events, it is embedded and questions are sprinkled throughout. TE 945 to 956 One has to believe that ST reviewers saw lessons and units that OSPI reviewers did not see. If it were possible for everyone who reviewed a program to discuss their scores and to share his or her rational, ST believes the scores would be much more similar. All this said, even without the perfectly correlated idea, the question to be answered is, “does the program meet the standard?” It is not necessary to have perfectly coordinated scores to answer that question. Curriculum Review 21 Mathematical Soundness in Elementary School The majority of the attention during OSPI’s curriculum review and ST’s subsequent curriculum review audit is focused on matching the program content to the standards. The mathematical analysis is an additional review of the material that looks carefully at selected topics across grade levels. It goes beyond the simple content match to determine whether central topics are correct from a mathematical viewpoint. The analysis is not about how the mathematics is taught—a question better answered by mathematics educators—but about how accurately mathematics is represented in the program. A mathematician is best prepared to make judgments about the mathematical soundness of representations of concepts, the accuracy of definitions. He or she notices when critical explanations are missing. ST selected three topics at the elementary school and two at middle school to be thoroughly analyzed from their introduction through closure of the topic. Grade placement, while noted, was not of particular import. For elementary school, the topics of whole number multiplication, area of the triangle, and the arithmetic of fractions were selected. A detailed discussion of each is found in Appendix A The following chart offers a visual overview of the results of ST’s examination. Please note it does not reflect or relate to a summary of scores—no rubric was used for this component of ST’S work. Detailed analysis of each of the programs can be found at XXXX. Whole Number Multiplication Area of Triangle Adding and Subtracting Fractions + All necessary concepts and skills related to the topic are introduced, developed, represented, and consolidated using correct mathematics. √ The topic is introduced and consolidated without mathematical errors, but some aspect is not thoroughly developed. - The typical student would not master the content or be ready for the next level of mathematics. Bridges In Mathematics + — — Investigations — — — Math Connects + — + Math Expressions + + + Curriculum Review Of the four programs, Math Expressions is mathematically the strongest. With great clarity and simplicity the program goes from a geometric representation of multiplication using place value through a sequence of notations to end up with the standard algorithm for whole number multiplication. The content is not just present and correct, it is elegantly done. The thread related to area of triangle is completely covered. The formulas for the areas of triangles, parallelograms, and rectangles are all thoroughly developed. The area of the triangle is related to that of the parallelogram and this to the area of a rectangle. All of this is done with clarity. The program also does an excellent job of developing the addition and subtraction of fractions. Fractions are defined as numbers by using fraction strips and the number line. Addition and subtraction are represented on the number line as well. There are good problems. All three topical threads are covered. With Math Expressions, students should be prepared for the next levels of mathematics. Math Connects covers the thread related to multiplication with a minor exception. It lacks a numerical model for multiplication that contains place value and demonstrates commutativity and distributivity all in one place. The definition and the formula for the area of a rectangle are well done, but the formula for the area of a parallelogram is not given. Students are shown how to compute the area, but the formula is not made explicit. The real failure of Math Connects is the lack of content about the area of a triangle. This thread is mathematically sound, but incomplete. Fractions and the arithmetic of fractions are thoroughly covered. The only weakness is the emphasis on a least common denominator for adding and subtracting fractions and downplaying the easily accessible common denominator, which is, however, covered. In all, with the exception of area of triangle, students should be prepared for further mathematics when coming from Math Connects. The mathematics is clearly presented and is available in the student materials. Bridges in Mathematics finishes most of the whole number multiplication thread in Grade five of the original program. The connections to other mathematical concepts are done well enough but not always explicitly. Missing are a numerical model that demonstrates the use of commutativity and distributivity and work with three-digit numbers. In the Bridges Washington State Supplement, the transitions from representations to partial products to the standard algorithm are handled nicely. But the numerical model using commutativity and distributivity is not 22 Curriculum Review 23 included and there is only minimal work with three-digit numbers. It should be mentioned that the program does not celebrate the standard algorithm, but seems to always be looking for alternative ways to do things. Giving students the opportunity to learn a variety of methods in order to build understanding can be part of good teaching and does not require devaluing the standard algorithm. The definition of area is never clearly given in the regular program, but is included in the Washington Supplement. In the supplement, Bridges properly defines area and derives the formula for the area of a rectangle. Unfortunately, the derivation of the formula for parallelograms is mathematically incorrect and the derivation of the formula for the area of a triangle is inadequate because it is based on the very limited examples from Geoboard triangles. There are no word problems. Bridges’ grade-three supplement explains well what it means to put fractions on the number line, but falls short with representing adding and subtracting fractions as numbers. There is inadequate opportunity for students to understand addition and subtraction, partly because the concepts and skills associated with common denominators are not developed. In the Grade-five supplement, there is a mathematical error in the primary explanatory example of fraction addition. There are no word problems for fractions or area after they are developed. Bridges does not provide the necessary foundation concepts or skills related to fractions that would prepare students to go on. The strength of Investigations in the multiplication thread is the development of strategies for single-digit multiplication leading to fluency. The primary weakness is that the program does not lead to fluency with the standard algorithm. The standard algorithm is present, but receives too little attention, this is illustrated by the fact that there are only a few single-step word problems and one multi-step word problem using multiplication. The second serious problem is the lack of connectivity in the development of the standard algorithm from place-value fundamentals. This idea is not completely missing, but there are only two representations using place value that illustrate their “breaking apart by place” algorithm and the representation and the algorithm are separated by significant time in the program. The numerical representation of the algorithm’s distributivity and commutativity is available in the teacher’s preliminary material, but not in the student’s materials. The materials do not make 1) an explicit connection using commutativity and distributivity between the area representation and the numerical representation or 2) the connection between the numerical representation and the “breaking apart by place” algorithm. Investigations is missing the formulas for the areas of rectangles, parallelograms, and triangles. Curriculum Review 24 By not developing fluency with and understanding of the standard algorithm and then applying this skill to multi-step word problems, students are not prepared for the next stage of their mathematical education. Investigations is missing the formulas for the areas of rectangles, parallelograms, and triangles. The areas of rectangles are computed, but the areas of parallelograms and triangles are not. The connection between the area of a parallelogram and that of a rectangle or that of a triangle is never made. A strength in Investigations is the use of the clock model and the rectangular model to represent addition and subtraction of fractions. However this follows mathematically flawed work in grades three and four and takes place before fractions are well defined as numbers. Work in Investigations is limited to fractions with sums of less than 2 and there is very limited work with mixed numbers, leaving students unprepared to deal with fractions as simple as 11/3. Most importantly, common denominators are not developed, so students are unable to add arbitrary fractions with what they are taught. Students using this program will be unprepared to go on in mathematics. A strength is the use of the clock model and the rectangular model to represent addition and subtraction Curriculum Review Comparison of Content Match for Middle School The following chart provides a comparison of the OSPI and ST results related to the amount of content match identified by the reviewers. Middle School Findings 25 Grade 7 OSPI Scores by Section ST Scores by Section Difference Holt 7.1 Rational numbers and linear equations 0.93 0.79 -14% 7.2 Proportionality and similarity 0.57 0.78 21% 7.3 Surface area and volume 0.94 1.00 6% 7.4 Probability and data 0.95 1.00 5% 7.5 Additional key content 0.94 1.00 6% 7.6 Reasoning, problem solving, and communication 0.95 0.94 -2% Holt Grade 7 Average Scale Score 0.85 0.89 4% Math Connects 7.1 Rational numbers and linear equations 0.71 0.74 2% 7.2 Proportionality and similarity 0.64 0.80 16% 7.3 Surface area and volume 0.59 0.50 -9% 7.4 Probability and data 0.93 0.65 -28% 7.5 Additional key content 0.88 1.00 13% 7.6 Reasoning, problem solving, and communication 0.86 0.60 -26% Math Connects Grade 7 Average Scale Score 0.75 0.71 -4% Math Thematics 7.1 Rational numbers and linear equations 0.81 0.74 -7% 7.2 Proportionality and similarity 0.43 0.74 31% 7.3 Surface area and volume 0.61 0.67 6% 7.4 Probability and data 0.80 0.80 0% 7.5 Additional key content 0.92 1.00 8% 7.6 Reasoning, problem solving, and communication 0.98 0.76 -11% Math Thematics 7 Average Scale Score 0.73 0.76 3% Prentice Hall 7.1 Rational numbers and linear equations 0.82 0.81 -1% 7.2 Proportionality and similarity 0.63 0.72 10% 7.3 Surface area and volume 0.31 0.58 27% 7.4 Probability and data 0.65 0.93 28% 7.5 Additional key content 0.88 1.00 13% 7.6 Reasoning, problem solving, and communication 0.59 0.79 20% Prentice Hall Grade 7 Average Scale Score 0.64 0.78 14% Curriculum Review With the exception of Prentice Hall, ST and OSPI found similar amounts of content in the examined programs at the program level. At the topic level, there is a consistent variability that is worth examination. ST found more—it ranges from 13% to 31%—of the content that is in the standards related to the topic of proportionality and similarity than did OSPI. While is not possible to know with certainty if different reviewers looked at different materials or if they looked at the same materials and came to different conclusions, comparing ST reviewer notes and publisher alignments provides insights. ST reviewers were under no compunction to record all supporting lessons, but the details in the scoring sheets suggest they often did so. The following table compares Holt’s and Prentice Hall’s alignments and ST citations for the topic of proportionality. It is not certain that any firm conclusions can be drawn from this examination, but it seems to indicate that for the topic of proportions, the higher scores granted by ST relate to both the review of extra material and to different judgments about the same materials. There are several instances, such as in standard 7.2.C for both Holt and Prentice Hall, when ST looked at material beyond what is listed in the publisher’s alignment and awarded a higher score. Sometimes this material was completely omitted by the publisher, as was the case for Prentice Hall and standard 7.2.H, but more often ST found related material in the lessons leading up to and away from those cited by the publisher. There are also times, such as in standard 7.2.F for Holt, that both groups appear to have reviewed the same material and ST reviewers granted higher scores. 26 Curriculum Review 27 Holt Alignment ST Citations for Holt Prentice Hall Alignment ST Citations for Prentice Hall 7.2.C • Lab 5.7 pp. 298-299 • Lesson 5.7 pp. 300-307 • 5-7 with lab • 5-8 • Reteach, practice, challenge, and problem solving resources. • 5.5a • 5.6a • Pg. 270 - 271 • 5-5a • 5-5b • 5-6a • Practice sheets, adapted practice sheets, reteaching sheets, and guided problem solving OSPI score = 0.88 ST score = 1.00 OSPI score = 0.50 ST score = 0.83 • Lesson 5.9 pp. 308-311 • 5-9 • Chapter 8 extension • 5.5b • 5.6a • Pg 270 - 271 • 5-6 • Practice sheets, adapted practice sheets, reteaching sheets, and guided problem solving OSPI score = 0.63 ST score = 0.67 OSPI score = 0.75 ST score = 1.00 7.2.E • Lesson 12.5 pp. 650-654 • 5-6 lab • 4-4 • 4-5 • 4-6 • 5-3 • 9-1 • 10-1 • Problem Solving Application • 9-1 • 9-1a • 9-2 • 9-3 • 9-4 • 9-5 • 9-6 • 9-7 • 9-8 OSPI score = 0.50 ST score = 0.67 OSPI score = 0.88 ST score = 0.83 7.2.F • Lesson 5.3 pp. 278-282 • 5-3 • 9-1 • 9-4a • 10-2 • 10-2b • 10-3 • 10-3a • Practice sheets, adapted practice sheets, reteaching sheets, and guided problem solving OSPI score = 0.250 ST score = 0.667 OSPI score = 0.50 ST score = 0.50 • Lesson 5.3 pp. 278-282 • 5-2 • 5-3 • 5-2 • 5-4 • Guided Problem Solving • 5-2 • 5-2 Extension • 5-3 • 5-3b • 5-4a • 5-4b • 5-5 OSPI score = 0.50 ST score = 0.667 OSPI score = 0.38 ST score = 0.33 • Lesson 5.4 pp. 283-286 • Lesson 12.5 pp. 650-654 • 5-4 • 5-7 • 9-2 • 10-2 • 5-1 • 5-2 • 5-3 • 5-3b • 5-4a • 5-5 • 5-5b OSPI score = 0.750 ST score = 1.00 OSPI score = 0.50 ST score = 1.00 Curriculum Review Mathematical Soundness 28 Multiplication and Division of Fractions Proportions + All necessary concepts and skills related to the topic are introduced, developed, represented, and consolidated using correct mathematics. √ The topic is introduced and consolidated without mathematical errors, but some aspect is not thoroughly developed. - The typical student would not master the content or be ready for the next level of mathematics. Holt √ √ Math Connects — + Math Thematics — — Prentice Hall √ √ Overall, Math Connects is strongest, with Prentice Hall and Holt tying for the middle place. It is disappointing to see the treatment of multiplication and division of fractions across the four texts chosen for examination. While all four programs make a good start on the topic, in the end, the rules for the multiplication and division of fractions are just stated and not explained. Students need the opportunity to learn to understand the procedures they are asked to apply. As they stand none of these programs completely does this, although some come closer than others. In particular, none of them use the inverse nature of multiplication and division. The good news is that this topic is relatively easy to address with supplementation. Ratios, rates, proportions and proportionality reach into all corners of middle school mathematics. Helping students see and use the connections among scale and slope and similarity is a complex task. It cannot be amended with a few supplemental lessons and added definitions. This is a topic that must be well developed within the chosen mathematics program. Holt In Holt, the multiplication of simple fractions is nicely modeled, but the text gives no explanation for the general rule for multiplying fractions. The only attempt to explain the division of fractions is for the cases where the answer is a whole number. The rule for dividing fractions is given without explanation, as a rule to be memorized. One section on solving simple equations could be turned into an explanation of division, but it is not taken advantage of in this way. It contains equations of the sort A = BX that are solved by multiplying both sides by the reciprocal of B. This could easily be Curriculum Review 29 tied to the inverse nature of multiplication and division. The topic could be extended by replacing A, which is usually a whole number, with a fraction. Rates are not properly defined when they are introduced. There are a lot of proportion problems, but they are of mixed quality and some are mathematically inappropriate. Cross products are taught, but the underlying logic is not included in the student text. Linear equations, their graphs, and slopes are included, but underdeveloped. Proportions, slopes, and graphs are also included but the program does not establish well enough the connections that should be made through similar triangles and the proportionality constant. Direct variation is included. Math Connects Multiplication of simple fractions is nicely modeled in Math Connects, but it gives no explanation for the general rule for multiplying fractions. Division of fractions is only explained using examples where the answer is a whole number. There is no explanation given for the rule for dividing fractions. Math Connects thoroughly develops ratios, rates, and proportion, including proper definition in the text each year of the middle school program. Proportion problems are numerous, of varying types, and such problems are present in all three years of the material. Cross products are appropriately and correctly explained. The connection to a proportionality constant and a graph is not emphasized, but it is there and the connection between slope and similar triangles is also present, although it is even weaker. Math Thematics Math Thematics includes solid modeling of multiplication of simple fractions, but again, without explaining the general rule for multiplying fractions. As is common, the only attempt to explain the division of fractions is with examples in which the answer is a whole number. The text gives no explanation for the rule for dividing fractions. Math Thematics 1 one sets up a confusing foundation for ratios, rates and proportions by not giving a definition of “ratio”. This is corrected in Book 2, a year later, but that delay puts students in the position of having to build a foundation of misconceptions. Cross products are taught, but how and why they work is not explained. There are proportion problems in limited number throughout the three years, but many of them are scattered throughout the text, there is never any intensive practice. Curriculum Review 30 There is a minimal connection of graphs to ratios and rates, but there is nothing explicit about how they connect to proportions, such as the constant of proportionality. Linear equations are not shown to produce graphs that are lines, and no reason is given for slopes of lines to be independent of points chosen to compute them. This series’ presentation of proportions would not develop the foundational understanding students need. Prentice Hall Prentice Hall includes good models of multiplication of simple fractions, but it does not provide an explanation for the general rule for multiplying fractions. Similar to the other reviewed programs, the only attempt to explain the division of fractions is by using examples that have a whole number for an answer. No explanation is given for the rule for dividing fractions. There is a missed opportunity in a section on solving simple equations that could have easily been turned into an explanation of division. This is a place that the skilled teacher can add to the explanation and make the connections that are missing. Information in the teacher’s edition indicates an awareness that these procedures can be explained, but the explanations are not included. The teacher’s edition does make the statement that multiplication and division are inverse operations, but this information is not reflected in the lessons. The definitions for ratios, rates, and proportions are included and correct. Cross products are not explained in grade six, when they are first taught, but they are in subsequent grade levels. There are a variety of word problems, although they are not particularly challenging. The real weakness comes from the lack of connection between proportions and graphs. The slope is not well developed and defined, nor are equations shown to give lines as graphs, but the real weakness in Prentice Hall comes from the lack of connection between proportions and graphs. Curriculum Review 31 Bridges in Mathematics With the help of the supplemental materials, Bridges mostly covers the thread of whole number multiplication. This is not true for the topic of area. The supplemental materials include the formulas for area missing from the main program, but the derivation of the formula for the area of a parallelogram is mathematically unsound and the derivation of the formula for the area of a triangle is inadequate. The handling of adding and subtracting fractions is also problematic. Materials in the supplemental materials do address addition and subtraction of fractions, but common denominators are not included. The materials also contain a serious mathematical error in a major example used to illustrate a central point. The typical student would only be prepared to go on in mathematics for the thread of whole number multiplication. OSPI Grade 2 Average Scale Score 0.86 ST Grade 2 Average Scale Score 0.66 OSPI Grade 4 Average Scale Score 0.62 ST Grade 4 Average Scale Score 0.85 Bridges is a program structured around a limited number of units each year. Unit topics relate to mathematical strands such as fractions or data, rather than themes or problem contexts. Typical lessons begin with a problem or investigation that small groups of students tackle at their desks. Students share their thinking, strategies, and solutions in the classroom discussion that follows. There does not appear to be a systematic emphasis to bring closure or to consolidate the mathematical ideas under examination. Most days include time for students to work at centers that have games and activities related to the mathematical content under investigation in order to practice skills and explore ideas. Direct instruction is rarely used in the main Bridges program and occasionally used in the supplemental Bridges materials. There are many commendable things about Bridges. Most impressive is the publisher’s responsiveness to creating lessons that exactly match to Washington’s standards. The teacher’s edition contains a great deal of information. Every unit is preceded with a well-developed overview. There are many examples of the types of classrooms discussions that should Individual Mathematics Programs Curriculum Review 32 be generated during the lesson. Because the classroom discussions are a mainstay of the program, these are important examples. Additionally there is information about classroom management and assessment tips scattered throughout. ST reviewers found the materials hard to navigate. The program exists in multiple books—1) number corner activities, 2) main materials, 3) practice workbook, 4) home connections, and 5) the supplemental materials— and lacks both a comprehensive table of contents and an index. It would be difficult for a teacher to isolate topics for review or extension and to remember that some lessons in the main program must be replaced with lessons from the supplemental materials. Curriculum Review 33 Holt Holt begins by modeling well multiplication of simple fractions, but does not bridge from the model to a general rule for multiplying fractions. Division of fractions is only explained for those problems with whole number answers, leaving students to assume that if it works for these numbers it must work for all numbers. Although the paper folding model is included and teachers are reminded that division is multiplication by the reciprocal, there is no explanation for dividing fractions included. There is a section on solving simple equations that could be turned into an explanation of division and in some ways comes close, but it is not taken advantage of in this way. Both multiplication and division are broken down into small bits for easier digestion, making it the best of the four programs in this respect Rates are not properly defined when they are introduced. There are a lot of proportion problems, of mixed quality. Cross product is not explained in the student text, but it is in the teacher edition. Linear equations, their graphs, and slopes are done, but not done thoroughly. Proportions, slopes, and graphs are not linked up well through similar triangles and the proportionality constant, but the concept of direct variation is included. Students would be prepared to go on, assuming teachers made explicit implicit content. OSPI Grade 7 Average Scale Score 0.85 ST Grade 7 Average Scale Score 0.89 Lessons usually begin with a warm up activity that activates prior knowledge and includes a short problem of the day. Using direct instruction, the teacher guides students through a set of (usually) three example problems, each one related to a different sub-topic, that illustrate how to do solve a problem type. For example, the lesson on customary measurements has a sample problem for 1) choosing the appropriate customary unit, 2) converting customary units, and 3) adding or subtracting mixed units of measure. This is followed by guided practice, independent practice and then practice problems and spiral review. There is a dependence on direct instruction with guided and independent practice, although the program includes some games to solidify skills. There is a well-developed section on test-prep including various types of problems and tools to aid teachers such as student work samples for extended response items. Curriculum Review 34 Units are by topic and each lesson stands alone. It usually builds on the lesson from the previous day, but sometimes the effect can be a bit jumbled. Chapter five includes lessons about ratios, rates, slope and rates of change, identifying and writing proportions, solving proportions, generate formulas to convert units, customary measurements, similar figures, similar figures and proportions, scale drawings and scale models, and proportional relationships. The connections among and between these topics are not as clear as they should be. The teacher’s edition is rich with calls-outs related to math background, interdisciplinary connections, ongoing assessment, student intervention, and teaching tips. A fair amount of content is included in Holt that does not map to Washington standards. This includes a lot of chapter five, which is about angles, circles and polygons, congruent figures, translations, reflections, rotations, and symmetry; most of chapter six, which is about percent, percent of change, and interest; and most of chapter nine, which is about measurement of two-dimensional figures. However, the program is designed in such a way that it is relatively easy for a teacher to skip chapters, as long as they know to do so, meaning the content does not have to take time away from teaching Washington’s standards. Curriculum Review 35 Investigations, as the name suggests favors an approach to learning that is exploratory in nature. A typical day begins with “10 Minute Math” which gives students the opportunity to maintain and practice learned skills and numerical fluency. Most of the remainder of the day is spent with students working in groups to investigate a mathematical situation or problem, which is different than applying mathematics they know to solve a problem. There is an emphasis on students discussing their strategies and solutions. The strength of the program is in the suggestions for conceptual development of key concepts. It is hard to stress how well this is done in Investigations. Students develop strong visual models for challenging concepts such as fractions, multi-digit multiplication, and area. However, the transition to abstract fluency or generalization is not always developed or evident. For example, students develop good number sense, but may not develop efficiency in operations. Two other issues should be mentioned. Pacing may prove challenging. While each session seems appropriately designed for a class period, the teacher is asked to assess frequently to decide on the number of additional problems to provide and to provide enough time for struggling students to understand the concept. At the same time, there are insufficient extensions or enrichments to engage adept students who may more easily learn the concepts. Also, at some grade levels, including grade four, Investigations includes a great deal of content that does not match grade four standards. As much as half of the content matches to standards at other grade levels and the grade level is almost always lower than Washington’s grade four. Investigations Investigations receives the lowest scaled score given for content from both OSPI and ST of the four top-ranked middle school programs. The necessary components of whole number multiplication are there, but the connections are not. Fluency with the standard algorithm is not developed. In the area thread, there is no formula for the area of a rectangle although students do computer area. Parallelograms and triangles are not considered. Addition and subtraction of fractions has a good start with strong models, but this foundation is not used to build the necessary mathematics. A typical student would not be prepared for the next level of Washington mathematics using this program. OSPI Grade 2 Average Scale Score 0.61 ST Grade 2 Average Scale Score 0.59 OSPI Grade 4 Average Scale Score 0.50 ST Grade 4 Average Scale Score 0.43 Curriculum Review 36 Math Connects Math Connects is the program that is ranked in the top four for both elementary and middle school. At the elementary level, the thread related to multiplication is covered except it lacks a numerical model for multiplication that contains place value and demonstrates commutativity and distributivity all in one place. The definition and the formula for the area of a rectangle are well done, but the formula for the area of a parallelogram is not given. Students are shown how to compute the area, but the formula is not made explicit. The one piece that is completely missing is the area of a triangle. This thread is mathematically sound, but incomplete. Fractions and the arithmetic of fractions are thoroughly covered. The only weakness is the emphasis on a least common denominator for adding and subtracting fractions and the de-emphasis of the easily accessible common denominator, Still, students should be prepared for further mathematics when coming from Math Connects. The mathematics is clearly presented and is available in the student materials. In middle school, Math Connects makes a good start with good modeling of multiplication of simple fractions but no explanation for how this extends to the general rule for multiplying fractions is given. While this is not always easy to see, one way is to show the connection to repeated addition. The only attempt to explain the division of fractions is for the cases where the answer is a whole number. No explanation is given for the rule for dividing fractions, when it could be as easy as making the connection to the inverse relationship of multiplication and division. Ratios, rates, and proportion are properly defined every year and proportion problems are numerous, of varying types, and in all three years of the middle school program. Cross products are used but not explained properly. The connection to a proportionality constant and a graph is made, but not emphasized enough and the connection to slope and similar triangles is there with even less development. With very little extra, students would be well prepared in the topic of proportionality. They would need a better understanding of why the procedures associated with multiplication and division of fraction work to go to the next level of mathematics. The missing pieces are easy to supplement. OSPI Grade 2 Average Scale Score 0.76 ST Grade 2 Average Scale Score 0.78 OSPI Grade 4 Average Scale Score 0.76 ST Grade 4 Average Scale Score 0.76 OSPI Grade 7 Average Scale Score 0.75 ST Grade 7 Average Scale Score 0.71 Curriculum Review 37 Math Expressions Math Expressions develops whole number multiplication completely and elegantly. The area thread is done extremely well. Adding and subtracting fractions, both the conceptual and procedural sides as well as the connections between the two, is explicitly developed. With the exception of the work on common denominators, all of the mathematics is in the student materials. This program provides a solid foundation to move ahead in mathematics. OSPI Grade 2 Average Scale Score 0.82 ST Grade 2 Average Scale Score 0.66 OSPI Grade 4 Average Scale Score 0.62 ST Grade 4 Average Scale Score 0.85 A typical day starts with a short skills building or practice and homework review. The actual lesson usually starts with a problem that students work on, in groups, at their desk. This is followed by “math talk,” Expressions’ version of student discourse, during which the teacher pulls out student thinking and summarizes and consolidates the important mathematical ideas for the day. Teachers sometimes use direct instruction during this time to teach specific skills or concepts. Students are then led through some guided practice or group work and on to independent practice or a group task. Every lesson has a homework worksheet and a spiral review worksheet. The teacher’s edition is very strong. Although the front material is a bit overwhelming, there is a complete table of contents and index, so content is very easy to find. The goal for each lesson is clearly identified, as is a suggested time frame and a list of specific materials that will be needed. “Best practice” ideas, such as the main points that should be included in the class discussion, are woven into the teacher’s edition. Sometimes it alerts the teacher to commonly held student misconceptions and ideas to support the student. Sometimes it offers general guidance on how to teach a lesson like, “Students will also probably say that both arrays have six dots; accept this answer because it works for these specific arrays. But it cannot be the basis for a general argument for all arrays asked for in the next exercise.” Curriculum Review In a typical lesson, the teacher begins with some scaffolding questions that active prior learning, the new concept is explained/demonstrated and the class is asked a few questions to check for understanding, the teacher works some examples, students work some problems to be sure they know what to do, and then the students work practice problems which include some spiral review problems. While every program has a predictability, this one seems to suffer from adherence to its format. At the elementary level, the teacher is often directed to \"Have students open their books and read the information in ...\" Hands-on activities are included in the teacher’s edition but identified as appropriate for use with below grade students. Similarly, at the middle school level, students rarely experience hands-on mathematics. The program does include approximately six mini-labs per year, designed to last about one-half hour, scheduled before a new major concept is introduced. With the exception of the mini-labs, students spend most of their time on the procedural side of mathematics. The program appears relatively easy to use, although the teacher’s edition has so much information—not all of it helpful—that a teacher might discount it. Alternative introductory lessons and teaching strategies for English Language Learners are included. The directions to the teacher are often very general, and not specific to the problem or concept. While there is a good deal of information about differentiation - some useful, some not - there is not enough good information about the whole group instruction. The additional worksheets provide plenty of additional material for teachers to use to reteach and enrich. 38 Curriculum Review 39 Math Thematics Math Thematics does a good job of modeling multiplication of simple fractions and giving students the opportunity to practice doing so in a meaningful context. What is missing is the bridge between the model for the simple case and the explanation for the general rule. The only attempt to explain the division of fractions is for the cases where the answer is a whole number. No explanation is given for the rule for dividing fractions. Book 1 sets up a confusing foundation for ratios, rates and proportions by not giving a definition of ratio. This is corrected in Book 2, but by then students are working from a shaky foundation. Why and how cross products work is not explained. There are proportion word problems, but they are limited in number and scattered throughout the material. A minimal connection of graphs to ratios and rates is made, but nothing explicit is done about proportions, such as the constant of proportionality. Linear equations are not shown to produce graphs that are lines and there is no explanation given for slopes of lines to be independent of points chosen to compute them. A student would be unprepared to go on in mathematics, although multiplication and division of fractions could be easily supplemented. OSPI Grade 7 Average Scale Score 0.73 ST Grade 7 Average Scale Score 0.76 Math Thematics organizes takes traditional content and organizes it into themes or modules. For example, the first module, “Search and rescue” uses an excerpt from a book, Hachet, in which a young man is stranded after his plane crashes. The unit relates the math in the unit—including angles and rays, coordinate grids, order of operations, adding and subtracting integers, modeling functions, writing equations to model situations, and using inverse operations to solve one-step equations—to search and rescue situations. In the second module, “Bright ideas,” the text bundles many important mathematical discoveries along with the key information about the person who developed the mathematics, thereby at least exposing students to some math history. The remaining modules are: Codes, The Art of Motion, Recreation, Flights of Fancy, Math-Thematical Mix, and Heart of the City. Every section includes “Reflecting” and “Spiral Review.” Reflecting has questions and problems that push and check student understanding. Each unit ends with a series of application problems Curriculum Review 40 that requires students to apply the mathematics learned in the unit in a realistic context. For example, the first problem in unit one requires the student to develop plan a rescue plan with limited daylight and gas for the rescue plane. These concepts and skills are put into realistic contexts. The lessons are highly engaging, but the mathematics gets a bit chopped up and the later units seem fragmented. Also, the organization means that it is difficult to omit the mathematics that is not included in Washington’s standards and there is quite a lot of that. The program is easy to use, with a table of contents and index. There is a lot of help for teachers. The margin in the TE include “Tips from Teachers,” “Developing math concepts,” which are ideas for how to introduce and teach a topic; “Teaching notes,” which are mathematical content background notes for teachers; “common errors,” (example: “when substituting a value for a variable in a multiplication expression, students will sometimes forget to perform the multiplication and simply write the number....Remind students to write the multiplication symbol or to use parentheses.” which describe not only errors students commonly make, but also ways to address them; “About Key Concepts” gives more information about the mathematics; “Absent Students,” which identifies materials to use that compress the missed lesson(s); “Classroom Management” has teaching tips that are germane to the specific lesson; “Differentiated Instruction” proves suggestions for that topic, and “Closure question,” which ensures students leave the lesson with the main idea. Unfortunately, not all of the important mathematics is included in the teacher help and not all of it is explained when it is included. Curriculum Review 41 Prentice Hall Multiplication of simple fractions is nicely modeled but no explanation for the general rule for multiplying fractions is given meaning students would not understand why the rules work. Division of fractions is explained only for the cases where the answer is a whole number. No explanation is given for the rule for dividing fractions. There is a section on solving simple equations that comes very close and could easily turned into an explanation of division. There is an indication, in the teacher’s edition, that these formulas can be explained, even though the explanations are missing. Similarly, the teacher’s edition makes the statement that multiplication and division are inverse operations, but this information is not included in the student text nor used in the lessons. Correct definitions are included for ratios, proportions, and rates when a concept is introduced. Cross products are not explained the first year it is taught, but they are in the year two. There are a variety of problems but none are particularly difficult. It is left to the teacher to fill in the connections between proportions, slopes, and graphs. Generally, slope is not well defined and equations are explicitly shown to give lines as graphs. While the material is present, but the teacher must do the teaching. OSPI Grade 7 Average Scale Score 0.64 ST Grade 7 Average Scale Score 0.78 Prentice Hall is a solid mathematics text that contains appropriate mathematics. Activity labs are used prior to the majority of lessons to develop conceptual understanding of mathematical procedures. These are an essential component of the text; it is during these labs that the conceptual base for students’ understanding of specific mathematical procedures occurs. Good models exist in the labs; however the “bridge” between the models and the rules are not made explicit in the teaching notes or in the actual lesson. Since fluency may not be certain without conceptual understanding, teachers will need to know to do this. The actual lessons begin with a summary of “what” will be learned and “why” it is needed, a list of essential skills and vocabulary, and multiple examples where key concepts are highlighted. In addition to labs, the program uses a mix of direct instruction and student group work that leads to guided and independent practice. Homework exercises offer “Go for Help” and “Go Online” notes to assist students with finding appropriate examples and tutoring help. Curriculum Review 42 The TE includes a “Mathematics Teacher Handbook” at the beginning of the text that explains how to use the teacher text effectively. This section provides information on where to find real-world applications, activity labs, chapter projects, and specific problem solving strategies throughout the text. Support is provided for lesson differentiation, formative and summative assessment, technology use, and additional professional development opportunities. A “Math Background” summary precedes each chapter. This includes essential math understandings for successfully teaching the concepts in the chapter; teachers that by-pass reading these two-page summaries risk omitting important connections between procedures and the conceptual understanding of those procedures. Included in the margins of each lesson are additional math backgrounds relevant to the individual lesson, guided instruction notes to deepen student understanding, differentiated instruction suggestions, lesson openers and closers, common error alerts, additional examples, and assessment tools. Even with all of this, the teacher’s edition often does make explicit why procedures work and connections between various strands of mathematics. The student edition does not stand alone. It does not make the connection explicit between the individual lessons and the lab used to develop the procedures used in the lesson. For example, in 2-4a Activity Lab Comparing Fractions, students model comparing fractions with paper strips. The lab includes a discussion of why it is easier to compare fractions with common denominators. In the lesson 2-4, students compare fractions by writing the fractions with common denominators and comparing the numerators. No reference to the lab is made and no alternate strategies are included; the teacher is needed to make the connection. Curriculum Review 43 Appendix A Whole Number Multiplication The standard algorithm for multiplication has particular importance in mathematics for several reasons. First, it is fairly easy to link it to the all-important place value system. Second, it is necessary in order to extend multiplication to decimals; the same algorithm is used, it is just the decimal place that has to be understood. Third, whole number multiplication is used for all four arithmetic operations with fractions: , and . Fourth, multiplication is used throughout all of mathematics. The standard algorithm solves the universal problem of whole number multiplication, taking the ad hoc out of multiplication. As such, it is one of the truly beautiful and powerful mathematical theorems that students can learn about in elementary school. Learning the algorithm is much more than just learning a way to multiply, it is learning a major mathematical structure. The thread we consider starts with the following: 3.2.D Apply and explain strategies to compute multiplication facts to 10x10 and the related division facts. This is an important preliminary, as is indicated in the Washington standards, to the following: 4.1.A Quickly recall multiplication facts through 10x10 and the related division facts. The standard algorithm is based on distributivity, commutativity, and these single digit multiplications. Knowing all without hesitation is important for fluency with the standard algorithm. Once these are all under control it is possible to connect multiplication with place value, as is expected in this standard: 4.1.C Represent multiplication of a two-digit number by a two-digit number with place value models. The usual representation is to break up numbers using expanded notation and to use the area model of multiplication, thus connecting multiplication to its foundation in the place value system. There are certainly other acceptable models, but it is essential these include a numerical model to € ab×cd=acbd€ ab±cd=ad±bcbd€ ab÷cd=adbc Curriculum Review 44 make the step from a representation to numbers. In the numerical model we need this standard: 4.1.D Multiply by 10, 100, and 1,000. Then, multiplication, as it is represented using place value, must be connected to the notation for the standard algorithm so that we can achieve the standard below: 4.1.F Fluently and accurately multiply up to a three-digit number by one- and two-digit numbers using the standard algorithm. There is a lot that is important here. The notation for the standard algorithm must be connected to a place value representation so that the notation makes sense to students. Obviously, learning this standard by rote is inadequate. This standard will be emphasized as the goal, because without it, even with the foundation of good developmental standards, a student is unprepared to go on in the study of mathematics. Once fluent, multiplication should cease to be a problem but should just be a skill that can be used in problem solving. Thus, ultimately, we need the standard below: 4.1.I Solve single- and multi-step word problems involving multi-digit multiplication and verify the solutions. The point of this thread is to develop an understanding of and a facility with the standard algorithm for multiplication so that it can be readily used as a tool for problem solving. This is of fundamental mathematical importance, and this is reflected by its place in the Washington State standards. Area of a Triangle Area is a fundamental concept in mathematics. The area of a triangle is derived from that for a parallelogram (by taking two copies of the triangle and making a parallelogram) and the area for that is, in turn, derived from the area of a rectangle (by cutting up the parallelogram and rearranging the pieces to make a rectangle). The area of a rectangle follows from multiplication and the area of a square, which essentially gives the definition of area. With the area of rectangles, parallelograms, and triangles under control, the area of polygons and planer surfaces can be calculated. In practice, measurement also comes into play. The computation of volumes frequently depends on knowing the areas connected to the solid. As the student progresses, the techniques for computing area also progress. The Integral Calculus allows students to extend their knowledge of area to much more Curriculum Review 45 complex figures, but that ability depends heavily on the understanding of area from elementary school. This thread begins with the area part of: 4.3.C Determine the perimeter and area of a rectangle using formulas, and explain why the formulas work. The “explain why the formulas work” part of this is essential. Next, we go to: 5.3.D Determine the formula for the area of a parallelogram by relating it to the area of a rectangle. The goal of this thread is given by the next two standards: 5.3.E Determine the formula for the area of a triangle by relating it to the area of a parallelogram. and 5.3.I Solve single- and multi-step word problems about the perimeters and areas of quadrilaterals and triangles and verify the solutions. Fractions and the Arithmetic of Fractions The importance of fractions to mathematics cannot be overstated. Students start with whole numbers and then develop fractions to get the rational numbers. Eventually, real numbers can be constructed from the rational numbers and, finally it is possible to add the last bit to get complex numbers. With the real numbers one can put coordinates on the line or the plane, and from this comes the ability turn geometry into algebra and algebra into geometry. The point is that fractions are an essential intermediary step in mathematics. Numbers and geometry are at the heart of mathematics, and fractions are required for both. You can’t do mathematics without an understanding of fractions and their operations. To do fractions requires a thorough understanding of whole numbers and the arithmetic operations with them. The step of going from whole numbers or integers to fractions is the same step taken to go from polynomials to rational expressions. This is a common transition in mathematics and elementary school fractions are the easiest place to start. The four arithmetic operations with fractions are divided between grades Curriculum Review 46 five and six in the Washington State standards. This crosses the boundary between elementary school and middle school and so it is likely that different programs will be used for these two grades. That boundary makes it all the more important to accomplish the grade five standards in grade five, and that middle school be able to assume mastery of the grade five standards. Because of this jurisdictional problem, the following is a discussion of the elementary school material. This thread begins with seeing a fraction as a number, and putting it on the number line will do, as in: 3.3.A Represent fractions that have denominators of 2, 3, 4, 5, 6, 8, 9, 10, and 12 as parts of a whole, parts of a set, and points on the number line. Next fractions need to be defined or represented as numbers in addition and subtraction: 5.2.A Represent addition and subtraction of fractions and mixed numbers using visual and numerical models, and connect the representation to the related equation. Finally we need a purely numerical way to add and subtract fractions: 5.2.E Fluently and accurately add and subtract fractions, including mixed numbers. In order to do this, students must understand what denominators mean and how to find and work with common denominators. We also need the fraction part of the following standard: 5.2.H Solve single- and multi-step word problems involving addition and subtraction of whole numbers, fractions (including mixed numbers), and decimals, and verify the solutions. Multiplication and Division of Fractions Continuing the fraction thread from elementary school we will focus on multiplication and division for fractions by doing the following: 6.1.D Fluently and accurately multiply and divide non-negative fractions and explain the inverse relationship between multiplication and division with fractions. And by presenting the corresponding fraction part of the application standard 6.1.H Solve single- and multi-step word problems involving operations with fractions and decimals and verify the solutions. Curriculum Review In order for word problems with fractions and decimals to make sense for students, there needs to be a good representation of the multiplication of fraction multiplication. It is difficult to represent fraction multiplication for anything but simple fractions so there has to be a logical connection and explanation that takes the student from the simple cases they can see with a representation to the general formula. There are multiple ways to do this, both geometric and algebraic (numerical), but the connection must be established. Although certain cases of division with fractions can be represented, it is not easy to represent even simple situations such as . There are a number of ways to make sense of fraction division but bringing in the inverse relationship between multiplication and division can make it very easy. Any explanation must make sense not only of the simple example just given, but also of examples such as . Proportions After establishing control over the rational numbers in elementary school, middle school includes a lot of work with rates, ratios, percents and proportions. Proportions are reviewed with the focus on student’s understanding well enough to work problems, as described in: 7.2.B Solve single- and multi-step problems involving proportional relationships and verify the solutions. To begin, students need to understand what ratios, rates and proportions are and see a good development of representations for proportions. Cross products always show up here and this technique should be justified. Washington would like to see multi-step problems. Proportions can be put at the center of a number of mathematical connections, as is the case in the standard: 7.2.F Determine the slope of a line corresponding to the graph of a proportional relationship and relate the slope to similar triangles. This means the proportion needs to be represented as linear graphs through the origin. To do this requires an understanding of slope. Slope is shown to be well defined when it is connected to the proportionality of similar triangles. This is required in order to show that a proportionality equation is a straight line. This standard connects much of middle school mathematics. 47 Curriculum Review 48 OSPI Rubric ST Rubric 2 Fully Met • The standard is fully developed at the expected grade level. A typical student would be able to achieve mastery with the available content. 3 All content and key teaching/ learning tools present • The content from the standard is fully present at the correct grade level. • There is adequate information about the content and sufficient teaching/learning ideas included in the program to ensure that students develop conceptual understanding and procedural skill. • There is sufficient practice to ensure mastery. A typical student would be able to achieve mastery with the core program materials. 1 Partially Met • The standard is fully developed at +/- one grade level from the expected grade. • 50% or more of the standard is met at grade level, but some aspect of the standard is not present. The standard is fully developed, but limited in practice or reinforcement. 2 All or most content present, but key teaching/learning tools are missing • The key content from the standard exists in the program at the correct grade level. • The core materials need supplementation to do such things as adding additional opportunities for practice or finding more representations to help students consolidate learning. Many students would achieve mastery with the core program material. 1 Missing important content • Some significant aspect of the content is not present. - Some of the content may be completely absent. - Some of the content may be less rigorous. - It would take significant time and knowledge to fill the content gaps in the program. A typical student would not achieve mastery with the core program materials. 0 Not Met • The standard is not covered at grade level or +/- one grade level. • The standard is briefly mentioned, but a typical student would not be able to achieve mastery with the available content. • The standard is partially met at +/- one grade level. 0 Little or no content • All or most of the content in the standard is missing in the program at the correct grade level. - It may be completely absent - It may be briefly mentioned, but it is not developed. - It may contain less sophisticated precursor content that would led to the content in the standard A typical student would not be able to achieve mastery with the core program materials. Appendix B Curriculum Review 49 Appendix C Relationship Between OSPI and ST Scores ST looked at standards that had been scored “one” by OSPI to see how we scored those standards. To make some sense of this, only those standards to which OSPI gave two “one’s” were considered. Put another way, ST looked at all possible combinations of standards scores that include at least two “one’s.” The score types are: 0-0-1-1 0-1-1-1 1-1-1-1/1-1-0-2 1-1-1-2 1-1-2-2 The possible combinations could have a range of between two and six points given by OSPI reviewers. Scores1-1-1-1 and 1-1-0-2 are considered together because a total of four points is awarded. The ST total scores that mapped to these standards were then averaged. The ST scores ranged from zero, when both reviewers judged there to be no match to six, when both ST reviewers gave the full three points. The following chart shows a clear relationship. Two things became apparent. First, even for those standards than received at least two “one” scores, many more standards received higher scores than lower scores. OSPI gave forty-five standards a total of two or three points compared to eighty-three standards that received a total of five or six point. Second, standards that received a low total score from OSPI tended to receive a low total score from ST. Standards that receive a high total score from OSPI, tended to receive a high total score from ST. The combination mean that such standards could have a range of between two and six points given by OSPI reviewers. Scores1-1-1-1 and 1-1-0-2 are considered together because a total of four points is awarded. The ST total scores that mapped to these standards were then averaged. The ST scores ranged from zero, when both reviewers judged there to be no match to six, when both ST reviewers gave the full 3 points. The following chart shows a clear relationship. Standards that received a low total score from OSPI tended to receive a low total score from ST. Standards that receive a high total score from OSPI, tended to receive a high total score from ST.</p>
<div></div>
</p>