Chicago Public Schools has been a national leader in computer science education since Mayor Rahm Emanuel launched the Computer Science for All (CS4All) initiative in December 2013.  Now - in partnership with The Graide Network - CPS leaders are working to strengthen and expand CS4All district-wide

Fast Facts

ChicagoPublicSchools
  • Location. Chicago Public Schools, the country’s 3rd-largest public school district
  • Population. 380,000 students; over 80% qualify for free or reduced price meals
  • Technology. Google Drive allowed for fast, easy, and secure file-sharing of hand-written student assessments
  • Funding. Largely funded through grants to support computer science education
  • Use case: Scoring benchmark exams for Exploring Computer Science
  • Recognition. Since 2013, several large urban districts have replicated a curriculum and professional development model similar to Chicago’s and the White House launched a national CS4All initiative in 2016

 

The Problem

Curriculum assessment, student performance analysis, and strategic decision-making at scale

In February 2016, the Chicago Board of Education approved the addition of computer science as a graduation requirement for all Chicago Public Schools (CPS) students beginning with the class of 2020, making CPS the first school district in the country to elevate computer science as a core requirement for high school. Currently, over 60 high schools offer at least one computer science course and all 106 district-run high schools in the district will have to offer computer science soon.

The mission of CS4All is to deliver an “innovative education program that provides equity, empowerment, and opportunities that maximize the innate potential of every student to transform and advance their community, nation, and world”, while also addressing the growing gap between technical job openings and computer science graduates in our country. A third aim of this ambitious program is to close the opportunity gap and ensure minorities, women, and children from low income families have the early exposure needed to spark an interest in computer science education and high-tech careers. 

Computer science education develops students’ computational and critical thinking skills and teaches them to create, not simply use, new technologies. CPS believes that “this fundamental knowledge is needed to prepare students for the 21st century, regardless of their ultimate field of study or occupation.” 

Implementing the CS4All program, however, is not without its challenges. One thorny problem is gathering the data to measure its success. This is a problem the National Science Foundation (among others) has rallied to fund through various grants

Brenda Darden Wilkerson, the Director of the CS4All initiative, summed up the challenge she and her team faced as they work to roll out the program district-wide:

“We need to closely measure student performance to validate the effectiveness of the curriculum and test various hypotheses around computer science education. This data is powerful for district leaders as we make decisions about the future of CS4All.”
— Brenda Darden Wilkerson, Director, Computer Science & Information Technology Ed., CPS
ECSlogo

The high school introductory class, Exploring Computer Science (ECS), has two key assessments, a pre-test in September and post-test in the spring. The exams feature open-ended responses that require students to demonstrate critical thinking and problem solving. This approach makes them rigorous, high-quality exams but difficult to grade. Compounding this issue, CPS has a shortage of qualified STEM teachers, like many districts across the country. There are more than 7,000 students enrolled in the ECS course but the district has fewer than 100 certified computer science teachers.

The pre- and post-tests for ECS are the first time there has been a consistent measure of student learning in ECS. Expanding the administration of these tests to all students lets us get a more complete picture of how our students are faring, helps test some of our hypotheses about CS education, and provides a district-wide baseline to build on in coming years.

- Dr. Andy Rasmussen, Computer Science Project Coordinator
at Chicago Public Schools

A further challenge is that the ECS pre- and post-tests are open-response exams with extended writing test items. They require students to think critically and analytically, measuring both their comprehension and problem-solving skills. The accompanying scoring guide is 47 pages in length, which makes scoring the exams exceptionally challenging and time consuming. To grade effectively, you need a highly-trained reader who can apply a complex, standards-based rubric and evaluate a student’s entire work process. 

“Given that CS is now a core subject area in Chicago, it is important that the district have a way to monitor the quality of implementations that are occurring through the district. Researchers at SRI International have worked closely with the ECS developers to create a series of written performance assessments to address this need. These performance tasks require grading by human scorers.”

- Dr. Steven McGee, President, The Learning Partnership &
Research Associate Professor, Northwestern University

Solution

Fast, high-quality programmatic support

The ECS pre- and post-tests are designed as important benchmark exams to assess student growth, evaluate the effectiveness of the curriculum, instruction, and course, and guide future programmatic decisions. They have the potential to serve as powerful tools for administrators, teachers, and students. 

During the summer of 2016, The Graide Network scored a subset of ECS pre- and post-tests. CS4All leaders quickly recognized the value of this on-demand grading support and robust student data, and in the fall of 2016, they hired The Graide Network to grade all of the exams across the district, or roughly 12,000 exams.

“Our goal with the research is to understand what the right mixture of accountability policies and teacher/school supports will provide the highest overall implementation quality across the entire district.”
— Dr. Steven McGee, President, The Learning Partnership & Research Associate Professor, Northwestern University

The Graide Network is an online platform connecting K-12 schools with remote teaching assistants to provide critical on-demand grading and feedback support. “Graiders” are highly qualified, vetted undergraduate and graduate students from colleges across the country who are aspiring teachers. They are assigned to student work based on subject and grade level expertise, and the network (hundreds deep and growing fast) is able to effectively support all K-12 subjects and grade levels. Graiders deliver fast, accurate scoring for even the most complex assessments. 

These assessments haven’t been used before, and one of our big concerns was having consistent scoring. The alternative would have been asking teachers to follow the detailed rubric on their own, creating an additional burden for them instead of providing a service.

- Dr. Andy Rasmussen, Computer Science Project Coordinator
at Chicago Public Schools

Implementation and Scope of Work

Quick results enable in-depth analysis and program evaluation

Andy and Neli organized thousands of exams.

Andy and Neli organized thousands of exams.

High school students across the district sat for the pre-test at the start of the 2016/17 school year.  Dr. Andy Rasmussen, the district’s Computer Science Project Coordinator, and Yaneli Briano, CS4All intern and senior at Benito Juarez High School, were responsible for collecting, scanning, anonymizing and digitally sharing the exams via Google drive. Roughly once every other week throughout November and December, Andy would share a batch of assignments (roughly 1,000 exams in each batch). 

Before receiving the first batch of exams, The Graide Network developed an in-depth training module to serve as a prerequisite for any computer science Graider who wanted to score ECS pre-tests. The Graiders who made it past the prerequisite exercise were provided with additional materials and coaching focused on calibration of the detailed (and lengthy) scoring guide. The Graiders scored mock assessment and discussed their variances via online chat facilitated by The Graide Network. The Graide Network team developed supplementary scoring materials to address gaps in the rubric discovered during training.

The researchers constructed a set of common assessments in order to gauge the reliability of the Graiders and create an index to scale the students’ scores.

“We used a common graded set as a means to model the variability in how each Graider interpreted the scoring rubrics. In scoring constructed response questions there is always natural variation in the severity or leniency with which each scorer interprets the rubric. We can use that information to make adjustments to individual student scores based on whether the scorer is more or less lenient relative to other scorers.”

- Dr. Steven McGee, President, The Learning Partnership &
Research Associate Professor, Northwestern University

Results

A valuable programmatic partner with great future potential

Thus far, Graiders have scored over 7,000 pre-tests with an exceptionally high degree of reliability. Turnaround times also exceeded expectations - scores were returned within five business days of receiving the scanned files. 

“I am excited about the future potential of this relationship. As The Graide Network builds new technology to support third-party grading at scale, there is the potential to enlist their help for additional computer science courses, for individual teacher support, interim assessment scoring, and incorporating feedback for students and teachers into the reporting process.”

- Brenda Darden Wilkerson, Director, Computer Science & Information Technology Ed., CPS

 

"I am impressed with the capability of the team at The Graide Network to take the written rubric that we provided and develop a scorer training process. Their process resulted in high levels of consistency within and between each of the Graiders. I am also pleased with the extent to which the online system and process evolves rapidly in response to feedback from users like us. They strive to make the process of transferring exams to them and receiving scores from them as seamless as possible."

- Dr. Steven McGee, President, The Learning Partnership &
Research Associate Professor, Northwestern University

We need data like this to be able to zoom out and see how our students are learning in the different content areas of CS, and to zoom in and see how individual schools and classrooms are doing relative to their peers around the district. This will give our initiative a better ability to provide support to our teachers and curriculum development where it is needed.

- Dr. Andy Rasmussen, Computer Science Project Coordinator
at Chicago Public Schools

An unexpected benefit

This project also did much for encouraging pre-service science teachers to pursue their computer science certification, increasing the pipeline of talent for districts like Chicago Public Schools. In addition to the opportunity for direct exposure to ECS student work, Graiders collaborated with one another during the norming process, learned about additional certification options, and received insights about the mission of the course directly from one of the lead developers of the curriculum, Gail Chapman. 
 

 
Sumaiya at her graduation from UIC. She hopes to be teaching high school English and computer science in the fall.

Sumaiya at her graduation from UIC. She hopes to be teaching high school English and computer science in the fall.

“My love for technology and teaching technology to middle and high school students was definitely reinforced while grading the ECS tests. The ECS curriculum covers concepts that I’m more than familiar with, so I am definitely looking into training and certification to teach this subject.”
— Sumaiya Qazi, Graider, University of Illinois at Chicago
 

In the end, CPS district leaders received the data needed to complete critical research and guide successful implementation, teachers and students received complete, timely, and normed scores on benchmark exams, and pre-service computer science teachers gained valuable, hands-on experience with the Exploring Computer Science curriculum - a win all around. 

To explore how The Graide Network can help your district’s computer science initiatives, schedule a consulting call.

 
 
 

What makes Exploring Computer Science so powerful?

Read a letter from our CEO.