Building a Big Data backend system for a classroom survey provider

A startup provider of classroom survey assessments hired us to build a platform for ingesting and analyzing survey data and converting it to quality feedback for its clientele of teachers and school administrators. Prior to working with us the company had been conducting and scoring the surveys by hand. The process of tabulating and analyzing the data was performed by team members, and the final results data was delivered to clients in the form of Excel Workbooks or Comma Separated Values tables. Such a system, in which employees were required for the entire survey process from start to finish, imposed obvious limitations on the company. Growth, in the form of more surveys for an expanded clientele, would have been accompanied by steep and endlessly increasing labor costs.

We built a backend system that quickly processes survey data and generates detailed reports for teachers, classes, schools and school districts. Our work enabled the client to transform themselves from being a consulting company to being a Software as a Service provider (SaaS). Surveys are now conducted via an automated system that has allowed the company to establish itself as the leader in the K-12 classroom survey field.

The Challenge

The survey process is highly data-driven. We not only faced large amounts of data but also a large amount of variability in the data. Different surveys involve different questions, the responses of which are aggregated differently. Furthermore, different scoring logic and parameters are applied to different school districts and even to different schools and classes within those districts. Survey response data also contains hierarchies that reflect the strength of the answer to each question (ex. strongly agree, agree, somewhat agree, etc.), and the system must be able to query the data across these hierarchies.

What We Delivered

The system we developed addressed all the challenges listed above and gave the company the ability to generate web-based reports of classes, teachers, schools and districts. Surveys are no longer conducted and scored manually, but given via the company’s website. Multiple versions of surveys and scoring parameters are available for different schools and districts, and custom scoring logic is also available for different groups and entities.

Our work allowed this client to grow their business exponentially. The automated backend system that we designed and implemented gave the company the capacity to process 7.43 million responses from 136,793 students in 7,588 classrooms for Hawaii’s Public Schools, and they have continued to expand since then. The company’s growth reinforces the quality of its surveys, as the pool of response data is the basis for the norms against which new survey results are measured. As of this writing, all surveys are calibrated using the collected data from over 125,000 classrooms. The company has gone on to become the leading provider of classroom surveys in the United States.

Find out how we can help you with your next project