“Seeing Through Photographs” — Evaluation of MoMA’s online course

“Seeing Through Photographs” — Evaluation of MoMA’s online course

written by Gloriana Amador and Marta Mikłaszewicz

This post is a summary of the research project conducted at Pratt Institute in the spring semester of 2019 as part of the INFO 685 course: Digital Analytics: Web, Mobile and Social Media in collaboration with The Museum of Modern Art.

The Museum of Modern Art (MoMA) is a world famous institution that offers access and education to modern and contemporary artworks and related resources. MoMA supports public involvement in the arts through their educational online courses on Coursera. The courses cover a range of topics from fashion to photography.

This blog post summarizes our analysis of the “Seeing Through Photographs” course digital analytics in order to provide valuable suggestions to increase course engagement and determine any structural changes needed. Our findings and suggestions are categorised by language, course objectives, accessibility, quiz & assignment structure and community support.

The Research Origin

“Seeing Through Photographs” course has the highest enrollment of MoMA’s courses on Coursera. However, the research origin lie in the completion rates, as “Seeing Through Photographs” had around 154,443 participants commence the course but only 4,767 complete it. This graphic shows the funnel contrasts between the total number of students enrolled in the course, and how many began and completed the course. While performing the Literature Review, we learned that the high drop-out rates are considered typical/normal for MOOC courses (Chuang, Isaac and Ho, Andrew, HarvardX and MITx, 2016).

Methodology

The report was based on the quantitative and qualitative analysis of the data concerning “Seeing Through Photographs” course delivered by MoMA via Coursera performed by 5 Pratt researchers: Shradha Shree, Gloriana Amador, JiSun Hong, Arielle Bakoss and Marta Mikłaszewicz. To ensure the validity of the findings, we used triangulation of the data.

While approaching this topic, we quickly realised that it was necessary to establish the focus of the project, research objectives and scope of the analysed data. On 29th of March 2019, we met with Kelly Cannon, the Associate Educator for Interpretation, Research and Digital Learning in the Department of Education at MoMA. During this meeting we agreed to focus on the “Seeing Through Photographs” course and provide recommendations aiming to improve its completion rate, reduce the drop-out rate, and increase learners’ engagement. The MoMA representative agreed to provide approved data retrieved from Coursera.*

Data retrieved from Coursera

Coursera offers a dashboard that visualises the general tendencies of course performance. We accessed dashboard visuals related to the Course’s aggregate demographics, enrollment rate, completion data, drop-off points and engagement rates. This allowed us to understand the data and identify potential areas of interest.

We decided to focus on the Course feedback file as it recorded 2178 comments received from learners throughout the duration of the course. These comments were divided between the 5 researchers and each researcher annotated her part of the data using 4 different labels:

• Week (Week 1, Week 2; Week 3, Week 4, Week 5, Week 6, General);
• Language;
• Sentiment (Positive, Negative, Neutral);
• Topic (Quiz, Peer Review, Readings, Videos, Photos, Instructors, Interaction, Course)

Comments in a language different than English were translated using Google Translate.

The Post-survey questionnaire was administered by MoMA, however, only 14 learners responded to it. Their quantitative responses were visualized in graphs and the qualitative data analyzed by the researchers. The main themes were noted down.

The Public reviews are available on the Course page. The researchers analyzed the top 30 reviews from persons who completed this course (with 5 star rating) and 18 least favorable reviews from persons who completed this course (1, 2 and 3 stars). Main points were noted down.

Literature Review

To understand the topic and MOOCs (Massive Open Online Course), we conducted a Literature Review.

Competitive Review (Class Central)

On Class Central, we looked into the comments for 3 courses with topics related to the “Seeing Through Photographs” course:

• “Cameras, Exposure, and Photography” by Michigan State University via Coursera;
• “The Art of Photography” via Open2Study;
• “Photography: A Victorian Sensation” by University of Edinburgh via Coursera.

Findings & Recommendations

It is important to emphasise that more than 80% of learners’ comments were positive, 14.46% were neutral and only 3% were evaluated as negative by the researchers. Learners praised the exceptional reading material and in-depth interviews. They were impressed with the course content and the professional insights.

In our analysis, we looked more into comments that allowed us to define areas of potential improvement, but it is crucial to note, that the “Seeing Through Photographs” course has exceptional ratings and learner feedback.

In our analysis, we listened to course learners and provided 5 findings and related recommendations based on their feedback.

Finding 1: Learners look for content in their own language

“Seeing Through Photographs” is a truly global experience. Its learners come from different parts of the world and speak many languages, such as Spanish, French or Portuguese.

In their comments, learners asked for high-quality video subtitles in their own languages and suggested providing translations of the readings.

Recommendation 1: Highlight Coursera’s Subtitle Feature

Coursera already has a video subtitle feature, however it appears that it is often missed and many participants are not aware of it. Therefore, we proposed adding a point of reference for participants in the introductory text or prior to the course commencing that directs participants to the subtitle options available.

“Es un curso hasta el momento muy bueno, organizado, pero para personas de habla hispana puede resultar un poco complejo, al no contar con subtitulos”

Translation: “It is a very good course, organized, but for Spanish speakers it can be a bit complex, as it does not have subtitles.”

Finding 2: Course title and purpose are not clear

While reviewing comments, it appeared that learners were confused about the purpose of the course. They often believed that it will introduce them to the history of photography or teach them how to improve their practical photography skills.

Recommendation 2: Clarify course description & course objectives in the course introduction

The researchers propose reformulating the course title to touch on critical thinking, history and theory. Additionally, we have suggested rewriting the existing course objectives in the introduction in more plainspoken language, which would be more accessible for non-native English speakers.

“I gained a broader understanding of the history of American photography and some of the forces that has shaped it. Not a new skill.”

Finding 3: Increase course accessibility

While learners appreciated the quality of course content and MoMA’s professionalism, they asked to increase the course accessibility. The transcript used in videos was deemed to be too small and difficult to follow. The text used in the scanned materials was too academic and scans themselves were difficult to read.

Recommendation 3: Rescan Readings

Unfortunately, MoMA cannot change the size of video transcripts (they can share this feedback with Coursera), but can improve the quality of the PDFs and upload readings with the same orientation and, if possible, larger font.

“Better scans, some of them were difficult to read, pdfs should be formatted to fit a single page in full screen. It’s almost impossible to read from a cell phone.”

Finding 4: Assignments & quizzes are not serving their function

In this finding, the assignments and quizzes are not aligned with course objectives. This means that, for instance, the questions in the quizzes are detailed and fact based, and the assignments were extensive and difficult to follow by most of the students.

Recommendation 4: Refocus quizzes and assignments

It is recommended to shorten assignments as it allows learners to stay focused and will enhance this section of the course. Additionally, the quiz questions should encourage more critical thinking rather than simply checking knowledge of the facts. For instance, instead of asking the students in what year did x occur? MoMA can ask, why was the notion of x influential in photography today?

Furthermore, clearly aligning assessments and quiz content with learning objectives and module content will improve the engagement of students with the course modules. Additionally, providing prompts to be done offline can increase the interactions between learners. E.g. Take their own photos outdoors or research inspiring figures in this craft.

Finding 5: Community building through course content

This finding is related to language as an issue for the students. Missing support in languages other than English is something that the students commented on. In this sense, learners would like to be a part of a learning community in which they can enhance their knowledge of the topics in their own language.

In the post-course questionnaire, learners requested more participation from instructors and artists in the discussion forums; as well as more ways to stay involved in the course with live-streamed question and answer sessions, for example.

Recommendation 5: Connect and Expand “Seeing Through Photographs” Community

New course offerings that further develop skills learners gained in “Seeing Through Photographs” and/or provide a chance to learn practical photography skills will improve the way in which learners engage more with the online courses.

Additionally, opening forum moderator roles to course alumni and past students can enhance the levels of interactivity within the community of students.

Finally, MoMA can increase the presence of “Seeing through Photographs” via social media, by promoting use of the hashtag #seeingthroughphotographs. As MoMA already has a course-dedicated Instagram page, they can also expand and create a Flickr or Facebook group.

Reflection

The Pratt researchers met in person the course instructor, Sarah Meister, Curator in MoMA’s Department of Photography. Meister explained that this course was created to teach learners how to look beyond the photographs and learn critical looking and thinking. While analyzing learners’ feedback it is clear that this objective has been achieved. As one of the learners said:

Because it is about understanding photographs, not about taking photographs.

The learners appreciated the opportunity to learn both the critical thinking and the history of photography and were inspired by the quality of materials and the professionalism of the course. In this report, we have listed the learners’ voices and provided recommendations. We believe that these recommendations will increase the learners’ satisfaction and completion rate of the course.

Our work has been presented to MoMA in a form of a report. It is important to emphasise that it would never be written without great collaboration with MoMA.

*Note: This project was completed in compliance with Coursera’s Data Sharing and Research Policies.

---

“Seeing Through Photographs” — Evaluation of MoMA’s online course was originally published in Museums and Digital Culture – Pratt Institute on Medium, where people are continuing the conversation by highlighting and responding to this story.