Speech Recognition and Understanding

Logistics

• All lectures are in person. We also use Zoom for recording (only recording, the link is not shared). An unedited recording of each lecture is posted right after class.
• We do not use Canvas. Slides, recordings, and course materials are released first on Piazza; assignments are released on Piazza and submitted on Gradescope.
• Course staff generally do not respond to questions during the weekend — weekends are for resting and chilling.

Computing Resources

The course provides the compute you need for the hands-on work. Each student receives individual AWS credits, which you can use for coding assignment 4 and the term project. In addition, the class has shared access to a national supercomputing cluster (NCSA Delta), and we will hold a tutorial on how to use it. Because the cluster is shared with the whole class, please use it considerately — your misuse affects all other students. The cluster is intended primarily for the term project: the coding assignments should be done with your AWS credits, and we can only provide limited support for cluster issues outside the term project. For the project itself, the cluster is sufficient — even for a follow-up study toward a publication.

Cluster access is funded by an NSF ACCESS Discover allocation awarded to the course (award CIS261832, CMU 11751/18781: Speech Recognition and Understanding, 2026 Fall, 750,000 ACCESS credits). If your term project leads to a publication, please acknowledge the use of ACCESS and NCSA Delta resources in the paper.

Learning Objectives

• Explain the mathematical formulations underlying modern speech recognition: hidden Markov model (HMM)-based, connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), and attention-based end-to-end models.
• Implement core algorithms — feature extraction, HMMs, and end-to-end neural architectures — in Python.
• Build, train, and evaluate state-of-the-art speech recognition systems, including data preparation and training/inference tuning.
• Design and conduct an original group research project on a core speech recognition problem, communicating results through written reports and oral presentations.

Schedule

Tentative — check Piazza for updates. Deadlines are at 11:59 pm ET.

Staff

Instructors

• Samuele Cornell — scornell [at] andrew.cmu.edu · office hours Tue 3–4 pm · GHC 5417
• Jee-weon Jung — jeeweonj [at] andrew.cmu.edu · office hours Tue 3–4 pm · GHC 5417

Teaching Assistants

• Jinchuan Tian — jinchuat [at] andrew.cmu.edu · office hours Thu 11 am–12 pm · GHC 5417
• Chin-Jou Li — chinjoul [at] andrew.cmu.edu · office hours Thu 1–2 pm · GHC 5417
• Masao Someki — msomeki [at] andrew.cmu.edu · office hours Tue 2–3 pm · GHC 5417
• Shikhar Bharadwaj — sbharad2 [at] andrew.cmu.edu · office hours Wed 10–11 am · GHC 5417

Grading

Your grade has four components: class participation (25%), assignments (30%), a mid-term exam (20%), and the term project (25%). All points are fractional, not all-or-nothing, and the same is true for every component below.

Class participation — 25%

During each lecture we release a short quiz on Piazza, and submitting that quiz is what counts as participation — there is no separate attendance sheet. This means that attending without submitting the quiz earns no point; if that ever happens to you (you were in class but could not submit), tell the TAs or me as soon as possible and we will fix it. A quiz with no answer earns 0%, a correct answer 100%, and an incorrect answer 90% — so a good-faith attempt is worth nearly as much as a right answer. The quiz generally closes right after the question is asked.

Each regular lecture has a short quiz worth one point, and participation is capped at 20 points. We plan 22 quiz sessions over the semester, which gives you about two points of buffer — you can miss two lectures and still earn full participation points. Because that buffer exists, there are no extensions on participation points — though in truly exceptional circumstances (hospitalization, a family emergency), talk to me and we will work something out. An unedited recording of every lecture is posted right after class so you can catch up on the content of anything you miss. The two ESPnet tutorial sessions are worth 5 additional points. In the last two years, most students have recovered from their absences based on these buffers — so please don't worry too much about having to miss a class.

Assignments — 30%

There are two kinds of assignments, and both must be completed individually. Weekly assignments (about 10, worth 1 point each) are released every Wednesday and due the following Wednesday at 3:30 pm, before class starts. They involve working through equations and submitting a PDF, or implementing a simple algorithm in Python and submitting the results and code. If we end up releasing fewer than 10, we will normalize so the category is still worth 10 points. Coding assignments (4, worth 5 points each) go deeper: feature extraction, hidden Markov models, CTC loss plus an N-gram or RNN-transducer, and end-to-end ASR. All assignments are released on Piazza and submitted on Gradescope.

Mid-term exam — 20%

The mid-term is held in class on Wednesday, October 28 (3:30–4:50 pm). It is a written exam taken in class, and it is closed book: you may not use a cheat sheet or a calculator, and you may not discuss the exam with others while taking it. The good news is that about 90% of the questions are based directly on the weekly assignments and in-class quizzes, so if you keep up with those, you are already preparing for the exam. The exam is graded out of 20 with bonus points available, and many students recover points they missed elsewhere here.

Term project — 25%

An original group research project on a core speech recognition problem — it has its own section below.

Late work

This course runs most smoothly when everyone submits on time: because assignments build on one another, late work tends to snowball — for you and for the teaching staff. The late-day system described below exists precisely so that ordinary disruptions can be absorbed without anyone having to ask for an extension.

Unless stated otherwise, deadlines are at 11:59 pm ET (the weekly assignments, due before class at 3:30 pm, are the main exception). If you cannot finish an assignment on time, submit what you have: partial submissions earn partial credit, and because several assignments build on one another, missed material can often be recovered at a later deadline.

Each student has a total of five late days (120 hours) to spend across all assignments, tracked hourly on Gradescope; no penalty applies while your allowance lasts. Once it is used up, an assignment submitted within five days of its deadline earns at most half of its score, and submissions after that are not accepted. Late days do not apply to participation points (the two bonus points are the buffer there), and they do not apply to the term project: project deadlines are hard, with no late submissions and no extensions (see the Term Project section). If something serious happens — hospitalization, a family emergency — contact me as early as you can, and we will handle it individually. If your situation calls for systematic accommodations rather than one-off fixes, the Office of Disability Resources can help set those up.

Letter grades and re-grading

Final letter grades are adjusted relative to the class's overall performance rather than set at fixed absolute thresholds. For example, if too many students score above 90, the thresholds for A+, A, and A− are raised; if too many students score below 70, the thresholds for B, B− and C are lowered. For that reason we cannot accept grade claims based on your absolute point total. Re-grading requests are welcome through Gradescope, normally within one week of a grade being released (shorter near the mid-term and final letter-grade deadlines). Please use them when something is genuinely wrong rather than routinely — that keeps the process fair for everyone.

Term Project

The term project is a group research project (teams of 2–4 people, formed through Piazza) worth 25% of your grade. The timeline: we release a list of suggested topics on September 25; teams and a project proposal (abstract, introduction with related work, and proposed method) are due October 9; a checkpoint adding your experimental setup and benchmark results is due November 13; the project event is on November 30; and the final report is due December 4. Project deadlines are hard deadlines: no late submissions are accepted and we cannot grant extensions for any project component. Because the project is group work and the final deadlines sit right against the end-of-semester grading timeline, there is simply no slack to give — please plan for this with your team from the start. We recommend picking one of the ESPnet-based topics the instructors prepare — and it is always a good idea to come discuss your project with us early.

Scope

The topic must involve core speech recognition and understanding. Applications of core speech recognition techniques to other domains — say, music transcription — are allowed, and are in fact very interesting to me. Segment-classification tasks alone (for example, pure emotion recognition) are not sufficient — but blending them with speech recognition and understanding is encouraged. Plain use of commercial APIs or off-the-shelf models (for example, just calling Whisper) is not allowed unless it comes with meaningful modification or innovation. Your project is expected to involve training or fine-tuning a model: projects that only prompt an existing model will be penalized, unless you have agreed on a specific reason with the instructors in advance. Multiple teams may work on the same topic, as long as their implementations and design choices differ.

What good projects look like

A great project can take several shapes: an original idea, a piece of good engineering, or a methodical scientific study — for example, a careful comparison of different techniques for a particular application. In every case we look for reproducible experiments with honest evaluation. You should be able to explain clearly your motivation, your design choices, and the strengths and weaknesses of what you built. So, you can use Claude and Codex, but make sure you are using also your brain! The best projects from past years have become workshop papers. To give you a flavor, past topics have included LLM-based error correction for speech recognition, adapters for low-resource multilingual ASR, new architectures such as Mamba and H-Nets for CTC-based recognition, audio question answering, and decoder-only ASR with very long context.

Policies

This course is run in the spirit of the School of Computer Science's reasonable person principle. We assume that everyone in the course is reasonable and acts in good faith, and we will treat you the same way.

Collaboration

Weekly and coding assignments must be completed individually. That said, discussing the problems with other students at a conceptual level — how to think about a formulation, how to attack a programming problem — is allowed and genuinely encouraged; this kind of discussion is how we learn to do research and solve problems together. Two conditions: the answers and code you submit must be entirely your own, and if you did collaborate conceptually, you must say so (just report who you discussed with). If part of a submission turns out not to be your individual work, your grade for all weekly and coding assignments will be reduced, and undisclosed collaboration is handled in compliance with CMU's policy on cheating and plagiarism.

One more rule, which applies to everything you submit and especially the term project: please do not copy and paste existing materials. Write the equations yourself (LaTeX, MathType, handwritten — whatever works, and feel free to re-define variables to your taste as long as you explain them), summarize tables yourself, and draw figures yourself. This is partly educational, partly ethical, and partly about copyright — and for what it is worth, we hold ourselves to the same standard when we prepare the lecture materials.

Generative AI

The rules differ between assignments and the term project, so here they are spelled out. For all assignments (weekly and coding): the use of generative AI tools — LLMs, chatbots, coding assistants — is not allowed. Not to create your answers, and not to polish or debug them either. These assignments are where you actually learn the material, and they lose their purpose if an AI does them for you. Each coding assignment includes an AI-usage statement that you must fill in. For the term project: you are encouraged to use AI tools for conducting the research itself — brainstorming ideas, surveying the literature, processing data, and coding. You may not use them to write your report from scratch. You are responsible for the report, so please check it carefully. Polishing a report with AI tools is fine.

Wherever AI use is allowed, AI-generated content must be acknowledged and cited, following the APA Style guidelines. And a practical warning: you are responsible for checking the validity of anything an AI tool produces — if inaccurate or invalid information ends up in your work, you may not earn full credit, no matter where it came from. Deviations from these rules are treated as violations of CMU's academic integrity policy.

Academic integrity

The CMU and ECE academic integrity policies — covering cheating, plagiarism, unauthorized assistance, and research misconduct — apply in full to this course. The short version is simple: do not present anyone else's work, human or machine, as your own. If you are ever unsure whether something is acceptable, ask me before submitting. Asking is always fine and will never be held against you.

Accommodations

If you have a disability and require accommodations, please talk to me as early in the semester as possible so everything is in place from the start. If you do not yet have approved accommodations, you can apply through the Disability Resources Online Portal; if you already have them, notify me through the Accommodations Management portal. These conversations stay between us, and the goal is simply that this course works as well for you as it does for everyone else. The Student Academic Success Center is also a great resource, accommodations or not.

Support from the Instructor

If something is getting in the way of your coursework — academic stress, difficult life events, anxiety, or depression — you are welcome to come talk to me. I will not judge you, and you do not owe me any explanation — but I may be able to help, or suggest someone who can.

Respect and inclusion

This course is about building machines that understand everyone who speaks — whatever their accent, language, or ability. We hold the humans in the room to the same standard. Students arrive here from many countries, backgrounds, and paths through life; that mix is one of the best things about a CMU classroom, and every one of you belongs in this course. We expect everyone — instructors and TAs very much included — to treat each other with fairness and respect, and if we ever fall short of that, or you see anyone in the course treated unfairly because of who they are, please tell us. You can also reach the Center for Student Diversity and Inclusion or report anonymously through Report-It.