Course Project
Your course project can take one of two forms:
-
Practice (preferred): An implementation of RL in a domain of
your choice---ideally one that you are using for research or in
another class. In this case, please describe the domain and
your initial plans for implementing learning. What
will the states and actions be? What algorithm(s) do you expect
to be most effective?
Example: Implement a reinforcement learning agent to manage a greenhouse irrigation system. The state could include soil moisture levels, temperature, humidity, and sunlight exposure; the actions could be turning irrigation on or off at different intensities; and the reward could balance crop growth, water efficiency, and energy cost. You might compare tabular Q-learning with Deep Deterministic Policy Gradients (DDPG) to evaluate which achieves more stable performance. -
Theory: A proposal, implementation, and testing of an algorithmic
modification or a new type of theoretical analysis for an RL
algorithm presented in class. In this case,
please describe the modification you propose to investigate and
the type of domain (possibly a toy domain) where it is likely to
show an improvement over the approaches discussed in the book or papers.
Example 1: Propose a modification to the policy gradient algorithm by adding an adaptive learning rate schedule that depends on variance in the estimated returns. Test your modified algorithm on the CartPole environment to see whether it converges faster than the standard REINFORCE algorithm.
Example 2: Conduct a theoretical analysis of Q-learning by studying its convergence properties under different exploration strategies (e.g., ε-greedy vs. softmax action selection). Provide formal arguments or proofs about how exploration choice affects sample complexity and long-term convergence, and support your claims with small-scale experiments on a toy environment such as GridWorld.
You may build on chapters or research papers you have read (or will read) in class; you may reimplement something you've found interesting from others' work; you can try something entirely new; you may write new code from scratch or modify existing code. It's up to you!
Our lab (Pi-Star) also has a few high-impact project ideas that you may consider for your course project. If you are interested, please review the project descriptions and attend the TA's office hours.
You are required to work in teams of two and are strongly encouraged to collaborate on all aspects together (i.e., pair programming rather than divide-and-conquer). Teams should submit only one copy of the project (only one team member should upload each file). However, each student must independently submit a short summary describing their individual contributions to the final product.
You may build on existing work and use existing code (your own or code found online), but you must give proper attribution and clearly identify your new contribution. Any unattributed or uncited work will be considered a breach of academic honesty and handled according to the course policy in the syllabus. Furthermore, you may not claim your own existing work as a new contribution. You may extend your own prior work, but it must be cited as such and include new contributions for this class project.
Submission
Submit your report through Canvas on or before the specified deadline. Include the full name and UIN of each team member. Submit only one copy of the report (submitted by one of the two team members).
Project Proposal due on Monday, October 6 at 11:59pm. The proposal structure should be as follows:
- Team members
- Proposed starting point. That is, what code base will you be starting with? Is there anything you need us to provide you with that you don't already have? (No guarantees you'll get what you ask for, but we'll try)
- What you plan to do and how you plan to do it?
The proposal should be written with the goal of convincing us that what you are proposing to do is interesting and non-trivial (though not necessarily completely original - see below).
It is completely legitimate to propose to do something based on something you read about provided that you are going to do the coding yourself. Just make sure to acknowledge any ideas (and code) that you borrow and be sure to clearly identify what you are going to do.
We encourage you to look ahead to topics that will be covered later in the course that may interest you, or to focus on a topic of interest that will not be covered in this course.
Be as specific as you can at this point. The more specific you are, the more detailed feedback you will get. For example, if you are doing an "applications" project:
- In what sense is your problem sequential?
- What is your problem's state space?
- What is your problem's action space?
- What is your problem's transition function?
- What reward function will you use?
- What is the simplest possible first result that you will try to get? What RL algorithm will you use? What will be the baseline you compare against?
- What will be the stretch goal for your project?
Literature Survey due on Monday, November 10 at 11:59pm.
- Submit a literature survey of the work most closely related to your project.
- It should begin with a summary of your current plans for the final project.
- If nothing has changed since the proposal, you may use the same text.
- If you have changed your plans in some way and would like further feedback, please make it very clear by placing the changes in bold or in a clearly marked separate section.
- The survey should include at least 10 references, some of which can be from the class readings. For each, you should discuss how it differs from or is similar to the work you plan to turn in for your final project.
- A good survey discusses each of the references at a technical level - not just what is done, but also how.
- Place the references at the end, with full reference information (authors, title, date, publication venue, etc.).
- We suggest writing this as if it were a section of a research paper, so that you can then use it directly in your final report.
Your final report is due on Monday, December 8 at 11:59pm. Your final submission should include:
- Source code, executable and README . We recommend you using a github repository to hold the source code, executable and a README file that provides a brief guide to run your code. In this case, you just need to provide the github link within your final report. If you want to keep your repository private for any reason, please zip your project folder including the source code, executable and the README file and attach to the final submission.
- A 5-minute mp4 YouTube video summarizing your project and the main results (make sure to choose unlist option so that only people with the URL can access your video). The URL should be included in your final report PDF, at the beginning of your report, along with your github link.
- A detailed written report describing your project, including its merits, and its deficiencies. As much as possible, you should relate your approach to the readings from throughout the course. View this report as a term paper. It is in place of a final exam and will be a large factor in your final grade for the project and for the course. The report should be roughly in the style of a conference paper, including introduction, motivation, related work, etc. All writing should be your own -- all quotes must be clearly attributed.
- Recall the points from the proposal and literature survey above. In particular, for applications projects be very clear about how you model your problem, and in what sense it's sequential.
- Include at least 10 citations with full bibliographic references to acknowledge where your ideas came from.
- Be very clear about what code you've used from other sources, if any. Clear citations are essential. Failure to credit ideas and code from external sources is cheating.
- Make sure you evaluate both the good and bad points of your approach.
- Show results of at least one experiment evaluating some aspect of or your entire approach, preferably showing error bars or some sort of statistical measure of the significance. Even if you didn't accomplish your goal, evaluate what you did do.
- A single well-analyzed experiment in a simple domain that compares clearly against a baseline is preferable to a shallow set of experiments across many domains.
- If any parameters are mentioned in the report, be sure to mention how you arrived at their values. Was it the first thing you tried? Trial and error? Roughly how many trials? etc.
- Remember to proofread and spell-check!
- Each team member should individually (and privately) identify what was your role in the overall project, and what was your partner's role. If everything was done together, a short statement to that effect is sufficient. If you feel that your partner has not contributed adequately, this is the opportunity to let us know. Please submit a single PDF to Canvas. This time, each team member should individually upload this file. Do not forget to include the project title, your name and UIN in this file.
The project instructions are based of text by Dr. Peter Stone from UT-Austin (with his permission).