Education
- Ph.D (2014)
- Computer Science, University of Minnesota.
- Advisers: John T. Riedl and Joseph A. Konstan
- B.S. (2007)
- Computer Engineering, Iowa State University.
Employment History
- 2023–present
- Assistant Professor, Dept. of Information Science, Drexel University
- PI/Lead, Impact, Novation, Effectiveness, and Responsibility of Technology for Information Access Lab (INERTIAL)
- 2022–2023
- Associate Professor, Dept. of Computer Science, Boise State University
- Co-director, People and Information Research Team (PIReT)
- 2016–2022
- Assistant Professor, Dept. of Computer Science, Boise State University
- Co-director, People and Information Research Team (PIReT)
- 2014–2016
- Assistant Professor, Dept. of Computer Science, Texas State University
- 2008–2014
- Graduate Research Assistant, GroupLens Research, University of Minnesota
- Su 2012, F 2013
- Instructor, Dept. of Computer Science, University of Minnesota
- Summer 2010
- Research Intern, Autodesk Research (Toronto)
- 2007–2008, 2011
- Teaching Assistant, Dept. of Computer Science, University of Minnesota
- 2005–2007
- Undergrad RA, Scalable Computing Laboratory, Iowa State University
Students
Current Ph.D. Students
- Samira Vaez Barenji (expected 2029)
- Sushobhan Parajuli (expected 2029)
Ph.D. Graduates
- Ngozi Ihemelandu (Ph.D. 2024, Boise State University; dissertation: Best Practices for Offline Evaluation for Top-N Recommendation: Candidate Set Sampling and Statistical Inference)
- Amifa Raj (Ph.D. 2023, Boise State University; dissertation: Fair Layouts in Information Access Systems: Provider-Side Group Fairness in Ranking Beyond Ranked Lists)
M.S. Graduates
- Srabanti Guha (M.S. 2023, Boise State University; project: Explaining Misallocated Exposure across Multiple Rankings)
- Carlos Segura Cerna (M.S. 2020, Boise State University; project: Recommendation Server for LensKit)
- Mucun Tian (M.S. 2019, Boise State University; thesis: Estimating Error and Bias of Offline Recommender System Evaluation Results)
- Vaibhav Mahant (M.S. 2016, Texas State University; thesis: Improving Top-N Evaluation of Recommender Systems)
- Sushma Channamsetty (M.S. 2016, Texas State University; thesis: Recommender Response to User Profile Diversity and Popularity Bias)
- Mohammed Imran R Kazi (M.S. 2016, Texas State University; thesis: Exploring Potentially Discriminatory Biases in Book Recommendation)
- Shuvabrata Saha (M.S. 2016, Texas State University; co-advised with Dr. Apan Qasem; thesis: A Multi-objective Autotuning Framework For The Java Virtual Machine)
Undergraduate Student Research
I have supported and mentored the following undergraduate research students: Christine Pinney (BSU, UGRA + REU), Liana Shiroma (Colby Coll., REU 2021), Stephen Randall (U. Pitt, REU 2021), Connor Wood (BSU, REU 2020 + UGRA), Ananda Montoly (Smith Coll., REU 2020), Sandra Ambriz (BSU, HERC + UGRA).
Funding key:
- UGRA: undergraduate research assistant hired from research funds
- REU: Research Experience for Undergraduates
- HERC: Higher Education Research Consortium
Research Funding
External Grants
- 2023–2025: NSF 22-32553: Collaborative Research: CCRI: New: A Research News Recommender Infrastructure with Live Users for Algorithm and Interface Experimentation ($1.4M; Drexel PI, my share $150K; Lead PI Joseph A. Konstan, UMN).
- 2018–2025: NSF 17-51278: CAREER: User-Based Simulation Methods for Quantifying Sources of Error and Bias in Recommender Systems ($514K incl. REU supplements; PI).
Internal Grants
- 2017: Boise State College of Education Civility Grant LITERATE: Locating Informational Texts for Engaging Readers And Teaching Equitably ($19K; co-PI; with PI Katherine Wright & co-PI Sole Pera).
- 2014: Texas State University Research Enhancement Program (competitive internal research grant) Temporal Analysis of Recommender Systems ($8K; PI).
Publications
Author formatting key: myself, advised student, other student; †presenter, §undergraduate student.
Citation counts from Google Scholar (total 5354, h-index 31).
◊ These publications have citations merged in Google Scholar; count is reported on the most most final version, such as the journal expansion of a conference article.
Journal Articles // 9
Jonathan Stray, Alon Halevy, Parisa Assar, Dylan Hadfield-Menell, Chloe Bakalar, Craig Boutilier, Amar Ashar, Lex Beattie, Michael Ekstrand, Claire Leibowicz, Connie Moon Sehat, Sara Johansen, Lianne Kerlin, David Vickrey, Spandana Singh, Sanne Vrijenhoek, Amy Zhang, McKane Andrus, Natali Helberger, Polina Proutskova, Tanushree Mitra, and Nina Vasan. 2024. Building Human Values into Recommender Systems: An Interdisciplinary Synthesis. Transactions on Recommender Systems 2(3) (June 5th, 2024; online November 12th, 2023), 20:1–57. DOI 10.1145/3632297. arXiv:2207.10192 [cs.IR]. Cited 63 times. Cited 43 times.
Michael D. Ekstrand, Ben Carterette, and Fernando Diaz. 2024. Distributionally-Informed Recommender System Evaluation. Transactions on Recommender Systems 2(1) (March 7th, 2024; online August 4th, 2023), 6:1–27. DOI 10.1145/3613455. arXiv:2309.05892 [cs.IR]. NSF PAR 10461937. Cited 16 times. Cited 9 times.
Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022. Fairness in Information Access Systems. Foundations and Trends® in Information Retrieval 16(1–2) (July 11th, 2022), 1–177. DOI 10.1561/1500000079. arXiv:2105.05779 [cs.IR]. NSF PAR 10347630. Impact factor: 8. Cited 184 times. Cited 84 times.
Michael D. Ekstrand and Daniel Kluver. 2021. Exploring Author Gender in Book Rating and Recommendation. User Modeling and User-Adapted Interaction 31(3) (February 4th, 2021), 377–420. DOI 10.1007/s11257-020-09284-2. arXiv:1808.07586v2. NSF PAR 10218853. Impact factor: 4.412. Cited 201 times (shared with RecSys18◊). Cited 107 times (shared with RecSys18◊).
Michael D. Ekstrand, Katherine Landau Wright, and Maria Soledad Pera. 2020. Enhancing Classroom Instruction with Online News. Aslib Journal of Information Management 72(5) (November 17th, 2020; online June 14th, 2020), 725–744. DOI 10.1108/AJIM-11-2019-0309. Impact factor: 1.903. Cited 19 times. Cited 12 times.
Michael D. Ekstrand and Michael Ludwig. 2016. Dependency Injection with Static Analysis and Context-Aware Policy. Journal of Object Technology 15(1) (February 1st, 2016), 1:1–31. DOI 10.5381/jot.2016.15.1.a1. Cited 16 times.
Joseph A. Konstan, J.D. Walker, D. Christopher Brooks, Keith Brown, and Michael D. Ekstrand. 2015. Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid MOOC. Transactions on Computer-Human Interaction 22(2) (April 1st, 2015). DOI 10.1145/2728171. Impact factor: 1.293. Cited 117 times (shared with L@S14◊). Cited 29 times.
Justin J. Levandoski, Michael D. Ekstrand, Michael J. Ludwig, Ahmad Eldawy, Mohamed F. Mokbel, and John T. Riedl. 2011. RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures. Proceedings of the VLDB Endowment 4(11) (August 1st, 2011), 911–920. Acceptance rate: 18%. Cited 22 times. Cited 9 times.
Michael D. Ekstrand, John T. Riedl, and Joseph A. Konstan. 2011. Collaborative Filtering Recommender Systems. Foundations and Trends® in Human-Computer Interaction 4(2) (February 1st, 2011), 81–173. DOI 10.1561/1100000009. Cited 1728 times. Cited 659 times.
Peer-Reviewed Conference Papers // 31
Andrés Ferraro, Michael D. Ekstrand, and Christine Bauer. 2024. It’s Not You, It’s Me: The Impact of Choice Models and Ranking Strategies on Gender Imbalance in Music Recommendation. Short paper in Proceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24). ACM. DOI 10.1145/3640457.3688163. arXiv:2409.03781 [cs.IR].
Ngozi Ihemelandu and Michael D. Ekstrand. 2024. Multiple Testing for IR and Recommendation System Experiments. Short paper in Proceedings of the 46th European Conference on Information Retrieval (ECIR ’24). Lecture Notes in Computer Science 14610:449–457. DOI 10.1007/978-3-031-56063-7_37. NSF PAR 10497108. Acceptance rate: 24.3%. Cited 1 time.
Michael D. Ekstrand, Lex Beattie, Maria Soledad Pera, and Henriette Cramer. 2024. Not Just Algorithms: Strategically Addressing Consumer Impacts in Information Retrieval. In Proceedings of the 46th European Conference on Information Retrieval (ECIR ’24, IR for Good track). Lecture Notes in Computer Science 14611:314–335. DOI 10.1007/978-3-031-56066-8_25. NSF PAR 10497110. Acceptance rate: 35.9%. Cited 6 times. Cited 3 times.
Amifa Raj and Michael D. Ekstrand. 2024. Towards Optimizing Ranking in Grid-Layout for Provider-side Fairness. In Proceedings of the 46th European Conference on Information Retrieval (ECIR ’24, IR for Good track). Lecture Notes in Computer Science 14612:90–105. DOI 10.1007/978-3-031-56069-9_7. NSF PAR 10497109. Acceptance rate: 35.9%. Cited 1 time. Cited 1 time.
Ngozi Ihemelandu and Michael D. Ekstrand. 2023. Candidate Set Sampling for Evaluating Top-N Recommendation. In Proceedings of the 22nd IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT ’23). pp. 88-94. DOI 10.1109/WI-IAT59888.2023.00018. arXiv:2309.11723 [cs.IR]. NSF PAR 10487293. Acceptance rate: 28%. Cited 3 times. Cited 1 time.
Amifa Raj, Bhaskar Mitra, Michael D. Ekstrand, and Nick Craswell. 2023. Patterns of Gender-Specializing Query Reformulation. Short paper in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23). DOI 10.1145/3539618.3592034. arXiv:2304.13129. NSF PAR 10423689. Acceptance rate: 25.1%. Cited 2 times. Cited 1 time.
Ngozi Ihemelandu and Michael D. Ekstrand. 2023. Inference at Scale: Significance Testing for Large Search and Recommendation Experiments. Short paper in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23). DOI 10.1145/3539618.3592004. arXiv:2305.02461. NSF PAR 10423691. Acceptance rate: 25.1%. Cited 2 times.
Christine Pinney, Amifa Raj, Alex Hanna, and Michael D. Ekstrand. 2023. Much Ado About Gender: Current Practices and Future Recommendations for Appropriate Gender-Aware Information Access. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (CHIIR ’23). DOI 10.1145/3576840.3578316. arXiv:2301.04780. NSF PAR 10423693. Acceptance rate: 39.4%. Cited 19 times. Cited 11 times.
Amifa Raj and Michael D. Ekstrand. 2022. Measuring Fairness in Ranked Results: An Analytical and Empirical Comparison. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22). pp. 726–736. DOI 10.1145/3477495.3532018. NSF PAR 10329880. Acceptance rate: 20%. Cited 59 times. Cited 44 times.
A. K. M. Nuhil Mehdy, Michael D. Ekstrand, Bart Knijnenburg, and Hoda Mehrpouyan. 2021. Privacy as a Planned Behavior: Effects of Situational Factors on Privacy Perceptions and Plans. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’21). ACM. DOI 10.1145/3450613.3456829. arXiv:2104.11847 [cs.SI]. NSF PAR 10223377. Acceptance rate: 23%. Cited 23 times. Cited 15 times.
Ömer Kırnap, Fernando Diaz, Asia J. Biega, Michael D. Ekstrand, Ben Carterette, and Emine Yılmaz. 2021. Estimation of Fair Ranking Metrics with Incomplete Judgments. In Proceedings of The Web Conference 2021 (TheWebConf 2021). ACM. DOI 10.1145/3442381.3450080. arXiv:2108.05152. NSF PAR 10237411. Acceptance rate: 21%. Cited 47 times. Cited 36 times.
Michael D. Ekstrand. 2020. LensKit for Python: Next-Generation Software for Recommender Systems Experiments. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM ’20, Resource track). ACM, pp. 2999–3006. DOI 10.1145/3340531.3412778. arXiv:1809.03125 [cs.IR]. NSF PAR 10199450. No acceptance rate reported. Cited 101 times. Cited 72 times.
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, and Ben Carterette. 2020. Evaluating Stochastic Rankings with Expected Exposure. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM ’20). ACM, pp. 275–284. DOI 10.1145/3340531.3411962. arXiv:2004.13157 [cs.IR]. NSF PAR 10199451. Acceptance rate: 20%. Nominated for Best Long Paper. Cited 187 times. Cited 166 times.
Mucun Tian and Michael D. Ekstrand. 2020. Estimating Error and Bias in Offline Evaluation Results. Short paper in Proceedings of the 2020 Conference on Human Information Interaction and Retrieval (CHIIR ’20). ACM, pp. 5. DOI 10.1145/3343413.3378004. arXiv:2001.09455 [cs.IR]. NSF PAR 10146883. Acceptance rate: 47%. Cited 11 times. Cited 10 times.
Michael D. Ekstrand, Mucun Tian, Mohammed R. Imran Kazi, Hoda Mehrpouyan, and Daniel Kluver. 2018. Exploring Author Gender in Book Rating and Recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18). ACM, pp. 242–250. DOI 10.1145/3240323.3240373. arXiv:1808.07586v1 [cs.IR]. Acceptance rate: 17.5%. Citations reported under UMUAI21◊. Citations reported under UMUAI21◊.
Michael D. Ekstrand, Rezvan Joshaghani, and Hoda Mehrpouyan. 2018. Privacy for All: Ensuring Fair and Equitable Privacy Protections. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAT* 2018). PMLR, Proceedings of Machine Learning Research 81:35–47. Acceptance rate: 24%. Cited 104 times. Cited 78 times.
Michael D. Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D. Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All The Cool Kids, How Do They Fit In?: Popularity and Demographic Biases in Recommender Evaluation and Effectiveness. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAT* 2018). PMLR, Proceedings of Machine Learning Research 81:172–186. Acceptance rate: 24%. Cited 290 times. Cited 212 times.
Michael D. Ekstrand and Vaibhav Mahant. 2017. Sturgeon and the Cool Kids: Problems with Random Decoys for Top-N Recommender Evaluation. In Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference (Recommender Systems track). AAAI, pp. 639–644. No acceptance rate reported. Cited 16 times. Cited 11 times.
Sushma Channamsetty and Michael D. Ekstrand. 2017. Recommender Response to Diversity and Popularity Bias in User Profiles. Short paper in Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference (Recommender Systems track). AAAI, pp. 657–660. No acceptance rate reported. Cited 21 times. Cited 19 times.
Michael D. Ekstrand and Martijn C. Willemsen. 2016. Behaviorism is Not Enough: Better Recommendations through Listening to Users. In Proceedings of the Tenth ACM Conference on Recommender Systems (RecSys ’16, Past, Present, and Future track). ACM. DOI 10.1145/2959100.2959179. Acceptance rate: 36%. Cited 142 times. Cited 95 times.
Michael D. Ekstrand, Daniel Kluver, F. Maxwell Harper, and Joseph A. Konstan. 2015. Letting Users Choose Recommender Algorithms: An Experimental Study. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys ’15). ACM. DOI 10.1145/2792838.2800195. Acceptance rate: 21%. Cited 139 times. Cited 100 times.
Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User Perception of Differences in Recommender Algorithms. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys ’14). ACM. DOI 10.1145/2645710.2645737. Acceptance rate: 23%. Cited 284 times. Cited 187 times.
Joseph A. Konstan, J.D. Walker, D. Christopher Brooks, Keith Brown, and Michael D. Ekstrand. 2014. Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid MOOC. In Proceedings of the First ACM Conference on Learning @ Scale (S ’14). ACM. DOI 10.1145/2556325.2566244. Acceptance rate: 37%. Citations reported under TOCHI15◊. Cited 77 times.
Tien T. Nguyen, Daniel Kluver, Ting-Yu Wang, Pik-Mai Hui, Michael D. Ekstrand, Martijn C. Willemsen, and John Riedl. 2013. Rating Support Interfaces to Improve User Experience and Recommender Accuracy. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys ’13). ACM. DOI 10.1145/2507157.2507188. Acceptance rate: 24%. Cited 60 times. Cited 42 times.
Michael Ekstrand and John Riedl. 2012. When Recommenders Fail: Predicting Recommender Failure for Algorithm Selection and Combination. Short paper in Proceedings of the Sixth ACM Conference on Recommender Systems (RecSys ’12). ACM, pp. 233–236. DOI 10.1145/2365952.2366002. Acceptance rate: 32%. Cited 88 times. Cited 73 times.
Daniel Kluver, Tien T. Nguyen, Michael Ekstrand, Shilad Sen, and John Riedl. 2012. How Many Bits per Rating?. In Proceedings of the Sixth ACM Conference on Recommender Systems (RecSys ’12). ACM, pp. 99–106. DOI 10.1145/2365952.2365974. Acceptance rate: 20%. Cited 48 times. Cited 38 times.
Justin J. Levandoski, Mohamed Sarwat, Mohamed F. Mokbel, and Michael D. Ekstrand. 2012. RecStore: An Extensible And Adaptive Framework for Online Recommender Queries Inside the Database Engine. In Proceedings of the 15th International Conference on Extending Database Technology (EDBT ’12). ACM, pp. 86–96. DOI 10.1145/2247596.2247608. Acceptance rate: 23%. Cited 19 times. Cited 16 times.
Michael D. Ekstrand, Michael Ludwig, Joseph A. Konstan, and John T. Riedl. 2011. Rethinking The Recommender Research Ecosystem: Reproducibility, Openness, and LensKit. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys ’11). ACM, pp. 133–140. DOI 10.1145/2043932.2043958. Acceptance rate: 27% (20% for oral presentation, which this received). Cited 255 times. Cited 195 times.
Michael Ekstrand, Wei Li, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2011. Searching for Software Learning Resources Using Application Context. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, pp. 195–204. DOI 10.1145/2047196.2047220. Acceptance rate: 25%. Cited 56 times. Cited 48 times.
Michael D. Ekstrand, Praveen Kannan, James A. Stempter, John T. Butler, Joseph A. Konstan, and John T. Riedl. 2010. Automatically Building Research Reading Lists. In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys ’10). ACM, pp. 159–166. DOI 10.1145/1864708.1864740. Acceptance rate: 19%. Cited 123 times. Cited 100 times.
Michael D. Ekstrand and John T. Riedl. 2009. rv you’re dumb: Identifying Discarded Work in Wiki Article History. In Proceedings of the 5th International Symposium on Wikis and Open Collaboration (WikiSym ’09). ACM, pp. 10. DOI 10.1145/1641309.1641317. Acceptance rate: 36%. Selected as Best Paper. Cited 37 times. Cited 28 times.
Book Chapters // 2
Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022. Fairness in Recommender Systems. In Recommender Systems Handbook (3rd edition). Francesco Ricci, Lior Roach, and Bracha Shapira, eds. Springer-Verlag. DOI 10.1007/978-1-0716-2197-4_18. ISBN 978-1-0716-2196-7. Cited 36 times. Cited 19 times.
Daniel Kluver, Michael D. Ekstrand, and Joseph A. Konstan. 2018. Rating-Based Collaborative Filtering: Algorithms and Evaluation. In Social Information Access. Peter Brusilovsky and Daqing He, eds. Springer-Verlag, Lecture Notes in Computer Science vol. 10100, pp. 344–390. DOI 10.1007/978-3-319-90092-6_10. ISBN 978-3-319-90091-9. Cited 146 times. Cited 100 times.
Workshops and Posters // 18
These papers have been peer-reviewed for workshops, poster proceedings, and similar venues.
Jacy Reese Anthis, Kristian Lum, Michael Ekstrand, Avi Feller, Alexander D’Amour, and Chenhao Tan. 2024. The Impossibility of Fair LLMs. In HEAL: Human-centered Evaluation and Auditing of Language Models, a non-archival workshop at CHI 2024. arXiv:2406.03198 [cs.CL]. Cited 5 times. Cited 4 times.
Amifa Raj and Michael D. Ekstrand. 2023. Towards Measuring Fairness in Grid Layout in Recommender Systems. Presented at the 6th FAccTrec Workshop on Responsible Recommendation at RecSys 2023 (peer-reviewed but not archived). arXiv:2309.10271 [cs.IR]. Cited 1 time.
Michael D. Ekstrand and Maria Soledad Pera. 2022. Matching Consumer Fairness Objectives & Strategies for RecSys. Presented at the 5th FAccTrec Workshop on Responsible Recommendation at RecSys 2022 (peer-reviewed but not archived). arXiv:2209.02662 [cs.IR]. Cited 6 times. Cited 4 times.
Amifa Raj and Michael D. Ekstrand. 2022. Fire Dragon and Unicorn Princess: Gender Stereotypes and Children’s Products in Search Engine Responses. In SIGIR eCom ’22. DOI 10.48550/arXiv.2206.13747. arXiv:2206.13747 [cs.IR]. Cited 11 times. Cited 5 times.
Lawrence Spear, Ashlee Milton, Garrett Allen, Amifa Raj, Michael Green, Michael D. Ekstrand, and Maria Soledad Pera. 2021. Baby Shark to Barracuda: Analyzing Children’s Music Listening Behavior. In RecSys 2021 Late-Breaking Results (RecSys ’21). DOI 10.1145/3460231.3478856. NSF PAR 10316668. Cited 8 times. Cited 3 times.
Ngozi Ihemelandu and Michael D. Ekstrand. 2021. Statistical Inference: The Missing Piece of RecSys Experiment Reliability Discourse. In Proceedings of the Perspectives on the Evaluation of Recommender Systems Workshop 2021 (RecSys ’21). arXiv:2109.06424 [cs.IR]. Cited 7 times. Cited 6 times.
Amifa Raj, Ashlee Milton, and Michael D. Ekstrand. 2021. Pink for Princesses, Blue for Superheroes: The Need to Examine Gender Stereotypes in Kids’ Products in Search and Recommendations. In Proceedings of the 5th International and Interdisciplinary Workshop on Children & Recommender Systems (KidRec ’21), at IDC 2021. arXiv:2105.09296. NSF PAR 10335669. Cited 9 times. Cited 5 times.
Amifa Raj, Connor Wood, Ananda Montoly, and Michael D. Ekstrand. 2020. Comparing Fair Ranking Metrics. Presented at the 3rd FAccTrec Workshop on Responsible Recommendation at RecSys 2020 (peer-reviewed but not archived). arXiv:2009.01311 [cs.IR]. Cited 37 times. Cited 29 times.
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, and Michael D. Ekstrand. 2019. Workshop on Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval (FACTS-IR). In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’19). ACM. DOI 10.1145/3331184.3331644. Cited 6 times.
Michael D. Ekstrand, Ion Madrazo Azpiazu, Katherine Landau Wright, and Maria Soledad Pera. 2018. Retrieving and Recommending for the Classroom: Stakeholders, Objectives, Resources, and Users. In Proceedings of the ComplexRec 2018 Second Workshop on Recommendation in Complex Scenarios (ComplexRec ’18), at RecSys 2018. Cited 8 times. Cited 3 times.
Mucun Tian and Michael D. Ekstrand. 2018. Monte Carlo Estimates of Evaluation Metric Error and Bias. Computer Science Faculty Publications and Presentations 148, Boise State University. Presented at the REVEAL 2018 Workshop on Offline Evaluation for Recommender Systems at RecSys 2018. DOI 10.18122/cs_facpubs/148/boisestate. NSF PAR 10074452. Cited 1 time. Cited 1 time.
Michael D. Ekstrand. 2018. The LKPY Package for Recommender Systems Experiments: Next-Generation Tools and Lessons Learned from the LensKit Project. Computer Science Faculty Publications and Presentations 147, Boise State University. Presented at the REVEAL 2018 Workshop on Offline Evaluation for Recommender Systems at RecSys 2018. DOI 10.18122/cs_facpubs/147/boisestate. arXiv:1809.03125v1 [cs.IR]. Cited 11 times. Cited 19 times.
Maria Soledad Pera, Katherine Wright, and Michael D. Ekstrand. 2018. Recommending Texts to Children with an Expert in the Loop. In Proceedings of the 2nd International Workshop on Children & Recommender Systems (KidRec ’18), at IDC 2018. DOI 10.18122/cs_facpubs/140/boisestate. Cited 7 times. Cited 6 times.
Rezvan Joshaghani, Michael D. Ekstrand, Bart Knijnenburg, and Hoda Mehrpouyan. 2018. Do Different Groups Have Comparable Privacy Tradeoffs?. In Moving Beyond a ‘One-Size Fits All’ Approach: Exploring Individual Differences in Privacy, a workshop at CHI 2018. NSF PAR 10222636. Cited 4 times. Cited 4 times.
Michael D. Ekstrand and Maria Soledad Pera. 2017. The Demographics of Cool: Popularity and Recommender Performance for Different Groups of Users. In RecSys 2017 Poster Proceedings. CEUR, Workshop Proceedings 1905. Cited 17 times. Cited 6 times.
Michael D. Ekstrand. 2017. Challenges in Evaluating Recommendations for Children. In Proceedings of the International Workshop on Children & Recommender Systems (KidRec), at RecSys 2017. Cited 11 times.
Jennifer D. Ekstrand and Michael D. Ekstrand. 2016. First Do No Harm: Considering and Minimizing Harm in Recommender Systems Designed for Engendering Health. In Proceedings of the Workshop on Recommender Systems for Health at RecSys ’16. Cited 16 times. Cited 11 times.
Michael D. Ekstrand. 2014. Building Open-Source Tools for Reproducible Research and Education. At Sharing, Re-use, and Circulation of Resources in Cooperative Scientific Work, a workshop at CSCW 2014.
Editorially-Reviewed Publications // 4
These articles have appeared in magazines and similar venues; they have typically undergone some from of editorial review, but usually not full peer review.
Michael D. Ekstrand, Maria Soledad Pera, and Katherine Landau Wright. 2023. Seeking Information with a ‘More Knowledgeable Other’. ACM Interactions 30(1) (January 11th, 2023), 70–73. DOI 10.1145/3573364. Cited 6 times. Cited 3 times.
Nasim Sonboli, Robin Burke, Michael Ekstrand, and Rishabh Mehrotra. 2022. The Multisided Complexity of Fairness in Recommender Systems. AI Magazine 43(2) (June 23rd, 2022), 164–176. DOI 10.1002/aaai.12054. NSF PAR 10334796. Cited 33 times. Cited 19 times.
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand, Adam Roegiest, Aldo Lipani, Alex Beutel, Ana Lucic, Ana-Andreea Stoica, Anubrata Das, Asia Biega, Bart Voorn, Claudia Hauff, Damiano Spina, David Lewis, Douglas W Oard, Emine Yilmaz, Faegheh Hasibi, Gabriella Kazai, Graham McDonald, Hinda Haned, Iadh Ounis, Ilse van der Linden, Joris Baan, Kamuela N Lau, Krisztian Balog, Mahmoud Sayed, Maria Panteli, Mark Sanderson, Matthew Lease, Preethi Lahoti, and Toshihiro Kamishima. 2019. FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval. SIGIR Forum 53(2) (December 12th, 2019), 20–43. DOI 10.1145/3458553.3458556. Cited 50 times. Cited 18 times.
Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, and Justin Zobel. 2018. The Dagstuhl Perspectives Workshop on Performance Modeling and Prediction. SIGIR Forum 52(1) (June 1st, 2018), 91–101. DOI 10.1145/3274784.3274789. Cited 17 times. Cited 17 times.
Tutorials // 3
Joseph A. Konstan, Robin Burke, and Michael D. Ekstrand. 2024. Conducting Recommender Systems User Studies Using POPROX. Tutorial presented at Proceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24).
Michael D. Ekstrand, Fernando Diaz, and Robin Burke. 2019. Fairness and Discrimination in Recommendation and Retrieval. Tutorial presented at Proceedings of the 13th ACM Conference on Recommender Systems (RecSys ’19). pp. 2. DOI 10.1145/3298689.3346964. Cited 48 times. Cited 37 times.
Michael D. Ekstrand, Fernando Diaz, and Robin Burke. 2019. Fairness and Discrimination in Retrieval and Recommendation. Tutorial presented at Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’19). pp. 2. DOI 10.1145/3331184.3331380. Cited 56 times. Cited 39 times.
Demos // 3
Tobias Vente, Michael Ekstrand, and Joeran Beel. 2023. Introducing LensKit-Auto, an Experimental Automated Recommender System (AutoRecSys) Toolkit. Demo recorded in Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). pp. 1212–1216. DOI 10.1145/3604915.3610656. Cited 11 times. Cited 8 times.
Ashlee Milton, Michael Green, Adam Keener, Joshua Ames, Michael D. Ekstrand, and Maria Soledad Pera. 2019. StoryTime: Eliciting Preferences from Children for Book Recommendations. Demo recorded in Proceedings of the 13th ACM Conference on Recommender Systems (RecSys ’19). pp. 2. DOI 10.1145/3298689.3347048. NSF PAR 10133610. Cited 17 times. Cited 8 times.
Michael D. Ekstrand, Michael Ludwig, Jack Kolb, and John T. Riedl. 2011. LensKit: A Modular Recommender Framework. Demo recorded in Proceedings of the 5th ACM Conference on Recommender Systems (RecSys ’11). ACM, pp. 349-350. DOI 10.1145/2043932.2044001. Cited 44 times. Cited 2 times.
Preprints and Reports // 5
Unreviewed preprints, technical reports, and similar manuscripts.
Alexandra Olteanu, Michael Ekstrand, Carlos Castillo, and Jina Suh. 2023. Responsible AI Research Needs Impact Statements Too. arXiv:2311.11776 [cs.AI]. Cited 8 times. Cited 7 times.
Amifa Raj and Michael D. Ekstrand. 2023. Unified Browsing Models for Linear and Grid Layouts. arXiv:2310.12524 [cs.IR]. Cited 1 time. Cited 1 time.
Michael D. Ekstrand. 2021. Multiversal Simulacra: Understanding Hypotheticals and Possible Worlds Through Simulation. arXiv:2110.00811 [cs.IR]. Cited 2 times. Cited 2 times.
Michael D. Ekstrand and Joseph A. Konstan. 2019. Recommender Systems Notation: Proposed Common Notation for Teaching and Research. Computer Science Faculty Publications and Presentations 177, Boise State University. DOI 10.18122/cs_facpubs/177/boisestate. arXiv:1902.01348 [cs.IR]. Cited 12 times. Cited 4 times.
Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, and Justin Zobel. 2018. From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442). Dagstuhl Manifestos 7(1) (November 21st, 2018), 96–139. DOI 10.4230/DagMan.7.1.96. Cited 20 times. Cited 16 times.
Workshop Summaries and Reports // 18
These are summaries for workshops and special issues I have co-organized, as well as outcome reports that aren't listed under another category.
Michael D. Ekstrand, Toshihiro Kamishima, Amifa Raj, and Karlijn Dinnissen. 2024. FAccTRec 2024: The 7th Workshop on Responsible Recommendation. Meeting summary in Proceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24). ACM.
Michael D. Ekstrand, Maria Soledad Pera, and Alan Said. 2024. AltRecSys: A Workshop on Alternative, Unexpected, and Critical Ideas on Recommendation. Meeting summary in Proceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24). ACM. DOI 10.1145/3640457.3687104.
Michael D. Ekstrand, Jean Garcia-Gathright, Nasim Sonboli, Amifa Raj, and Karlijn Dinnissen. 2023. FAccTRec 2023: The 6th Workshop on Responsible Recommendation. Meeting summary in Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). ACM. DOI 10.1145/3604915.3608761. Cited 1 time. Cited 1 time.
Michael D. Ekstrand, Graham McDonald, Amifa Raj, and Isaac Johnson. 2023. Overview of the TREC 2022 Fair Ranking Track. Meeting summary in The Thirty-First Text REtrieval Conference (TREC 2022) Proceedings (TREC 2022). arXiv:2302.05558. Cited 37 times. Cited 13 times.
Michael D. Ekstrand, Graham McDonald, Amifa Raj, and Isaac Johnson. 2022. Overview of the TREC 2021 Fair Ranking Track. Meeting summary in The Thirtieth Text REtrieval Conference (TREC 2021) Proceedings (TREC 2021). https://trec.nist.gov/pubs/trec30/papers/Overview-F.pdf
Michael D. Ekstrand, Pierre-Nicolas Schwab, Toshihiro Kamishima, and Nasim Sonboli. 2021. FAccTRec 2021: The 4th Workshop on Responsible Recommendation. Meeting summary in Proceedings of the 15th ACM Conference on Recommender Systems (RecSys ’21). ACM. DOI 10.1145/3460231.3470932. Cited 2 times. Cited 2 times.
Michael D. Ekstrand, Allison Chaney, Pablo Castells, Robin Burke, David Rohde, and Manel Slokom. 2021. SimuRec: Workshop on Synthetic Data and Simulation Methods for Recommender Systems Research. Meeting summary in Proceedings of the 15th ACM Conference on Recommender Systems (RecSys ’21). ACM. DOI 10.1145/3460231.3470938. Cited 19 times. Cited 15 times.
Robin Burke, Michael D. Ekstrand, Nava Tintarev, and Julita Vassileva. 2021. Preface to the Special Issue on Fair, Accountable, and Transparent Recommender Systems. User Modeling and User-Adapted Interaction 31(3) (July 24th, 2021), 371–375. DOI 10.1007/s11257-021-09297-5. Cited 10 times. Cited 7 times.
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sergey Feldman, and Sebastian Kohlmeier. 2021. Overview of the TREC 2020 Fair Ranking Track. Meeting summary in The Twenty-Ninth Text REtrieval Conference (TREC 2020) Proceedings (TREC 2020). arXiv:2108.05135. Cited 15 times. Cited 7 times.
Michael D. Ekstrand, Pierre-Nicolas Schwab, Jean Garcia-Gathright, Toshihiro Kamishima, and Nasim Sonboli. 2020. 3rd FATREC Workshop: Responsible Recommendation. Meeting summary in Proceedings of the 14th ACM Conference on Recommender Systems (RecSys ’20). ACM. DOI 10.1145/3383313.3411538. Cited 6 times. Cited 6 times.
Bamshad Mobasher, Stylani Kleanthous, Michael D. Ekstrand, Bettina Berendt, Janna Otterbacher, and Avital Schulner Tal. 2020. UMAP 2020 Fairness in User Modeling, Adaptation and Personalization (FairUMAP 2020) Chairs’ Welcome. Meeting summary in Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’20). ACM. DOI 10.1145/3386392.3399565.
Bamshad Mobasher, Stylani Kleanthous, Michael D. Ekstrand, Bettina Berendt, Janna Otterbacher, and Avital Schulner Tal. 2020. FairUMAP 2020: The 3rd Workshop on Fairness in User Modeling, Adaptation and Personalization. Meeting summary in Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’20). ACM. DOI 10.1145/3340631.3398671. Cited 5 times. Cited 2 times.
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, and Sebastian Kohlmeier. 2020. Overview of the TREC 2019 Fair Ranking Track. Meeting summary in The Twenty-Eighth Text REtrieval Conference (TREC 2019) Proceedings (TREC 2019). arXiv:2003.11650. Cited 46 times. Cited 14 times.
Bettina Berendt, Veronika Bogina, Robin Burke, Michael D. Ekstrand, Alan Hartman, Stylani Kleanthous, Tsvi Kuflik, Bamshad Mobasher, and Janna Otterbacher. 2019. FairUMAP 2019 Chairs’ Welcome Overview. Meeting summary in Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization (UMAP ’19). ACM. DOI 10.1145/3314183.3323842.
Toshihiro Kamishima, Pierre-Nicolas Schwab, and Michael D. Ekstrand. 2018. 2nd FATREC Workshop: Responsible Recommendation. Meeting summary in Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18). ACM. DOI 10.1145/3240323.3240335. Cited 13 times. Cited 11 times.
Bamshad Mobasher, Robin Burke, Michael D. Ekstrand, and Bettina Berendt. 2018. UMAP 2018 Fairness in User Modeling, Adaptation and Personalization (FairUMAP 2018) Chairs’ Welcome & Organization. Meeting summary in Adjunct Publication of the 26th Conference on User Modeling, Adaptation, and Personalization (UMAP ’18). ACM. DOI 10.1145/3213586.3226200.
Michael D. Ekstrand and Amit Sharma. 2017. The FATREC Workshop on Responsible Recommendation. Meeting summary in Proceedings of the 11th ACM Conference on Recommender Systems (RecSys ’17). ACM. DOI 10.1145/3109859.3109960. Cited 6 times. Cited 14 times.
Martijn Willemsen, Dirk Bollen, and Michael Ekstrand. 2011. UCERSTI 2: Second Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces. Meeting summary in Proceedings of the 5th ACM Conference on Recommender Systems (RecSys ’11). ACM, pp. 395–396. DOI 10.1145/2043932.2044020. Cited 8 times. Cited 8 times.
Other Publications // 5
Publications and presentations that don't fit elsewhere; these have not been peer-reviewed or have been lightly reviewed on the basis of an abstract.
Michael D. Ekstrand, Ben Carterette, and Fernando Diaz. 2021. Evaluating Recommenders with Distributions. In Proceedings of the RecSys 2021 Workshop on Perspectives on the Evaluation of Recommender Systems (RecSys ’21). Cited 2 times.
Katherine Landau Wright, David McNeill, Michael D. Ekstrand, and Maria Soledad Pera. 2019. Supplementing Classroom Texts with Online Resources. At 2019 American Educational Research Association Conference. Cited 17 times.
Katherine Landau Wright, Michael D. Ekstrand, and Maria Soledad Pera. 2018. Supplementing Classroom Texts with Online Resources. At 2018 Annual Meeting of the Northwest Rocky Mountain Educational Research Association.
Michael D. Ekstrand. 2017. Yak Shaving with Michael Ekstrand. CSR Tales no. 4 (December 29th, 2017). PURL https://purl.org/mde/alpaca.
Michael D. Ekstrand. 2014. Towards Recommender Engineering: Tools and Experiments in Recommender Differences. Ph.D thesis, University of Minnesota. HDL 11299/165307. Cited 8 times. Cited 4 times.
Software and Data
I have built several open-source software packages and data sets in the course of my research and other work. Open-source software distribution and open data are key pieces of my research dissemination strategy. My most significant development efforts are:
LensKit, a toolkit for building, researching, and studying recommender systems. As of Sep. 2024, the Python version (2018–) has been used in 63 papers, theses, and educational resources, including the PBS show Crash Course AI, and has been downloaded over 10K times from the Python Package Index in the last 6 months (according to PyPIStats). The original Java software (in development 2010–2018) is known to be used in 71 papers and theses and was used by over 2500 students to complete programming assignments in the Recommender Systems MOOC.
The current version is 0.14.4, released on Feb. 16, 2024; it is the 25rd release of LensKit for Python. Home page: https://lenskit.org (current list of known uses: https://lenskit.org/research/).
Book Data Tools, software tools to integrate multiple public sources of book and book consumption data into a data set for studying social effects in book publication, reading, and recommendation. Used in UMUAI21 and RecSys18. https://bookdata.piret.info
My work has also produced a number of utility packages to support this software and other efforts, including:
seedbank, a Python package for consistently seeding random number generators. https://seedbank.lenskit.org
csr, a Python package for managing sparse matrices in CSR format compatible with the Numba JIT for scientific python, and with Intel MKL acceleration for several operations. https://csr.lenskit.org
binpickle, a Python package for saving scientific data structures (such as machine learning models) to disk in either compressed or memory-mappable format. LensKit uses this package to serialize models for both storage and shared-memory parallelism. https://binpickle.lenskit.org
happylog, a Rust package for easily configuring log output for command-line programs. https://github.com/mdekstrand/happylog
Grapht, a dependency injection framework for Java with novel configuration and static analysis capabilities (paper JOT16). http://grapht.grouplens.org
Invited Talks // 41
- Oct 2024
- Keynote at ROEGEN (workshop at RecSys 2024, Bari, Italy)
“Responsible Recommendation in the Age of Generative AI” - Jul 2024
- Panelist at Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval (at SIGIR 2024)
- May 2024
- Overview talk at Dagstuhl Seminar 24211
- May 2024
- Mar 2024
- Feb 2024
- Oct 2023
- May 2023
- Invited talk at Beyond Nudging, Towards Diversity: Understanding Transparent Algorithmic Recommendation Practices for Media and Communications (post-conference panel at ICA 2023, virtual)
“Beyond Diversity and Transparency: Normative Recommendation Goals in Human Context” - Mar 2023
- Feb 2023
- Seminar at Drexel University
“Maps and Lenses on Fairness in Information Access Systems” - Jan 2023
- Nov 2022
- Keynote at IBIS2022 (Information-Based Inductive Systems and Machine Learning) (Japanese machine learning conference, Tsukuba, JP)
“The Complexity of Fairness in Information Access” - Nov 2022
- Oct 2022
- Keynote at EvalRS workshop at CIKM 2022
“Do You Want To Hunt A Kraken? Mapping and Expanding Recommendation Fairness” - Aug 2022
- Guest lecture at University of Maine IR course (virtual)
“Fair IR and Test Collections” - Mar 2022
- Seminar at University of Michigan School of Information (virtual)
“You Might Also Think This Is Unfair” - Nov 2021
- Oct 2020
- Guest lecture at Carnegie Mellon University Human-AI Interaction course
“Recommender Systems and Fairness” - Apr 2020
- Guest lecture at Emory University recommender systems course
“Recommender Systems and Fairness” - Mar 2020
- Seminar at Boise State University Ph.D in Computing Colloquium
“User, Agent, Subject, Spy” - Nov 2019
- Seminar at University of Texas at Austin
“Use,r Agent, Subject, Spy” - Oct 2019
- Session at Idaho Library Association 2019 Conference
“Online Recommendation: What? Where? Why? How?” - Aug 2019
- Lecture at IVADO Summer School (Montréal, QC)
“Fairness and Discrimination in Recommendation and Retrieval” - Aug 2019
- Seminar at Microsoft Research Montréal
“User, Agent, Subject, Spy” - Jul 2019
- Seminar at Criteo AI Labs (Paris, France)
“User, Agent, Subject, Spy ” - May 2019
- Invited talk at CRA CCC Visioning Workshop on Economics and Fairness
“Recommendations, Decisions, Feedback Loops, and Maybe Saving the Planet” - Dec 2018
- Seminar at Clemson University
“User, Agent, Subject, Spy” - Nov 2018
- Seminar at Carnegie Mellon University Human-Computer Interaction Institute
“User, Agent, Subject, Spy” - Nov 2018
- Guest lecture at Carnegie Mellon University Human-AI Interaction course
“Recommender Systems” - Nov 2017
- Seminar at Whitman College (Walla Walla, WA)
“Making Information Systems Good for People” - Oct 2017
- Overview talk at Dagstuhl Seminar 17442
- Jun 2017
- Seminar at RecSysNL at TU Delft (Delft, NL)
“Recommending for People” - Jun 2017
- Seminar at Jheronimus Academy of Data Science (’s-Hertogenbosch, NL)
“Recommending for People” - Jun 2017
- Seminar at UCL Mons (Mons, BE)
“Recommending for People” - Jun 2017
- Keynote at Brussels Big Data and Ethics Meetup (inaugural event of the DigitYser Big Data community, Brussels, BE)
“Responsible Recommendation” - Nov 2016
- Seminar at University at Albany
“Recommending for People” - Oct 2016
- Lecture at Clearwater Developer Conference (Boise, ID)
“Introduction to Recommender Systems ” - Sep 2015
- Invited talk at Large-Scale Recommender Systems (workshop at RecSys ’15)
“Challenges in Scaling Recommender Systems Research” - Sep 2015
- Invited talk at RecSys Doctoral Symposium
“Levelling Up your Academic Career” - Sep 2012
- Invited talk at RecSys Challenge (workshop at RecSys ’12)
“Flexible Recommender Experiments with LensKit” - Sep 2012
- Invited talk at RecSys Challenge (workshop at RecSys ’12)
“The MovieLens Data Set”
Teaching
Drexel University
| Term | Course | Title | Credits | In-Person | Online |
|---|---|---|---|---|---|
| F24 | INFO 659 | Intro to Data Analytics | 3 | 5 | 20 |
| S24 | INFO 659 | Intro to Data Analytics | 3 | 16 | 15 |
| W24 | DSCI 641 | Rec. Sys. for Data Science | 3 | 16 | 10 |
| F23 | INFO 659 | Intro to Data Analytics | 3 | 9 | 16 |
Boise State University
| Term | Course | Title | Credits | Students |
|---|---|---|---|---|
| S23 | CS 538 | Recommender Systems | 3 | 12 |
| F22 | CS 533 | Intro to Data Science | 3 | 27 |
| S22 | CS 230 | Ethics in Computing | 3 | 61 |
| F21 | CS 533 | Intro to Data Science | 3 | 43 |
| S21 | CS 538 | Recommender Systems | 3 | 11 |
| F20 | CS 533 | Intro to Data Science | 3 | 22 |
| S20 | CS 697 | Equity and Discrimination | 3 | 3 |
| S20 | CS 410 | Databases | 3 | 36 |
| F19 | CS 533 | Intro to Data Science | 3 | 28 |
| S19 | CS 538 | Recommender Systems | 3 | 12 |
| F18 | CS 410/510 | Databases | 3 | 40 |
| Su18 | CS 310-HU | Intro to Databases | 1 | 6 |
| S18 | CS 410/510 | Databases | 3 | 22 |
| F17 | CS 533 | Intro to Data Science | 3 | 22 |
| S17 | CS 597 | Recommender Systems | 3 | 13 |
| F16 | CS 410/510 | Databases | 3 | 28 |
Texas State University
| Term | Course | Title | Students |
|---|---|---|---|
| S16 | CS 3320 | Internet Software Development | 48 |
| S16 | CS 5369Q/4379Q | Recommender Systems | 26 |
| F15 | CS 4332 | Intro to Database Systems | 39 |
| S15 | CS 5369Q/4379Q | Recommender Systems | 28 |
| S15 | CS 4350 | Unix Systems Programming | 32 |
| F14 | CS 4332 | Intro to Database Systems | 50 |
Coursera
I co-created the Recommender Systems specialization on Coursera, along with its two previous single-class versions, with Joseph A. Konstan. This course has reached over 95,000 learners across its 3 iterations.
University of Minnesota
| Term | Course | Title | Role | Students |
|---|---|---|---|---|
| F13 | CSCI 5980-1 | Intro to Recommender Systems | Inst | ~35 |
| S12 | CSCI 1902 | Structure of Comput. Prog. II | Inst | ~25 |
| S11 | CSCI 5125 | Collaborative and Social Computing | TA | ~30 |
| Su08 | CSCI 1902 | Structure of Comput. Prog. II | TA | ~30 |
| S08 | CSCI 1902 | Structure of Comput. Prog. II | TA | ~120 |
| F07 | CSCI 1902 | Structure of Comput. Prog. II | TA | ~120 |
Teaching Professional Development
- Boise State University teaching portfolio faculty learning community.
- Boise State University Ten for Teaching program.
- Boise State University Center for Teaching and Learning Course Design Institute, a one-week intensive session in Summer 2017.
- CTL workshops on service learning, mastery-based grading, and other topics.
- Texas State University’s Program for Excellence in Teaching and Learning (2014–2015).
- Preparing Future Faculty at the University of Minnesota.
Service
Ongoing Professional Service, Memberships, and Honors
- Associate editor, ACM Transactions on Recommender Systems (2024–)
- Editorial board, Foundations and Trends in Information Retrieval (2023–)
- Co-chair, FAccT Network, 2019–
- Steering committee, ACM Conference on Recommender Systems (RecSys), 2017–
- Senior Member, Association for Computing Machinery (since 2019)
- Distinguished Reviewer, ACM Transactions on Interactive Intelligent Systems (TiiS) (2017–present)
Past Service Highlights
- Executive committee, ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2020–2023
- Program co-chair, 16th ACM Conference on Recommender Systems (RecSys 2022)
- General co-chair, 12th ACM Conference on Recommender Systems (RecSys 2018)
Program Committee and Editorial Service
2024
- Program committee, ECIR 2025 (technical program, short papers)
- Program committee, ACM CIKM 2024
- Senior program committee, ACM RecSys 2024
- Area chair, ACM FAccT 2024
- Area chair, ACM SIGIR 2024
- Reviewer, ACM Transactions on Information Systems
- Reviewer, ACM Transactions on Recommender Systems
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, ACM Transactions on Social Computing
- Reviewer, ACM Computing Surveys
- Workshop reviewing: RecSys in HR @ RecSys 2024, FAccTRec @ RecSys 2024, RecSoGood @ RecSys 2024, ROGEN @ RecSys 2024, and EXUM @ UMAP 2024
2023
- Program committee, ECIR 2024 (technical program, short papers, IR for Good, tutorials)
- Senior program committee, ACM RecSys 2023
- Program committee, ACM RecSys 2023 (Reproducibility)
- Senior program committee, SIGIR Asia-Pacific 2023
- Best paper committee, ACM SIGIR 2023
- Best paper committee, TheWebConf 2023
- Program committee, ACM SIGIR 2023
- Track chair, UMAP 2023 (Responsibility, Compliance, and Ethics)
- Area chair, ACM FAccT 2023
- Reviewer, ACM Transactions on Recommender Systems
- Reviewer, ACM Transactions on Information Systems
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, ACM Computing Surveys
- Reviewer, ACM Transactions on Social Computing
- Reviewer, ACM Journal of Responsible Computing
- Reviewer, Information Retrieval Journal
- Workshop reviewing: RecSys4HR at RecSys and PERSPECTIVES at RecSys
2022
- Reviewer, ACM Transactions on Recommender Systems
- Reviewer, ACM Computing Surveys
- Reviewer, ACM Transactions on Intelligent Systems and Technology
- Reviewer, Information Retrieval Journal
- Reviewer, Journal of Librarianship & Information Science
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, User Modeling and User-Adapted Interaction
2021
- Guest editor, 2021 special issue of User Modeling and User-Adapted Interaction (UMUAI) on fairness in user modeling.
- Track Chair, UMAP 2021
- Program committee, ACM SIGIR 2021 (technical program, Perspectives, short papers, resource track)
- Senior program committee, ACM RecSys 2021
- Program committee, ACM RecSys 2021 (Reproducibility)
- Program committee, ACM CIKM 2021 (resource track)
- Program committee, ACM WSDM 2021
- Ethics reviewer, NeurIPS 2021
- Reviewer, User Modeling and User-Adapted Interaction
- Reviewer, ACM Computing Surveys
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, ACM Transactions on Information Systems
- Reviewer, ACM/IMS Transactions on Data Science
- Reviewer, ACM Transactions on Computer-Human Interaction
- Workshop reviewing: BIAS 2021 at ECML-PKDD, BIAS 2021 at ECIR, ExUM at UMAP, RecSysHR at RecSys, IntRS at RecSys, and Perspectives at RecSys
2020
- Program committee, ACM SIGIR 2020
- Program committee, UMAP 2020
- Senior program committee, ACM RecSys 2020
- Program committee, ACM CIKM 2020 (resource track)
- Program committee, ACM RecSys 2020 (LBR)
- Program committee, ACM WSDM 2020
- Program committee, ACM FAccT 2021
- Senior program committee, TheWebConf 2021 (User Modeling, Behavior, & Personalization)
- Reviewer, ACM Transactions on Computer-Human Interaction
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, ACM Transactions on Information Systems
- Reviewer, User Modeling and User-Adapted Interaction
- Additional reviews for UIST, CSCW, and CHI
- Workshop reviewing: ExSS-ATEC at IUI, Bias at ECIR, ExUM at UMAP, BIAS at ECMLPKDD, ImpactRS, and IntRS
2019
- Senior program committee, ACM RecSys 2019
- Program committee, ACM RecSys 2019 (LBR)
- Program committee, CHI 2019 (Posters)
- Program committee, UMAP 2019
- Program committee, TheWebConf 2020 (User Modeling, Behavior, & Personalization)
- Reviewer, User Modeling and User-Adapted Interaction
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, Communications of the ACM
- Additional reviews for FAT, FAT, CSCW, and CHI
- Workshop reviewing: HUMANIZE, Handling Web Bias @ WWW 2019, ATEC @ IUI 2019, IntRS @ RecSys 2019, and ImpactRS @ RecSys 2019
2018
- Program committee, TheWebConf 2019 (User Modeling, Behavior, & Personalization)
- Program committee, UMAP 2018
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, ACM Transactions on the Web
- Reviewer, IEEE Transactions on Knowledge and Data Engineering
- Reviewer, Communications of the ACM
- Additional reviews for FAT
- Workshop reviewing: HealthRecSys and KidRec
2017
- Program committee, FLAIRS 2017 (Special Track on Recommender Systems)
- Program committee, TheWebConf 2018 (User Modeling, Behavior, & Personalization)
- Program committee, ACM RecSys 2017 (technical program, Posters)
- Reviewer, Interacting with Computers
- Additional reviews for FAT and UIST
- Workshop reviewing: IntRS at RecSys, KidRec at RecSys, FATML at KDD, EnCHIres, and SOAP at UMAP
2016
- Program committee, FLAIRS 2016 (Special Track on Recommender Systems)
- Program committee, TheWebConf 2016 (User Modeling, Behavior, & Personalization)
- Program committee, ACM SAC 2016 (Recommender Systems)
- Program committee, ACM RecSys 2016 (Posters, technical program)
- Reviewer, Artificial Intelligence Review
- Reviewer, ACM Transactions on Interactive Intelligent Systems
- Reviewer, International Journal of Artificial Intelligence Tools
- Reviewer, PLOS ONE
- Reviewer, JMLR Open Source
- Reviewer, IBM Journal of Research and Development
- Additional reviews for IUI, CSCW, CHI, CHI (Late-Breaking Work), ICSOC, and UIST
- Workshop reviewing: LSRS at RecSys
2015
- Program committee, ACM RecSys 2015
- Program committee, FLAIRS 2016 (Special Track on Recommender Systems)
- Program committee, TheWebConf 2016 (Behavior Analysis and Recommendation)
- Reviewer, ACM Transactions on Intelligent Systems and Technology
- Reviewer, User Modeling and User-Adapted Interaction
- Reviewer, PeerJ Computer Science
- Reviewer, ACM Computing Surveys
- Reviewer, Information Retrieval Journal
- Additional reviews for CHI
- Workshop reviewing: Social Personalization and Search at SIGIR
2014
- Program committee, ACM RecSys 2014
- Program committee, FLAIRS 2015 (Special Track on Recommender Systems)
- Reviewer, ACM Transactions on Intelligent Systems and Technology
- Reviewer, IEEE Transactions on Dependable and Secure Computing
- Additional reviews for CHI and CSCW
2013
- Reviewer, IEEE Transactions on Knowledge and Data Engineering
- Reviewer, ACM Transactions on the Web
2012
- Program committee, ACM SAC 2013 (Recommender Systems)
- Reviewer, Advances in AI
- Reviewer, Information Retrieval
- Additional reviews for UIST and CHI (Work in Progress)
Other Professional Service
- Track co-oragnizer, Product Search and Recommendation track at TREC 2025
- Doctoral symposium co-chair, ACM RecSys 2024
- Founder and co-organizer, FATREC Workshop on Responsible Recommendation at RecSys 2017–2018, 2020–2021, 2023–2024
- Co-organizer, AltRecSys Workshop on Alternative, Unexpected, and Critical Ideas in Recommendation at RecSys 2024
- Participant, Dagstuhl Seminar 24211: Evaluation Perspectives of Recommender Systems: Driving Research and Education (2024)
- Steering committee, ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2017–2023 (inaugural member)
- Co-author and signatory, FAccT Statement on AI Harms and Policy (2023); covered by VentureBeat and The Hill (op-ed)
- Co-organizer, CRAFT panel “Theories of Change in Responsible AI” at FAccT 2023
- Ph.D. symposium mentor, CIKM 2023
- Co-organizer, TREC Track on Fairness in Information Retrieval (2019–2022)
- Co-organizer, SimuRec Workshop on Simulation and Synthetic Data for Recommender Systems at RecSys 2021
- Sponsorship co-chair, ACM FAccT 2021–2022
- Doctoral symposium co-chair, ACM RecSys 2022
- Co-organizer, FairUMAP workshop at UMAP 2018–2020
- Organized and moderated panel at RecSys 2019 on responsible recommendation
- PR & Publicity co-chair, 2nd Conference on Fairness, Accountability, and Transparency (ACM FAT* 2019)
- Co-organizer, Workshop on Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval (FACTS-IR) at SIGIR 2019
- Publications working group, FAccT steering committee (2017)
- Participant, Dagstuhl Perspectives Workshop 17442: Towards Cross-Domain Performance Modeling and Prediction: IR/RecSys/NLP (2017)
- Publicity co-chair, ACM RecSys 2016
- External advisor, CrowdRec (EU Framework Programme collaborative research project, 2014–2016)
- Proceedings co-chair, ACM CHI 2012–2013
- Demos co-chair, ACM RecSys 2012
Department and University Service
- Drexel IS 2023-2024 Faculty Search Committee
- Drexel IS Ph.D. committee (2023-2024)
- Boise State 2020–2021 CS Faculty Search Committee
- Boise State COEN SAGE Scholars Program Mentor (2019–2021)
- Boise State College of Engineering Curriculum Committee (2019–2022)
- Boise State Ph.D. in Computing Steering Committee (2017–2022)
- Boise State CS Dept. Curriculum Committee (2017–2022)
- Boise State CS Dept. Graduate Recruiting Committee (2017)
- Texas State CS Dept. Undergraduate Committee (2014–2016)
- Texas State CS Dept. Written Comp Exam Grading (2014–2016)
- UMN CS Graduate Student Association secretary (2009–2010)
Community and Civic Service
- January 2023 — joined amicus brief before SCOTUS on Gonzalez v. Google.
- July 2020 — taught continuing education session for Idaho Council for Libraries.
- October 2019 — presented at Idaho Library Association Annual Conference.
- February 2019 — addressed Idaho State House Judiciary Committee on H.B. 118, regulating pretrial risk assessment algorithms; through subsequent engagement, I contributed language that is in the final enacted legislation.
- December 2017 — Boise Public Library panel on preparing for a career in computer science.
- 2015 — Judge for Travis Elementary School Science Fair.
Media Mentions
- “FACT FOCUS: Google autocomplete results around Trump lead to claims of election interference”. (Melissa Goldin, AP, July 30, 2024, https://apnews.com/article/fact-check-misinformation-google-autocomplete-trump-b31855c23eb6e387dc324983ea4859bc).
- “Getting to know you: This is what chatGPT says Philly is famous for”. (Vicky Diaz-Camacho, Billy Penn at WHYY, March 11, 2024. https://billypenn.com/2024/03/11/chatgpt-artificial-intelligence-philadelphia-known-for-cheesesteaks/).
- “The Deadline Dilemma”. (Carolyn Kuimelis, Teaching newsletter from Chronicle of Higher Education, December 1, 2022. https://www.chronicle.com/newsletter/teaching/2022-12-01).
- “Out of the Blue”. (Ravi Shankar, The New Indian Express, May 1, 2022. https://www.newindianexpress.com/opinions/columns/ravi-shankar/2022/may/01/outof-theblue-2447591.html). Quotes from Washington Post article below.
- “Elon Musk wants Twitter’s algorithm to be public. It’s not that simple.” (Reed Albergotti, The Washington Post, April 16, 2022. https://www.washingtonpost.com/technology/2022/04/16/elon-musk-twitter-algorithm/).
- Quoted at length about how artificial intelligence learns from social signals in “Can AI be horny?” (Chris Stokel-Walker, Input, April 28, 2021; Bustle Digital Group. https://www.inputmag.com/culture/artificial-intelligence-ai-archillect-twitter-horny-sex).
- Quoted in several articles about FAccT suspending Google’s sponsorship for the 2021 conference, in my role as FAccT Sponsor Co-chair and a member of the Executive Committee. These articles include:
- “AI ethics research conference suspends Google sponsorship.” (Khari Johnson, VentureBeat, March 2, 2021. https://venturebeat.com/2021/03/02/ai-ethics-research-conference-suspends-google-sponsorship/)
- “Conference suspends Google sponsorship after ethics experts’ exit.” (D. Matthews, Times Higher Education, March 8, 2021. https://www.timeshighereducation.com/news/conference-suspends-google-sponsorship-after-ethics-experts-exit)
- “Tech transparency conference suspends Google sponsorship over transparency concerns.” (Colleen Flaherty, Inside Higher Ed, March 9, 2021. https://www.insidehighered.com/news/2021/03/09/tech-transparency-conference-suspends-google-sponsorship-over-transparency-concerns)
- “Google offered a professor $60,000, but he turned it down. Here’s why.” (Rachel Metz, CNN Business, March 24, 2021. https://www.cnn.com/2021/03/24/tech/google-ai-ethics-reputation/index.html). I am not the professor who declined funding, but am quoted for context.
- “How one employee’s exit shook Google and the AI industry.” (Rachel Metz, CNN Business, March 11, 2021. https://www.cnn.com/2021/03/11/tech/google-ai-ethics-future/index.html).
- Quoted about voter file data leaks in “D.C. makes it shockingly easy to snoop on your fellow voters.” (Brian Fung, The Switch [a blog by The Washington Post], June 14, 2016. https://www.washingtonpost.com/news/the-switch/wp/2016/06/14/d-c-s-board-of-elections-makes-it-shockingly-easy-to-snoop-on-your-fellow-voters/)
- Quoted about recommender systems principles in “TV seems to know what you want to see; algorithms at work.” (Scott Collins, Los Angeles Times, November 21, 2014. https://www.latimes.com/entertainment/tv/la-et-st-tv-section-algorithm-20141123-story.html)