Smartly Report - Final 08-10-2016

Foundational
MBA accounting and finance

knowledge acquisition:
Pedago’s Smartly vs. traditional MBA
programs

Assessment Design Study Design and Data Analysis

● Pamela Drake, PhD, CFA, ● Lindsay Oishi, PhD, Educational
Chandler/Universal Eminent Professor of Psychology, Stanford University
Finance at James Madison University ● Meg Gordon, MA, International
● Patrick Gosselin, MBA, CFP Comparative Education, Stanford
University

July 2016
Contents
Executive summary ...................................................................................................................................... 3
Introduction ............................................................................................................................................. 3
Assessment ............................................................................................................................................... 3
Coursework .............................................................................................................................................. 3
Findings .................................................................................................................................................... 3
Conclusions .............................................................................................................................................. 5
Detailed report ............................................................................................................................................. 6
Introduction ............................................................................................................................................. 6
Recruitment and eligibility ....................................................................................................................... 6
Test development ..................................................................................................................................... 7
Participation ............................................................................................................................................. 8
Smartly coursework .................................................................................................................................. 8
Testing ...................................................................................................................................................... 8
Final study sample .................................................................................................................................... 9
Findings .................................................................................................................................................. 11
Conclusion .............................................................................................................................................. 17
Limitations of the study .......................................................................................................................... 17
Possible future studies ........................................................................................................................... 17
Appendices ................................................................................................................................................. 19
Appendix A: Eligibility funnel .................................................................................................................. 19
Appendix B: Accounting - Detailed results ............................................................................................. 20
Appendix C: Finance - Detailed Results .................................................................................................. 23
Appendix D: Details on tests .................................................................................................................. 27
Appendix E: Screeners ............................................................................................................................ 40
Appendix F: Course Syllabi - Accounting I & II (128 minutes) ................................................................. 42
Appendix G: Course Syllabi - Finance (111 minutes) .............................................................................. 44
Appendix H: Fair assessment of subject matter knowledge? - Student comments ............................... 46

Tables and Figures

Table 1. Participation by gender ................................................................................................................ 10
Table 2. Participation by Age ...................................................................................................................... 10
Table 3. Comparison of Smartly pre-test results with MBA group ............................................................. 11
Table 4. Comparison of Smartly participants' pre- and post-test scores ................................................... 12
Table 5. Explaining score improvement ..................................................................................................... 13
Table 6. Comparison of post-test performance between Smartly and MBA groups ................................. 14
Table 7. Representative Comments: Test was fair assessment of accounting knowledge? ...................... 15
Table 8. Representative Comments: Test was fair assessment of finance knowledge? ............................ 16

Executive summary
Introduction
This study is a comparison of the outcomes of foundational MBA accounting and finance concepts learned
through Smartly online courses compared to those learned through traditional brick-and-mortar business
schools in the United States. We recruited 105 recent MBA graduates and current students (“MBA group”)
who had completed relevant accounting or finance coursework, with 51 and 54 participants in the
Accounting and Finance groups, respectively. MBA participants came from 26 U.S. universities. The
comparison group that went through the Smartly program (“Smartly group”) is composed of 77 college
graduates (34 Accounting, 43 Finance) who had not studied anything in the respective subject matter
areas.
Pedago funded this study and provided significant input and support for the design and execution of the
research. The authors, however, provided independent data analysis and conclusions, and advised on all
aspects of participant recruitment, selection, communication, and assessment.

Assessment
Pedago partnered with subject-matter experts to create two custom, online assessments in each subject.
Participants in the Smartly group took a one-hour pre-test, completed Smartly courses in their respective
subject matter areas (including receiving at least an 80 percent pass rate on quizzes), and then completed
a post-test. Participants in the MBA group completed the post-test only. Preliminary analyses show that
learners in the Smartly groups performed as well as or better than MBA participants at post-test.
Coursework
The Smartly Accounting participants completed the Smartly courses Accounting I: Fundamentals and
Accounting II: Revenue and Expenses. These courses help students discover financial statement
preparation principles, learn bookkeeping mechanics for basic transactions, and take a deeper look at how
to account for revenues and expenses, including the cost of goods sold and depreciation.
The Smartly Finance participants completed the Smartly courses Finance: Time Value of Money, Valuation
I: Discounted Free Cash Flow, and Cost of Capital I: Capital Structure. These courses are intended to help
students acquire basic mathematical ability around the time value of money, gain familiarity with some
financial terminology, use tools to forecast a company’s future free cash flows, apply financial math to
value a company by discounting those cash flows back to the present, and evaluate a company’s balance
between debt and equity as a means of assessing the profitability of proposed projects.
Findings
Improvement (pre-test to post-test): Smartly participants

Smartly participants improved an average of 29 percentage points (Accounting) and 33 percentage points
(Finance) from pre-test to post-test. Statistical analyses indicate that there are significant changes from
pre-test to post-test for the participants in both Accounting and Finance.
3
Mean test score
Number of Score
Course participants Pre-test Post-test Improvement
Accounting 34 57% 86% 29 points
Finance 43 49% 82% 33 points
Though Smartly participants took a similar test twice within two weeks, it is unlikely that this improvement
is due to practice or memory effects. After the pre-test, Smartly participants did not receive feedback on
their performance, could not view correct answers, and could not view the test again.
Post-test performance: Smartly versus MBA

Smartly groups performed as well as, in the case of Finance, or better than, in the case of Accounting,
the MBA groups on assessments of the topics studied.
Smartly MBA
Mean Post- Number of Mean Post-test Number of

Course test score participants score participants
Accounting 86% 34 75% 50
Finance 82% 43 81% 54
Time Spent in Smartly Courses

On average, Smartly participants spent 147 minutes completing accounting coursework and 118 minutes
completing finance coursework. These times were in line with course estimates of 128 minutes for
Accounting and 111 minutes for Finance.
Participant evaluation of Smartly custom assessment

At the end of the post-test, all participants were asked, “Do you think this was a fair assessment of your
[accounting / finance] knowledge? (Yes / No)”, and were also asked to explain their response. A majority
of participants said that the custom Smartly Accounting assessment was fair (85 percent Smartly, 86
percent MBA); the Finance assessment was also considered fair (84 percent Smartly, 69 percent MBA).1
Representative comments include:
MBA (Accounting) “It asked the fundamentals that I learned in Accounting I”
Smartly (Finance) “I was pleasantly surprised that the answers I had zero clue about in the
pre-assessment, I now understood.”

1
Only 26 of 54 the MBA Finance participants answered this question. The question for this group was sent out via email a few weeks after the
post-test was taken, so the lower response rate may have just been circumstantial and should not be taken as an intentional unwillingness among
this group to answer this question.
4
Smartly participant satisfaction (Net Promoter Score)
Smartly participants were asked to respond (at the conclusion of the second test) to a standard
satisfaction question, “How likely are you to recommend Smartly to a friend?”, on a scale of 0 (Would not
recommend) to 10 (Definitely would recommend). The Net Promoter Score, NPS, is calculated as the
percentage of respondents answering 9 or 10 (promoters), less the percentage of respondents answering
0 to 6 (detractors), with scores ranging from -100 to 100.
The Smartly accounting course had 58 percent promoters and 6 percent detractors, receiving an NPS score
of 52. The Smartly finance course had 49 percent promoters and 9 percent detractors, receiving an NPS
score of 40. These scores are in line with recent NPS scores for the Harvard MBA program and Wharton
MBA program of 41 and 51, respectively. 2,3 Representative comments include:
Accounting “I actually really loved the courses. I felt like I learned a lot and was consistently
engaged - the layout was great and constantly answering questions and clicking
through was super entertaining/helpful. I would definitely take more courses in the
future if offered.”
Finance “Excellent. I feel I learned an extraordinary amount in a short period of time. The
challenges and questions were the perfect balance between challenging and fun.”
Conclusions
Many companies are looking for the skills developed while pursuing an MBA. However, the pursuit of an
MBA may not always be the right choice for all learners, either due to financial, geographic or time
constraints.
This study supports the assertion that some of the foundational accounting and financial concepts taught
in traditional brick-and-mortar MBA program can be learned independently, online through Pedago’s
targeted Smartly active-learning courses. Significant improvement in students’ knowledge can be gained
in as little as two hours of engagement with these courses.
In addition to being taken on their own as part of a comprehensive independent study MBA program,
these courses could be used as supplements to existing MBA programs. First, they could be used as
remedial materials for incoming MBA students who need to enhance their understanding of basic
concepts in certain subject matter areas. Second, they could be used by MBA instructors as pre-requisite
work for blended-learning classes. By having students learn basic concepts and complete structured
practice exercises on their own, instructors could devote more classroom time to deeper learning through
case studies and group work.

2
Steven Hind, “What is HBS’s NPS?” The Harbus, February 5, 2015, (www.harbus.org/2015/the-harbus-survey-
says-hbs-mba-program-more-walgreens-than-apple/)
3
Natalie Kitroeff, “MBAs are more Satisfied at Wharton than at Harvard Business School”, Bloomberg, March 31,
2015 (www.bloomberg.com/news/articles/2015-03-31/mbas-are-more-satisfied-at-wharton-than-at-harvard-
business-school).
5
Detailed report
Introduction
Smartly is an online educational technology platform by Pedago LLC that replaces traditional lecture-
based instruction with active, individualized lessons available on any device. Smartly lessons rely on the
principles of active learning, requiring learners to solve real problems several times per minute by
answering questions that include multiple choice, fill in the blank, and matching formats. Every lesson
builds on previously learned concepts so that students’ understanding is continually tested as they are
introduced to new concepts.
After taking an action within Smartly, the learner receives immediate constructive explanations about
errors; learners are encouraged to explore why answers are right or wrong, aiding a better understanding.
Smartly’s “bite-sized” lessons make learning more manageable by building on previously learned
concepts, and providing immediate feedback. Smartly is optimized for both desktop and mobile devices,
so learners can complete coursework at their convenience. As of May of 2016, Smartly lessons had a 96-
percent learner approval rating.
The Smartly MBA curriculum consists of nine core subject areas. Each area contains required courses, with
up to 120 interactive, feedback-based lessons, each between 5 and 10 minutes in duration. A course
consists of 5-7 lessons, the final lesson being a SMARTCASE, a short assessment of the concepts taught in
the previous lessons.
The purpose of this study was to investigate whether accounting and finance concepts learned in
foundational courses at traditional brick-and-mortar business schools in the United States are learned as
effectively through Smartly courses. This question was addressed by comparing the scores of recent
graduates and current students from U.S. MBA programs to those of Smartly students on custom, online
assessments of foundational concepts in the respective subject matter areas.
Recruitment and eligibility
Recruitment
All participants were recruited from 100+ online social media MBA groups using one of three screens. No
compensation was received for completing a screen, and over 810 people completed at least one question
on the screen. Candidates were required to be in the age range of 18-44 years to be considered for
participation in the study.
Eligibility
MBA groups
All MBA participants had to have completed an undergraduate degree. Participants in the MBA
Accounting group must have also completed at least one accounting course in an accredited MBA
program. Participants in the MBA Finance group must have also completed both corporate finance and
financial accounting (or equivalent) courses in an accredited MBA program. From those screened, 51
participants were selected to take the Accounting post-test and 54 participants were selected to take the
Finance post-test.
6
Smartly groups
Participants in the Smartly groups could not have completed any MBA program nor completed an
undergraduate degree in business, finance, or accounting. Additionally, participants must not have
completed related courses outside of an MBA program or held a job in that field. For Smartly Accounting
participants, this included courses in accounting or a job as an accountant. For Smartly Finance
participants, this included courses in finance or a job with “finance” in the title. From those screened, 93
participants were selected to take the Smartly Accounting pre-test and 119 participants were selected to
take the Smartly Finance pre-test. Of those invited, 62 participants in each group completed the pre-test.
For Accounting, 49 participants started the coursework and 40 participants both finished the coursework
requirements and took the post-test. For Finance, 47 participants started the coursework and 43
participants both finished the coursework requirements and took the post-test.
Participants in both MBA and Smartly groups were assigned either to a Finance or Accounting group, and
could not be included in both even if they completed tests and/or coursework for both subject areas.
Neither MBA nor Smartly participants had previously completed any Smartly courses prior to the study.
Test development
Both the Accounting and Finance tests were created by Pedago’s content development team, which
includes full-time employees and contracted professors. Existing MBA course exams in these fields were
used as a guideline for development.
Evaluation and pilot testing

The tests were evaluated by Pamela Drake, PhD, CFA, Chandler/Universal Eminent Professor of Finance
at James Madison University and Patrick Gosselin, MBA, CFP. A Spanish-language version of the test was
piloted with students from Universidad Politécnica de Madrid in Spain (an institutional partner of Pedago)
to evaluate the time required to complete each test.
Pilot tests were conducted with groups of current MBA students recruited through ten online social media
MBA groups to ensure that questions were a good test of MBA-level knowledge. One pilot test with eight
participants was conducted for the Accounting test, while two pilots were conducted for the Finance test
with 15 and 31 participants. Participants were paid $25 each and met the same eligibility requirements as
MBA participants in the main study. One question on the Accounting test was removed and ten questions
on the Finance test were either removed or changed based on this testing.
Final tests
The final pre-test and post-test in Accounting contained fourteen questions and the final pre-test and
post-test in Finance contained sixteen questions. Many questions in the Accounting tests contained
multiple parts. Therefore, participants were required to provide forty-three answers for the Accounting
tests. The Finance tests included seventeen answers.
Pre-tests included a question again asking students about prior coursework in the respective subject
matter areas. Post-tests also contained one question asking whether the test was a fair assessment of
subject matter knowledge (either MBA course knowledge or Smartly course knowledge, respectively) and
a follow up question where participants could explain their answer. Post-tests also included a Net
Promoter Score (NPS) question for Smartly participants to indicate whether they would recommend the
Smartly course to a friend, and why or why not.
7
Participation
MBA group
Once selected, participants in the MBA group were given a few days from when they were notified of
being accepted into the study to complete the MBA post-test. Smartly students were given up to two
weeks to complete their pre-test, Smartly coursework, and post-test. All participants communicated with
a research lead who monitored their progress, answered any questions about the research study
procedures, and sent reminders to complete the study.
MBA group participants were paid $50 to complete the one-hour post-test. Smartly participants were paid
$50 to complete the one-hour pre-test, $100 to complete all necessary coursework (approximately 2-2.5
hours), and $50 to complete the one-hour post-test.
107 MBA participants were sent a link to complete the Accounting post-test and 95 MBA participants were
sent the link to complete the Finance post-test. From this group, 51 participants completed the
Accounting post-test and 54 participants completed the Finance post-test.
Smartly group
Each Smartly participant received an account for the respective courses and was emailed a username and
a temporary password. Smartly lessons provide detailed feedback on both correct and incorrect answers
and allow students to retake SMARTCASES, short quizzes at the end of each chapter. Smartly participants
were required to receive at least 80 percent on all course quizzes to be eligible to take the post-test. All
participants that met these requirements were emailed the post-test to complete by the end of the initial
allotted time for the entire study period.
Smartly coursework
Accounting participants completed the Smartly courses Accounting I: Fundamentals and Accounting II:
Revenue and Expenses. These courses are intended to help students discover financial statement
preparation principles, learn bookkeeping mechanics for basic transactions, and take a deeper look at how
to account for revenues and expenses, including the cost of goods sold and depreciation.
Finance participants completed the Smartly courses Finance: Time Value of Money, Cost of Capital I:
Capital Structure, and Valuation I: Discounted Free Cash Flow. These courses help students apply the
principles related to a company’s balance between debt and equity, assess the profitability of proposed
projects, acquire basic mathematical ability with the time value of money, gain familiarity with financial
terminology, forecast a company’s future free cash flows, and value a company by discounting those cash
flows back to the present.
Based on course estimates, students should have been able to complete all lessons within 128 minutes
for Accounting and 111 minutes for Finance. Detailed course syllabi including course structure, as well as
concepts about and time requirements for each lesson within the required courses, are available in the
Appendices.
Testing
Once participants qualified, instructions for participating in the study were emailed to each candidate.
The pre- and post-tests were administered over Google Forms.
Participants were told that each test should take less than an hour to complete. Though strict time limits
were not enforced, students were encouraged to spend no more than 3 minutes on each question.
8
Participants were informed that they would be unable to change an answer once it was submitted, and
participants were advised not to look up any answers to questions using external references. However,
this could not be enforced due to the nature of the testing platform and protocol used.
The mean time between pre-tests and post-tests in the Smartly group was five days for Accounting
participants and six days for Finance participants; the minimum was less than one day and the maximum
was twelve days. After the pre-test, Smartly participants did not receive any feedback on their
performance, could not view correct answers, and could not view the test again. Based on these factors,
it is unlikely that any improvement would be due to practice or memory effects, despite the short interval
between tests. Additionally, more than 95 percent of MBA participants were current MBA students.
Therefore, they had completed the required courses within at least the last two years and should have
also had relatively recent exposure to the knowledge being tested.
Final study sample

Final groups
The final MBA groups consisted of 51 participants for Accounting and 54 participants for Finance. From
the eligible Smartly groups, 40 participants and 45 participants completed the coursework and took the
post-test for Accounting and Finance, respectively. Some of those candidates were excluded from the final
analysis based on updated information on prior knowledge in the subject areas being tested. The final
Smartly groups consisted of 34 participants for Accounting and 43 participants for Finance.
Test answers
All answers were included in the final analysis of the Finance pre-and post-tests (17). Two answers for the
same question were excluded from the analysis of the Accounting pre- and post-tests due to ambiguity in
the wording of the question. Therefore, the final analysis of both Accounting tests included only 41
answers.
Participation by gender
Most of the groups are fairly equally-divided by gender, as we show in Table 1. The MBA Accounting group
is composed of 39 percent women and 61 percent men, and the Smartly Accounting group is composed
of 44 percent women and 53 percent men. For Finance, the MBA group contained 43 percent women and
58 percent men, while the Smartly group contained 57 percent women and 43 percent men. One
participant in each group did not indicate gender.
Participation by age
Smartly participants tended to be slightly younger than MBA participants, as we show in Table 2.
Whereas over 90 percent of participants in the MBA Finance and MBA Accounting groups were 25-34
years of age, the Smartly groups tended to be more evenly divided between ages 25-34 (53 percent
Accounting, 58 percent Finance) and ages 18-24 (44 percent Accounting, 37 percent Finance). Most
groups also had 1-2 participants in the range of 35-44 years old.
9
Table 1. Participation by gender
Accounting Finance
MBA Smartly MBA Smartly
Number of Number of Number of Number of

Gender Proportion participants Proportion participants Proportion participants Proportion participants
Women 39% 20 44% 15 43% 23 57% 24
Men 44% 31 53% 18 57% 31 43% 18
Total 51 33 54 100% 43

Table 2. Participation by Age
Accounting Finance
MBA Smartly MBA Smartly
Number of Number of Number of Number of

Age range Proportion participants Proportion participants Proportion participants Proportion participants
18-24 4% 2 44% 15 4% 2 37% 16
25-34 92% 47 53% 18 93% 50 58% 24
35-44 2% 1 0% 0 4% 2 4% 2
Not disclosed 2% 1 3% 1 0% 0 2% 1
Total 51 34 54 43

10
Findings
Smartly pre-course performance vs. MBA performance

All Smartly participants had at least a bachelor’s degree, so we expect that most of the participants in the
Smartly group would have had a lay knowledge of at least some of the concepts in each of the subject
areas being tested, but not as much knowledge as MBA students who had completed formal coursework
in these subjects. We compared pre-test scores for Smartly participants with post-test scores for MBA
participants to examine this issue.
As we show in Table 3, the mean score for the 34 Smartly Accounting participants is 56.7 percent and the
median is 53.7 percent, with a low score of 26.8 percent and a high score of 80.5 percent. For the 43
Smartly participants in finance, the mean score for Smartly participants is 49.2 percent and the median is
52.9 percent, with a low score of 11.8 percent and a high score of 82.4 percent.
We calculated the t-statistic for the test of the means of the Smartly participants versus the MBA
participants to ensure that the initial knowledge of the Smartly participants was different from that of the
MBA participants. We provide these results in Table 3. Our test of difference in means indicates that there
is a statistically significant difference between the mean pre-test scores for Smartly participants and the
mean post-test scores for MBA.

Table 3. Comparison of Smartly pre-test results with MBA group
Panel A: Accounting
Number of Mean Score Lowest

participants (Mean) Median score Highest Score
Smartly pre-test 34 57% 54% 27% 80%
MBA post-test 51 75% 76% 41% 93%
t-statistic for test of

6.40
difference in means,
(<0.0001)
p-value in ()
Panel B: Finance
Lowest
Number of s Mean score Median score Highest Score
Smartly pre-test 43 49.2% 53% 12% 82%
MBA post-test 54 82.4% 8% 29% 100%

-8.58
difference in means,
(<0.0001)
p-value in ()

11
Time spent in Smartly courses
Accounting participants spent an average of 147 minutes to complete all required lessons, with the
shortest time being 81 minutes and the longest time being 342 minutes. Finance participants spent an
average of 118 minutes to complete all required lessons, with the shortest time being 60 minutes and the
longest time being 186 minutes.
Course effectiveness: Pre-test to post-test Improvement for Smartly participants

On average, participants who took the Smartly course improved by 29 percentage points and 33
percentage points from pre-test to post-test for Accounting and Finance, respectively. All but one
participant in each group achieved a higher score at post-test than at pre-test. We provide the results of
the test of the mean of the differences between pre- and post-test scores for Smartly participants in
Table 4.
Table 4. Comparison of Smartly participants' pre- and post-test scores
Including participants completing both the pre- and post-tests
Mean score

difference pre- to
Smartly Number of Change from pre to post-test
course participants Pre-test Post-test post test p-value in ()
12.39
Accounting 34 57% 86% 29.1%
(<0.0001)
12.11
Finance 43 49.2% 82.4% 33.2%
(<0.0001)

Some Smartly participants did relatively well even at pre-test (with scores up to 80 percent in Accounting
and 82 percent in Finance), so we also examine whether pre-test performance was related to
improvement at post-test.
We split each Smartly group into low and high pre-test scores, and low and high improvement, using the
median scores as cutoffs. We calculated a chi-squared test of independence, with the result of a significant
relationship between pre-test performance and improvement for both Accounting and Finance.4 For both
Smartly groups, participants with lower pre-test scores were more likely to be in the high improvement
group. However, even participants who started with higher scores at pre-test improved, on average. This
result is consistent with reversion to the mean -- a statistical phenomenon in which high and low scorers
tend to get closer to the mean scores at re-test -- and with an overall learning effect for Smartly
participants.
We also take another, more robust approach to examining the scores for the Smartly group. We estimate
the regression of post-test scores on the pre-test scores, controlling for the time spent on the Smartly
material (Time). We estimate this regression for both the Finance and the Accounting Smartly groups, and

4
For Accounting, chi-squared statistic is 10.25 with one degree of freedom, resulting in a p < 0.01. For Finance, the
chi-squared statistic is 12.64 with one degree of freedom, and a p-value 0.01
12
provide the results in Table 5. The results are consistent with participants with higher pre-test scores
showing the most improvement in scores. For accounting participants, the greater the time spent on the
material, the greater the change in the test scores from pre-to-post-test.5

Table 5. Explaining score improvement
The dependent variable is the change in the test score. Estimated coefficients, with p-values in
parentheses below coefficients
Finance Accounting
Intercept 0.5398 0.63865
(<0.0001) (<0.0001)
Pre-test score 0.2842 0.3857
(0.0241) (0.0164)
Time 0.0013 0.0001
(0.0383) (0.8335)

R-squared 0.2041 0.1889
F-statistic 5.13 3.61

(0.0104) (0.0390)
Smartly Post-test performance vs. MBA performance

We provide the results of comparing the Smartly post-test scores with the MBA group scores for the
Accounting and Finance courses, respectively, in Table 6. As we indicate with a t-statistic to test the
difference in means scores between the Smartly group and the MBA group, the Smartly Accounting group
outperformed the MBA group, with the difference in mean scores significant at most reasonable levels of
significance. The Smartly Finance course participants score no differently than those in the MBA group, as
indicated by a p-value of the test statistic that is above reasonable levels of significance.

5
The pre-test score and time together explain 20 percent of the variation in test scores for Finance participants,
and almost 19 percent for Accounting participants.
13
Table 6. Comparison of post-test performance between Smartly and MBA groups
t-statistic (with p-value in

parentheses) for test of
Smartly MBA difference in mean scores
Assuming Assuming
Number of Number of equal unequal
Course Mean participants Mean participants variances variances
Accounting 86% 34 75% 51 3.95 3.97

(0.0002) (0.0002)
Finance 82% 43 81% 54 0.44 0.46

(0.6594) (0.6473)

Feedback from participants

All participants completing the post-test were asked if they thought the test was a fair assessment of their
subject matter knowledge in the respective areas. For Accounting, 85 percent of Smartly participants and
86 percent of MBA participants indicated that the test was a fair assessment. For Finance, 84 percent of
Smartly participants and 69 percent of MBA participants indicated that the test was a fair assessment of
their subject matter knowledge.6 We provide representative comments in Table 7 for Accounting groups,
and in Table 8 for Finance groups.

6
Only 26 of 54 the MBA Finance participants answered this question. The question for this group was sent out via
email a few weeks after the post-test was taken, so the lower response rate may have just been circumstantial and
should not be taken as an intentional unwillingness among this group to answer this question.
14
Table 7. Representative Comments: Test was fair assessment of accounting knowledge?
MBA Smartly
Yes ● All of the questions were relevant to ● All questions were very similar to the smartly
general accounting. modules and seemed to be a fair overall test
● Because it covered a wide variety of of knowledge.
questions and the information was ● I have not had a thorough accounting lesson
presented clearly. in many years, so going through the course
● It asked about concepts important to and taking this post-exam shows my current
accounting, especially those that are most understanding of what I retained.
frequently applied in real-world settings of ● It was comprehensive and included realistic
reading financials and analyzing examples, similar to what one working in
accounting transactions. The only thing accounting would have.
missing was testing my understanding of
what goes into different financial
statements (e.g., income statement, cash
flow statement, etc.).
No ● Some of the questions were poorly ● I have not had to apply it in a human-to-
worded. It was not clear what I was human interaction. My knowledge is
supposed to answer. circumstantial and I would not feel confident
● There's much more to it than sharing this knowledge in real life.
remembering easy-to-reference formulas ● Yes and no. While this assessment addressed
by heart. the basics, I do not believe it tackled the more
● I studied accounting with the case challenging lessons learned in Part 2 of the
method, learning to make tough study materials.
judgments about unique accounting ● I think it's pretty specific to the Smartly test
situations that actual companies face. I environment. I think there are a lot of things
don't remember T accounts, but I do that I know through experience that weren't
remember principles that would be explored or captured here, but some more
applied in actual situations. technical things like LIFO vs. FIFO that I was
unfamiliar with that are. I would say it's a
pretty good assessment based on the content
of the lectures.

15
Table 8. Representative Comments: Test was fair assessment of finance knowledge?
MBA Smartly
● It assessed basic building concepts of ● I was a blank slate for finance knowledge.
Yes finance that should be learnt during So anything I have or have not learned came
introductory courses. directly from this platform!
● The question itself was simple but require ● I was pleasantly surprised that the answers I
certain knowledge about finance. had zero clue about in the pre-assessment, I
● It captured the essentials any finance now understood.
professional should know.. Of course, this ● The questions pertained to the material
only scratches the surface, but you managed learned through the online modules. These
to capture the essentials. questions were not too difficult, but also did
not give away the answers.
● It looked more at specifics instead of general ● Because the questions seems very simple
No principles and similar to smartly content. It felt like the
● Finance is more about conceptual course taught to the test.
knowledge than just numbers.. This ● Simply taking a course online without any
assessment concentrated more on extra study time does not accurately reflect
quantitative stuff. my ability to take in and learn information.
● It was too basic The questions in the final assessment were
very detailed. It is hard to remember these
details from just taking an online course
once.
● There should be case based questions.
Smartly participant satisfaction (Net Promoter Score)

Smartly participants were asked (at the conclusion of the second test) to respond to a standard
satisfaction question, the Net Promoter Score (NPS) question, “How likely are you to recommend Smartly
to a friend?”, on a scale of 0 (Would not recommend) to 10 (Definitely would recommend). The NPS is
calculated as the percentage of respondents answering 9 or 10 (promoters), less the percentage of
respondents answering 0 to 6 (detractors), with scores ranging from -100 to 100.
The Smartly Accounting courses had 58 percent promoters and 6 percent detractors, receiving an NPS
score of 52. The Smartly Finance courses had 49 percent promoters and 9 percent detractors, receiving
an NPS score of 40. These scores are in line with recent NPS scores for the Harvard MBA program and
Wharton MBA program, with NPS of 41 and 51, respectively.7,8

7
Steven Hind, “What is HBS’s NPS?” The Harbus, February 5, 2015, (www.harbus.org/2015/the-harbus-survey-
says-hbs-mba-program-more-walgreens-than-apple/).
8
Natalie Kitroeff, “MBAs are more Satisfied at Wharton than at Harvard Business School”, Bloomberg, March 31,
2015 (www.bloomberg.com/news/articles/2015-03-31/mbas-are-more-satisfied-at-wharton-than-at-harvard-
business-school).
16
Conclusion
Many companies are looking for the skills developed while pursuing an MBA. However, the pursuit of an
MBA may not always be the right choice for all learners, due to financial, geographic, or time constraints.
This study supports the assertion that some of the foundational accounting and financial concepts taught
in traditional brick-and-mortar MBA programs can be learned independently, online through Pedago’s
targeted Smartly active-learning courses. Significant improvement in students’ knowledge is acquired in
as little as two hours of engagement with these courses.
In addition to being taken on their own as part of a comprehensive independent study MBA program,
these courses could be used as supplements to existing MBA programs. First, they could be used as
remedial materials for incoming MBA students who need to enhance their understanding of basic
concepts in certain subject matter areas. Second, they could be used by MBA instructors as pre-work for
blended-learning classes. By having students learn basic concepts and complete structured practice
exercises on their own, instructors could devote more classroom time to deeper learning through case
studies and group work.
Limitations of the study

The Smartly courses are still under development and few students have taken all of the courses that would
be equivalent to all of the coursework in a given MBA subject matter area. Therefore, it is not currently
possible to test comprehensive subject-matter knowledge. The tests used were developed in-house and
intended specifically to cover the concepts taught in the current courses.
Whereas the tests used were deemed by students to be a fair assessment of their knowledge, many MBA
students mentioned that the tests covered basic accounting principles, but did not cover other concepts
they had learned in their accounting courses. For future studies with students who have completed an
entire course series, it would be helpful to use a more-widely accepted test. First, such a test would allow
for the assessment of a broader set of knowledge. Additionally, this sort of assessment would allow for
comparison across a wider set of participants in traditional MBA programs. A more-widely accepted test
would permit more cross-sections of the group, as well as a deeper analysis.
Additionally, the timeframe for the study was compact. Therefore, the time between starting a course
and taking the post-test was relatively short, and may not represent the time that most students would
take to finish a given course. It would be useful to test this same group again in 3-6 months to see how
much of the course knowledge is retained over time.
Possible future studies
Comparisons to other online, self-study platforms

There are many free, online self-guided learning systems now available for students who are trying to
improve their knowledge skills at a time and place that is most convenient for them. Pedago has designed
Smartly’s learning system to provide a different experience from these other systems by focusing on “bite-
sized” lessons that incorporate active learning. It would be interesting to conduct pre-/post- evaluations
with students taking specific subject matter courses on other platforms with those completing the courses
in Smartly. This comparison would help to normalize some conditions that were not available in this study,
including independent vs. classroom-based learning, and digital vs. analog learning.
17
Item analysis
In this study, we did not examine learning outcomes for specific items on the test. However, we should
review this data based on the concepts being assessed. Concept-specific analyses of the Smartly
participant data is expected to highlight where students had the smallest gains in knowledge. Such results
might indicate that the lessons on those concepts either may need to be improved or that additional
lessons in those concepts may be required for the knowledge to be more fully acquired. Concept-specific
analyses of the MBA data might be used to identify the concepts least likely to be retained over time,
which could also help Pedago to make strategic decisions about how to structure their own content
related to those items.

18
Appendices
Appendix A: Eligibility funnel
Screen participants
(810)
ACCOUNTING FINANCE
SMARTLY SMARTLY
MBA MBA
Invite to Invite to
invited to Invited to
participate (93) participate (119)
participate participate
(107) (95)

SMARTLY
SMARTLY
take pre-test
Pre-test taken
(62)
(62)
SMARTLY SMARTLY
Coursework Start
started (49) coursework
started (47)
SMARTLY MBA SMARTLY MBA

Coursework Take Coursework Take post-
completed and post-test completed and test (54)
post-test taken (51) post-test taken
(40) (43)
SMARTLY MBA SMARTLY MBA

Analysis group Analysis Analysis group Analysis
(34) group (51) (43) group (54)
19
Appendix B: Accounting - Detailed results
Smartly Accounting group results (N = 34)
Mean Median Mode Minimum Maximum
Pre-test scores 57% 54% 54% 27% 80%
Post-test scores 86% 89% 98% 49% 100%
% Change 59% 46% 38% -9% 191%
Point Change 29 27 27 -5 61
Smartly Accounting individual participant results (N = 34)

Minutes to
complete
Smartly Retook
Student Pre-test Post-test % change Point change course Quizzes
Smartly Acctg Student 1 34% 61% 79% 27 135 Yes
Smartly Acctg Student 3 78% 98% 25% 20 126 No
20

Smartly Accounting individual participant results (N = 34) (cont’d)

Minutes to
complete
Smartly Retook
Student Pre-test Post-test % change Point change course Quizzes
Smartly Acctg Student 30 54% 49% -9% -5 127 Yes
21
MBA Accounting group results (N = 51)
Mean Median Mode Min Max
Post-test scores 75% 76% 88% 41% 93%
MBA Accounting individual participant results (N = 51)

MBA Accounting Post-test MBA Accounting Post-test MBA Accounting Post-test
Student score Student score Student score
MBA Acctg Student 1 93% MBA Acctg Student 18 88% MBA Acctg Student 35 88%
22
Appendix C: Finance - Detailed Results
Smartly Finance group results (N = 43)
Pre-test scores 49% 53% 65% 12% 49%
Post-test scores 82% 82% 88% 41% 82%
% Change 94% 60% 50% 0% 94%
Point change 33 35 29 0 33
Smartly Finance individual participant results (N = 43)

Minutes to
Point complete Retook
Student Pre-test Post-test % change change Smartly course quizzes
Smartly Finance Student 1 24% 88% 275% 65 164 No
Smartly Finance Student 9 29% 41% 40% 12 106 Yes
23

24
Smartly Finance individual participant results (N = 43) (cont’d)
Minutes to
Point complete Retook
Student Pre-test Post-test % change change Smartly course quizzes
25

MBA Finance participant results (N = 54)

Post-test scores 81% 88% 100% 29% 100%
MBA Accounting individual participant results (N = 54)
MBA Finance Student Post-test MBA Finance Student Post-test MBA Finance Student Post-test
score score score
MBA Finance Student 1 82% MBA Finance Student 19 65% MBA Finance Student 37 71%
26
Appendix D: Details on tests
Accounting tests
Question 1
Concept Question Answer options Correct answers Answer Question Answer Correct
Tested (Pre-test) (Pre-test) (Pre-test) # (Post-test) options (Post- answers
test) (Post-test)
Balance Select the equation that is true of all assets = liabilities + A1
assets = liabilities + Same as pre-test
sheet balance sheets. equity; equity =
equity
liabilities + assets;
liabilities = assets +
equity
Question 2
Concept Question Answer options Correct answers Answer # Question Answer Correct
Tested options answers
Balance Label the following balance sheet Asset; Liabilities; Assets A2
Same as pre-test
sheet accounts as assets, liabilities, or Equity Liabilities A3
equity. Assets A4
Assets A5
Accounts Receivable; Bank Loan Equity A6
Payable; Cash; Inventory; Paid-In Assets A7
Capital; Prepaid Expenses; Assets A8
Property, Plant & Equipment; Equity A9
Retained Earnings
27
Question 3
Tested (Pre-test) (Pre-test) (Pre-test) (Post-test) options answers
(Post-test) (Post-test)
Balance Money that a company owes to its Accounts Receivable; Accounts Payable A10
Same as pre-test
sheet vendors is recorded in the account Accounts Payable;
_____________________________. Allowance for
Net income that is reinvested in the Doubtful Accounts; Retained earnings A11
company is known as Credits; Current;
_____________________________. Debits;
Liabilities and assets that are Noncurrent/Long- Current A12
expected to be paid off or converted term; Retained
into cash within one year are said to Earnings
be
_____________________________.
Money due to a company from its Accounts Receivable A13
customers is recorded in the account
_____________________________.
Liabilities and assets with a longer Non-current/long-term A14
timeframe are said to be
_____________________________.
In T- Debits A15
accounts,_______________________
______ always go on the left.
In T-accounts, Credits A16
_____________________________
always go on the right.
Question 4
Tested (Pre-test) (Pre-test) (Pre-test) (Post-test) options (Post- answers
test) (Post-
test)
General Which of the following general Cash; Expenses; Net A17
Expenses; Revenues Same as pre-test
ledger ledger accounts are closed out at the Income; Retained
end of each accounting period? Earnings; Revenues
(Select all that apply.)
28
Question 5
Tested (Pre-test) (Pre-test) (Pre-test) (Post-test) options (Post- answers
test) (Post-
test)
General For each type of account below Plus; Minus Plus A18
Same as pre-test
ledger determine whether the left side of Minus A19
the balance sheet records increases Minus A20
(+) or records decreases (-). Minus A21
Plus A22
Asset Account; Liability Account;
Equity Account; Revenues; Expenses
Question 6
Tested (Pre-test) (Pre-test) (Pre-test) (Post-test) options answers
Journal A delivery of merchandise was made 1; 2; 3; 4; 5; Doesn't 2 A23
Same as pre-test
to a customer. The merchandise cost Belong 1 A24
the seller $3,500 to make. The 4 A25
customer paid $1,500 and owes 5 A26
another $4,000. In which order do 3 A27
the following terms appear in the first Doesn't belong A28
column of the balance sheet above?
(One account doesn’t belong.).

Accounts Receivable; Cash; Expenses;
Inventory; Revenues; Deferred
Revenue
Question 7
Tested (Pre-test) (Pre-test) (Pre-test) (Post- options (Post- answers
test) test) (Post-test)
Conservatism According to the an order is placed it is reasonably A29
Same as pre-test
principle conservatism concept, cash is received and not before certain that they
revenues are recorded inventory is ready to be sold will be collected
when: it is reasonably certain that they
will be collected
29
Question 8
Tested (Pre-test) (Pre-test) (Pre-test) # (Post-test) options (Post- answers
test) (Post-test)
Conservatism Expenses are recorded as Incurred; paid with cash; incurred A30
Same as pre-test
principle soon as they are: projected in a proposal;
reasonably certain
Question 9
Concept Question Answer options Correct Answer Question Answer Correct
Tested (Pre-test) (Pre-test) answers # (Post-test) options (Post- answers
(Pre-test) test) (Post-test)
Manufacturing Last year, a company had $3 million ÷ $1.5 million = $3 million ÷ A31 Last year, a company had $4.5 million ÷ $4.5 million
Overhead $3 million in production $2; $1.5 million = $4.5 million in production $1.5 million = ÷ $1.5
overhead and spent $1.5 $1.5 million ÷ $3 million = $2 overhead and spent $1.5 $3; $1.5 million = $3
million on direct labor $0.50; million on direct labor million ÷ $4.5
costs. Using labor cost as $3 million × $1.5 million = costs. Using labor cost as million =
its basis, select the correct $4.5 million; its basis, select the correct $0.33; $4.5
overhead rate. ($3 million − $1.5 million) overhead rate. million × $1.5
÷ $1.5 million = $1; million =
$6.75 million;
($4.5 million −
$1.5 million) ÷
$1.5 million =
$2
Question 10
Tested (Pre-test) (Pre-test) answers # (Post-test) options (Post- answers
(Pre-test) test) (Post-
test)
Depreciation A fixed asset that cost open-ended 40,000 A32 A fixed asset that cost open-ended 50,000
$200,000 has a service life $300,000 has a service life
of 5 years. Using straight- of 6 years. Using straight-
line depreciation, what will line depreciation, what will
the annual depreciation the annual depreciation
expense be? expense be?
30

Question 11
Tested (Pre-test) (Pre-test) answers # (Post-test) options answers
(Pre-test) (Post-test) (Post-test)
Journal An asset has accumulated its entire 1; 2; 3; 4 2 A33 An asset has accumulated its entire 1 2
$4 million cost in depreciation over a 3 A34 $3 million cost in depreciation over 2 3
10-year period. A scrap dealer A35 a 10-year period. A scrap dealer 3
1 1
purchases the asset for $5,000. Place purchases the asset for $4,000. 4
4 A36 4
the accounts below in the correct Place the accounts below in the
order that they would appear in the correct order that they would
journal appear in the journal.
[Accumulated Depreciation]
Accumulated Depreciation; Gain on [Gain on Asset Disposal]
Asset Disposal; Cash; PP&E [Cash]
[PP&E]
Question 12 (Excluded from analysis)

Tested (Pre-test) (Pre-test) (Pre-test) # (Post-test) options answers
Inventory A store purchased inventory three LIFO; FIFO Method X X1 A store purchased inventory three LIFO Method X
times in June. 45 units of that times in June. 45 units of that FIFO
inventory were sold. In the journal inventory were sold. In the journal
above, which corresponds to each above, which corresponds to each
of the following inventory cost of the following inventory cost
methods? [Method X] methods? [Method X]
A store purchased inventory three LIFO; FIFO Method Y X2 A store purchased inventory three LIFO Method Y
times in June. 45 units of that times in June. 45 units of that FIFO
inventory were sold. In the journal inventory were sold. In the journal
above, which corresponds to each above, which corresponds to each
of the following inventory cost of the following inventory cost
methods? [Method Y] methods? [Method Y]
31

Question 13
Concept Question Answer Correct Answer # Question Answer Correct
Tested (Pre-test) options answers (Post-test) options answers
(Pre-test) (Pre-test) (Post-test) (Post-test)
Journal A craftsman purchased $700 A; B; C B A37 A craftsman purchased $900 A B
worth of materials on credit. worth of materials on credit. B
She finished crafting a piece of She finished crafting a piece C
merchandise worth $200 and of merchandise worth $300
displayed it in her store with a and displayed it in her store
price tag of $450. Select which with a price tag of $600.
of the transactions above need Select which of the
to be recorded. transactions above need to
be recorded.
Question 14
Concept Question Answer options Correct Answer # Question Answer options Correct
Tested (Pre-test) (Pre-test) answers (Post-test) (Post-test) answers
(Pre-test) (Post-test)
Journal Complete the journal entries $4,000; $6,000; $4,000.00 A38 Complete the journal $4,000 $9,000.00
for selling $2,000 of inventory $2,000; $8,000 entries for selling $5,000
for $4,000 on credit. $5,000 of inventory $14,000
Reference this image. for $9,000. $9.000
Blank A; Blank B; Blank C; Cash; Revenues; revenues A39 [Reference image] Cash revenues
Blank D Inventory; Revenues
Deferred Inventory
Revenue Deferred Revenue
$4,000; $6,000; 2,000 A40 $4,000 $5,000.00
$2,000; $8,000 $5,000
$14,000
$9.000
Cash; Revenues; Inventory A41 Cash; Revenues; Inventory
Inventory; Inventory;
Deferred Deferred Revenue
Revenue
32
Accounting Test – Demographic and Opinion Questions
Pre-test questions
Question Question Answer options Participants Answer
# #
Question Have you ever completed an accounting course? Yes, in an MBA program; Yes, as an undergraduate or Smartly A42
15 in a graduate degree program other than an MBA; No,
I have never completed an accounting course
Post-test questions
# #
Question Do you think this was a fair assessment of your accounting knowledge? Yes; No Smartly and A43
15 MBA
Question Why or why not? A44
open-ended
15
Question What did you think of the Smartly course experience? Please tell us Smartly A45
open-ended
16 anything positive or negative about the Smartly courses that you took.
Question How likely is it that you would recommend Smartly to a friend or A46
open-ended
17 colleague?
33
Finance tests
Question 1
Concept Question Answer options Correct Answer # Question Answer options Correct
Tested (Pre-test) (Pre-test) answers (Post-test) (Post-test) answers
Time Value Discounting money allows A1 Compounding money
open-ended present value open-ended future value
of Money you to calculate a allows you to calculate a
__________. __________.
Question 2
Concept Question Answer options Correct Answer # Question Answer options (Post- Correct
Tested (Pre-test) (Pre-test) answers (Post-test) test) answers
Time Value Which of the above A: CF0 x (1 + r)n A2 Which of the A: rn + CF0 x (rt - rn)
C B
of Money formulae calculates the B: CF0 x rn above formulae B: CF0 x (1 + r)n
present value of a future C: CFn / (1 + r)n A3 calculates the C: CFn / (1 + r)n
cash flow? D: ern / CFn A future value of a D: CFn x ern C
present cash
flow.
Question 3
Concept Question Answer options Correct answers Answer # Question Answer options (Post- Correct
Tested (Pre-test) (Pre-test) (Pre-test) (Post-test) test) answers
(Post-test)
Time Value Which of the A: ($6,000 x 1.05) A4 Which of the A: ($4,000 x 1.06)/5
net present value D
of Money expressions above / 4 expressions above tells B: ($4,000 x (1.06)5
tells us how much B: $6,000 x us how much $4,000 C: 1.065/$4,000
$6,000 will be in 4 (1.05)4 was worth 5 years ago D: $4,000/1.065
years assuming it C: 1.054 / $6,000 assuming it had grown
grows at 5% annually? D: $6,000 / 1.054 at 6% annually?

Question 4
34
Concept Question Answer options Correct answers Answer # Question Answer options (Post- Correct answers (Post-
Tested (Pre-test) (Pre-test) (Pre-test) (Post-test) test) test)
Net present The present value of net present; A5 What is net the present value of
net present value the present value of
value benefits (cash inflows) value; gross present benefits (cash inflows)
benefits (cash inflows)
minus the present margin; revenue; value? minus the present value
minus the present value
value of costs (cash operating profit of costs (cash outflows);
of costs (cash outflows)
outflows) is known as the present value of costs
________________. (cash outflows) minus the
present value of benefits
(cash inflows); the future
value of benefits (cash
inflows) minus the
present value of costs
(cash outflows); the
future value of costs (cash
outflows) minus the
present value of benefits
(cash inflows)
Question 5
(Post-test)
Cost of Match the source creditors A6 Match the source with creditors
creditors owners
capital with each type of owners each type of funding. owners
funding. [debt A7 [debt funding]
funding] owners creditors
Question 6
(Post-test)
Cost of Interest payments cost of debt; A8 Before taxes are Interest payments;
cost of debt interest
capital divided by debt will cost of equity; factored in, cost of debt dividends;
payments
calculate a company's return on debt; is equal to principal payments;
before-tax debt margin ______________ net income;
__________________. divided by debt.

Question 7
35
(Post-test)
Cost of If B is the cost of debt B x T; A9 If B is the cost of debt B x T;
B × (1 − T) B × (1 − T)
capital before taxes and T is B x (T-1); before taxes and T is the B x (T-1);
the effective tax rate, B x (1-T); effective tax rate, what B x (1-T);
what is the formula T x (B-1) is the formula for T x (B-1)
for computing the computing the after-tax
after-tax cost of debt? cost of debt?
Question 8
Concept Tested Question Answer options Correct Answer # Questio Answer Correct
(Pre-test) (Pre-test) answers n options answer
(Pre-test) (Post- (Post- s (Post-
test) test) test)
Cost of capital Which of the the growth rate of the economy as a whole; A10
the 10-year Same as pre-test
following is most the 10-year U.S. Treasury Constant Maturity Rate;
U.S. Treasury
likely to be used as a the average of ten years of monthly returns of the
Constant
risk-free rate? S&P 500;
Maturity Rate
a company's beta times the growth rate of the
stock market as a whole
Question 9
Concept Question Answer Correct Answer # Question Answer options (Post-test) Correct
Tested (Pre-test) options answers (Post-test) answers
(Pre-test) (Pre-test) (Post-test)
Cost of The volatility of a particular stock A11 how volatile a particular stock is
alpha (α); beta (β)
capital compared to the market as a compared to the market as a
how
whole is known as beta (β); whole.;
volatile a
_________________. change in the market as a whole
gamma (γ); particular
Beta (β) is a from day to day;
stock is
delta (δ) measure of change in the market as a whole
compared
_________________ from the same day the previous
to the
year;
market as
how volatile a particular stock is
a whole.
compared to its performance
during the last quarter
36
Question 10
Concept Question Answer options Correct Answer Question Answer options (Post-test) Correct
Tested (Pre-test) (Pre-test) answers # (Post-test) answers
Capital asset If the risk-free rate is 2% + 1.6 × (13% − 2%) A12 If the risk-free rate is 3% + 0.6 × (13% − 3%) = 9%;
2% + 1.6 × 3% + 0.6 ×
pricing 2%, the expected = 19.6%; 3%, the expected 0.6 × (13% − 3%) = 6%;
(13% − 2%) = (13% − 3%)
model market return is 13%, 1.6 × (13% − 2%) = market return is 0.6% + (13% − 3%) = 10.6%;
19.6% = 9%
and β is 1.6, what is 17.6%; 13%, and Î² is 0.6, 13% + 0.6 × (13% − 3%) = 19%
the cost of equity 1.6% + (13% − 2%) = what is the cost of
using CAPM? 12.6%; equity using CAPM?
13% + 1.6 × (13% −
2%) = 30.6%
Question 11
Concept Question Answer options Correct answers Answer Question Answer options (Post- Correct
Tested (Pre-test) (Pre-test) (Pre-test) # (Post-test) test) answers
(Post-test)
Cost of A company with a leveraged; A13 A company that is highly leveraged debt;
leveraged debt
capital high proportion of profitable; has a high proportion of equity
debt is said to be solvent; ___________.
highly ___________. liquid
Question 12
Concept Question Answer options Correct Answer Question Answer options (Post-test) Correct
Tested (Pre-test) (Pre-test) answers # (Post-test) answers
Weighted What is the weighted (11% × 40%) + (7% × 60% × A14 What is the weighted (12% × 60%) + (6% × 40% ×
(11% × (12% ×
average average cost of capital 75%) = 7.55%; average cost of capital 70%) = 8.88%;
40%) + 60%) + (6%
cost of (WACC) of a company (11% × 60%) + (7% × 40% × (WACC) of a company that (12% × 40%) + (6% × 60% ×
(7% × × 40% ×
capital that has an effective tax 75%) = 8.70%; has an effective tax rate of 70%) = 7.32%;
60% × 70%) =
rate of 25%, cost of debt (11% × 40%) + (7% × 60% × 30%, cost of debt of 6%, (12% × 60%) + (6% × 40% ×
75%) = 8.88%
of 7%, cost of equity of 25%) = 5.45%; cost of equity of 12%, 40% 30%) = 7.92%;
7.55%
11%, 40% equity (11% × 60%) + (7% × 40% × debt financing, and 60% (12% × 40%) + (6% × 60% ×
financing, and 60% debt 25%) = 7.30% equity financing? 30%) = 5.88%
financing?
37
Question 13
Concept Question Answer options Correct answers Answer Question Answer options Correct
Tested (Pre- (Pre-test) (Pre-test) # (Post-test) (Post-test) answers
test) (Post-test)
Valuation What is A15
the present value at a point in time (whether now the present value at a Same as pre-test
the
or in the future) of all cash flows from that point; a point in time (whether
terminal
company's cost of capital weighted according to its now or in the future) of all
value?
capital structure; the return investors can expect cash flows from that point
to receive by investing in the stock market as a onwards
whole; the total market value of a company's
assets
Question 14
Concept Question Answer options Correct Answer Question Answer options Correct answers (Post-test)
Tested (Pre-test) (Pre-test) answers # (Post-test) (Post-test)
(Pre-test)
Valuation Using the income A16 Using the income statement
$168,938 × (1 − $168,938 × $956 × (1 − 28%) ($956 − $201) × (1 − 28%) =
statement above and above and an effective tax
26%) = $125,014.12; (1 − 26%) = = $688.75; $543.60
an effective tax rate of rate of 28%, calculate the
$125,014.12
26%, calculate the ($168,938 − company's net operating ($956 − $201) ×
company's net $24,415) × (1 − 26%) profit after tax (NoPAT). (1 − 28%) =
operating profit after = $106,947.02; (This is also called net $543.60;
tax (NoPAT). (This is operating profit less
also called net $70,975 × (1 − 26%) adjusted taxes, or NoPLAT.) $518 × (1 − 28%)
operating profit less = $52,521.50; = $372.96;
adjusted taxes, or ($168,938 − ($956 − $237) ×
NoPLAT.) $73,548) × (1 − 26%) (1 − 28%) =
= $70,588.60 $517.68
Question 15
Concept Question Answer options Correct answers Answer Question Answer options Correct answers
Tested (Pre-test) (Pre-test) (Pre-test) # (Post-test) (Post-test) (Post-test)
Valuation If FCF stands for free cash flow, N is the A: FCFN+1 x (r-g) A17
D Same as pre-test
year for which the terminal value is B: FCFN+1 x (r+g)
calculated, r is the discount rate, and g is C: (FCFN+1 ) / (r+g)
the terminal growth rate, which of the D: (FCFN+1) / (r-g)
formulae above will calculate the Note: FCFN+1 + FCFN x
terminal value? (1 + g)
38

Finance – Demographic and Opinion questions
Question 16 (Pre-test only)
# (Pre-test) #
Question Have you completed financial accounting and corporate Yes, I have completed financial accounting in an MBA program; Smartly only A18
16 finance (or equivalent courses) in an MBA program? Yes, I have completed corporate nance in an MBA program.;
No, but I have completed financial accounting or corporate finance
in an undergraduate or graduate program other than an MBA.;
No, I have not completed either of these courses (or their
equivalents).
Question Have you completed any other finance courses besides the A19
open-ended
17 ones above in an MBA or another program?
Post-test questions
Question # Question Answer options Participants Answer
(Pre-test) #
Question Do you think this was a fair assessment of your finance knowledge Yes; No Smartly A20
16 [gained through Smartly]? Only
Question Why or why not?
open-ended
17 A19
Question What did you think of the Smartly course experience? Please tell us A20
open-ended
18 anything positive or negative about the Smartly courses that you took.
Question How likely is it that you would recommend Smartly to a friend or A21
0,1,2,3,4,5,6,7,8,9,10
19 colleague?
Question Were there topics covered in this assessment that you were not exposed Yes; No MBA Only A24
18 to during business school?
Question Please explain more A25
Open-ended
19
Question Have you completed financial accounting and corporate finance (or Yes; A22
20 equivalent courses) in an MBA program?
No
Question Have you completed any other finance courses besides the ones above in Yes; A23
21 an MBA or another program?
No
39
Appendix E: Screeners
1. Undergrad location
2. Undergrad School Acceptance Rate
3. What is your name?

4. What is your email address? (We will contact you at this email address) - Open-Ended Response
5. Have you started or completed any course on Smartly?
6. Version 1 - Do you have an undergraduate degree? Version 2 & 4 - Do you have an undergraduate
degree or are you currently enrolled in an undergraduate program?
7. From what university did you receive your undergraduate degree? - Open-Ended Response
8. What year did you graduate from your undergraduate program?
9. What was your major? - Open-Ended Response
10. Version 1 - Do you have an MBA (or are you currently enrolled)? OR Version 2 & 4 - Have you ever
been accepted into an MBA program?
11. Version 1 - Do you have a non-MBA graduate degree? If yes, please specify institution, degree,
subject studied, and year of graduation. OR Version 2 & 4 - Do you have or are you pursuing a
graduate degree that is not an MBA? If yes, please specify institution, degree, subject studied, and
year of graduation.
12. Version 1 - Do you have a non-MBA graduate degree? If yes, please specify institution, degree,
subject studied, and year of graduation. - Please specify institution, degree, subject studied, and year
of graduation OR Version 2 & 4 - Do you have or are you pursuing a graduate degree that is NoT an
MBA? If yes, please specify institution, degree, subject studied, and year of graduation. - Please
specify institution, degree, subject studied, and year of graduation
13. Have you ever completed any accounting courses outside of an MBA program? If yes, please list
them, and specify at which institution(s) the courses were taken.
14. Have you ever completed any accounting courses outside of an MBA program? If yes, please list
them, and specify at which institution(s) the courses were taken. - Please specify courses and at what
institutions the courses were taken.
15. Version 4 ONLY - Have you ever completed any finance courses outside of an MBA program? If yes,
please list them, and specify at which institution(s) the courses were taken.
16. Version 4 ONLY - Have you ever completed any finance courses outside of an MBA program? If yes,
please list them, and specify at which institution(s) the courses were taken. - Please specify courses
and at what institutions the courses were taken.
17. Version 4 ONLY - How many finance courses did you complete during business school and what were
they?
18. Version 4 ONLY - How many finance courses did you complete during business school and what were
they? - The names of the finance courses I completed are:
40
19. At what university did you receive or are you pursuing your MBA? - Open-Ended Response
20. What is your year of graduation from your MBA program?
21. How many accounting courses did you complete during business school and what were they?
22. How many accounting courses did you complete during business school and what were they? - The
names of the accounting courses I completed are:
23. Since graduating from your undergraduate institution, how many years of work experience do you
have?
24. What is your current position? - Open-Ended Response
25. In what country do you currently live?
26. In what country do you currently live? - Other (please specify)
27. What is your country of citizenship (list all that apply)? - Open-Ended Response
28. What is your age?
29. What is your gender?
30. Which race/ethnicity best describes you? (Please choose only one.)
31. Which race/ethnicity best describes you? (Please choose only one.) - Multiple ethnicity / Other
(please specify)

41
Appendix F: Course Syllabi - Accounting I & II (128 minutes)
Accounting I
Chapter 01: Balance Sheet Basics (31 minutes)
Lesson Lesson Title Concepts taught Estimated

time to
complete
Lesson 1 Introduction to the Assets, liabilities, and equity; The dual-aspect concept 7 minutes
Balance Sheet
Lesson 2 Fundamental The entity and money-measurement concepts; Double-entry 6 minutes

Principles of bookkeeping; Which items you would like to see on a balance sheet;
Accounting Measuring assets at either fair value or cost
Lesson 3 All about Assets Grouping assets into Current and Noncurrent Assets on the balance 6 minutes
sheet; Current assets, including Accounts Receivable, Inventory, and
Prepaid Expenses; Noncurrent assets, including PP&E; The going-
current concept
Lesson 4 Liabilities & Equity Current and noncurrent liabilities, including Accounts Payable, Bank 6 minutes
Loan Payable, and Estimated Tax Liability; Equity, including Retained
Earnings and Total Paid-In Capital.
Lesson 5 Sew You Think You Apply what you've learned about the balance sheet to a new business 6 minutes
Can Knit: Smartcase
Chapter 02: The Practice of Accounting (37 minutes)
Lesson # Lesson Title Concepts taught Estimated

time to
complete
Lesson 1 Debits & Credits How to use T-accounts to keep track of every transaction a business 5 minutes
makes; Where to record debits and credits on a T-account
Lesson 2 Revenues & Expenses How to read and record transactions in Revenue and Expense 5 minutes
accounts
Lesson 3 Income Statements How to calculate net income; How to create an income statement 5 minutes
Lesson 4 General Journal How to record transactions in the general journal 4 minutes
Lesson 5 General Ledger How to keep track of all account balances with the general ledger; 6 minutes
How to transfer transactions from the general journal to the general
ledger; How to close out revenue and expenses; How to create a
balance sheet and income statement using the general ledger
Lesson 6 Practice Makes Review debits and credits, T-accounts, Revenues and Expenses, 6 minutes
Perfect journals, and the general ledger
Lesson 7 Wrestling with Apply what you've learned about the balance sheet, income 6 minutes
Accounts statement, general journal, and general ledger to a new business
42
Accounting II
Chapter 01: Realizing Revenues and Expenses (27 minutes)

time to
complete
Lesson 1 Revenue and cash Cash and revenue do not measure the same thing; Revenues are 6 minutes
recorded when they are reasonably certain; revenues are realized
when products are delivered
Lesson 2 Expenses and the Expenses are matched with the revenues they generate; The 6 minutes
matching concept difference between expenditures and expenses.
Lesson 3 Realizing Revenues When to recognize revenues and expenses that span multiple periods. 5 minutes
and Expenses over
time
Lesson 4 Bad debt How to handle bad debt; Making provision for doubtful accounts 6 minutes
Lesson 5 Furnishing the A review of the conservatism, realization, and matching concepts 4 minutes
Accounts: Smartcase
Chapter 02: Accounting for Assets (33 minutes)

time to
complete
Lesson 1 Inventory and the How to calculate cost of goods sold (COGS); First-in, first-out(FIFO) 6 minutes
Cost of Good Sold versus last-in, last-out(LIFO)
Lesson 2 Manufacturing How to account for the manufacture of inventory; How to assign cost 5 minutes
Inventory of inventory
Lesson 3 Fixed assets & Review non-current and tangible assets; Understand straight-line 7 minutes
straight-line depreciation; Learn how salvage value affects depreciation
depreciation
Lesson 4 Other depreciation How to use units-of-production and accelerated depreciation 6 minutes
methods methods; The implications of accelerated depreciation and taxes;
Disposal of fixed assets
Lesson 5 Depletion & Learn when to use depreciation, depletion, and amortization; 5 minutes
Amortization Calculate depletion and amortization
Lesson 6 Golden Assets: Review the accounting of inventories and depreciation 4 minutes
Smartcase
43
Appendix G: Course Syllabi - Finance (111 minutes)
Finance: The Time Value of Money
Chapter 01: The Time Value of Money (Part I) (24 minutes)
Lesson Lesson Title Concepts taught Estimated

time to
complete
Lesson 1 The Time Value of The value of money depends on time; What the present value of a 5 minutes
Money cash flow is; What the future value of a cash flow is.
Lesson 2 The Timeline Basic conventions used to solve time value of money questions; How 6 minutes
to create timelines for solving time value of money problems.
Lesson 3 Moving Money The difference between compounding and discounting; How to 6 minutes
Through Time: calculate the future value (FV) of a lump sum of money in one year’s
Compounding time.
Lesson 4 Moving Money How to calculate the present value (PV) of a lump sum of money 7 minutes
Through Time: from one year to the future; How to quickly distinguish between the
Discounting effects of compounding and discounting.
Chapter 02: The Time Value of Money (Part II) (29 minutes)

time to
complete
Lesson 1 Moving Money How to solve future value problems over multiple years; How the 6 minutes
Forward Over compounding formula and timeline change when multiple years are
Multiple Years involved.
Lesson 2 Moving Money How to solve present value problems over multiple years; How the 5 minutes
Backward Over discounting formula and timeline change when multiple years are
Multiple Years involved.
Lesson 3 Moving Cash Flows How to move a series of cash flows to the present: How to move a 7 minutes
Through Time series of cash flows to the future.
Lesson 4 Net Present Value What net present value (NPV) is; How to calculate net present value 7 minutes
(NPV); How to use net present value (NPV) to make investment
decisions
Lesson 5 Fair Trade Finance Review the basic concepts for moving money through time; How to 4 minutes
(SMARTCASE) use TVM concepts to make investment decisions.
44
Cost of Capital I - Capital Structure
Chapter 01 - Cost of Capital (23 minutes)

time to
complete
Lesson 1 Cost of Debt What cost of debt is; Variables that influence the cost of debt; How 6 minutes
to compute the cost of debt.
Lesson 2 Cost of Equity What cost of equity is; The relationship between risk and reward; 6 minutes
Variables that influence the cost of equity; Using the Capital Asset
Pricing Model to estimate cost of equity.
Lesson 3 Capital Structure and What capital structure is; How to calculate the weighted average 7 minutes
WACC cost of capital; How companies and investors use WACC to
determine investment plans.
Lesson 4 Capital Construction Real life applications of the cost of debt, cost of equity (CAPM), 4 minutes
(SMARTCASE) capital structure, and Weighted Average Cost of Capital formulas.
Valuation I - Discounted Free Cash Flow
Chapter 01 - Discounted Free Cash Flow Valuation (35 minutes)

time to
complete
Lesson 1 Discounted Free Cash Flow Basic steps for free cash flow valuation 5 minutes
Valuation
Lesson 2 Free Cash Flow What free cash flow is; Why we use free cash flow in 5 minutes
discounted cash flow (DCF) valuation.
Lesson 3 Calculating Free Cash Flow How to calculate free cash flow; How to calculate net 7 minutes
operating profit after tax; Where to find relevant free cash
flow information on a company’s financial statements.
Lesson 4 Forecasting Free Cash Flow How to forecast free cash flow; How to use a multi-stage 6 minutes
growth forecast; How to calculate terminal value.
Lesson 5 Free Cash Flow Valuation Learn to discount free cash flows to the present; Use WACC in 6 minutes
a free cash flow valuation.
Lesson 6 How Much Is That Cat in Test your knowledge of discounted free cash flow valuation 6 minutes
the Catalog? (SMARTCASE)
45
Appendix H: Fair assessment of subject matter knowledge? - Student comments
MBA Accounting participant comments

Fair? Comment Participant
Yes All of the questions were relevant to general accounting. MBA Acctg Student 27
Yes Basic knowledge focused around credit, debit, assets, liabilities, and retained MBA Acctg Student 37
earnings. Only concern is if you did not recall which side possessed the credits
and which the debits you could reverse most of the exam
Yes Basic questions asked although some parts of my knowledge needed to be MBA Acctg Student 41
refreshed.
Yes Because it covered a wide variety of questions and the information was MBA Acctg Student 34
presented clearly.
Yes Consistent to topics learned although I am rusty. Basic level accounting metrics MBA Acctg Student 17
and assessment. Thank you!
Yes covered basic accounting principles that we'd learned MBA Acctg Student 29
Yes Covered most all of the general accounting knowledge that was taught in my MBA Acctg Student 50
MBA accounting course.
Yes covers topics reviewed in class; format of some questions is challenging to follow MBA Acctg Student 33
and different than typical format in class.
Yes Did it off the top of my head MBA Acctg Student 15
Yes Fair, clear questions. MBA Acctg Student 14
Yes Felt comprehensive and straightforward MBA Acctg Student 40
Yes For a person who took only several accounting courses, this assessment is fair MBA Acctg Student 20
because they are not tricky questions.
Yes Great Questions MBA Acctg Student 42
Yes Having taken accounting courses before, I was somewhat familiar with the terms MBA Acctg Student 49
and concepts. Even if I was unfamiliar with the terms and concepts, this would
have been a fair assessment as I would immediately know what areas I need to
develop.
Yes I am not an accountant, so can only answer the most basic accounting questions MBA Acctg Student 32
without references. Lack of practice has led to some erosion.
Yes I guess, it is best to test knowledge when you are not just ramming it up and MBA Acctg Student 12
trying to understand the concepts. This conceptual check was good. Better would
be if i get to know how i performed and which areas i need to focus more on.
Yes I have an accountant and CFO for this ;) MBA Acctg Student 38
Yes I recognized all the terms MBA Acctg Student 26
Yes I remembered somethings, and forgot others. So I would say it was fair. MBA Acctg Student 11
46
MBA Accounting participant comments (cont’d)
Yes In terms of the MBA first-semester core finance class, yes; however, managerial MBA Acctg Student 36
accounting (rather than the fundamentals of T-accounts, etc.) seems to be more
of an emphasis
Yes It answered questions that I should've learned in my accounting class. MBA Acctg Student 19
Yes It asked about concepts important to accounting, especially those that are most MBA Acctg Student 35
frequently applied in real-world settings of reading financials and analyzing
accounting transactions. The only thing missing was testing my understanding of
what goes into different financial statements (e.g., income statement, cash flow
statement, etc.).
Yes it asked the fundamentals that I learned in Accounting I MBA Acctg Student 45
Yes It covered all the basic aspects on Accounting in a succinct way. MBA Acctg Student 7
Yes it covered many concepts MBA Acctg Student 48
Yes It covered the topics I learned during the last quarter, and I was able to answer MBA Acctg Student 23
with reasonable level of confidence.
Yes It covers the basic accounting concepts one should know. MBA Acctg Student 43
Yes It seems pretty fair for a short, basic assessment. However, it does not cover MBA Acctg Student 46
many of the other concepts learned in accounting.
Yes It tested basic accounting principles MBA Acctg Student 8
Yes It was a fairly good assessment but in the real world, one can always refer to the MBA Acctg Student 30
basics again.
Yes It was basic accounting, but fair as a general assessment. MBA Acctg Student 16
Yes It was difficult to understand how the images needed to be explained in a few of MBA Acctg Student 21
the questions, other than that yes.
Yes Questions included covered a variety of topics, assessing my foundational MBA Acctg Student 6
knowledge of accounting.
Yes Remembering account principles off the top of my head were difficult because I MBA Acctg Student 10
was not an accounting major. Since I was unable to refer to notes or other
sources, it took me longer to respond to questions. However, this assessment
was a refresher of what I need to learn and apply during and after my MBA
program.
Yes Seemed like basic questions but not surprised I didn't remember all the concepts MBA Acctg Student 28
from my core semester class. It's part of a curriculum that makes me learn the
very basics and I've retained some but not enough probably. I think one more
deeper level class would have made sense for my total curriculum.
Yes Some easy, some need to think MBA Acctg Student 51
47
MBA Accounting participant comments (cont’d)
Yes Tested balance sheet and basic journal entries MBA Acctg Student 44
Yes They were all concepts that had been covered and that should be understood if MBA Acctg Student 13
you've taken an introductory accounting class
Yes This tested me on various aspects of accounting. MBA Acctg Student 47
Yes up to the par MBA Acctg Student 5
Yes Very basic knowledge in accounting MBA Acctg Student 22
Yes Very focused on t-acounnts, which was less a focus of our class, but still useful MBA Acctg Student 24
Yes well, not every actually uses the T method, or we never learned it in class, but I MBA Acctg Student 31
think the organization and journal entry questions were fair
Yes Yes, because all of this is content that I have covered in my intro accounting MBA Acctg Student 18
course (but have not remembered everything!).
No I didn't study MBA Acctg Student 39
No I studied accounting with the case method, learning to make tough judgments MBA Acctg Student 4
about unique accounting situations that actual companies face. I don't
remember T accounts, but I do remember principles that would be applied in
actual situations.
No It focused a little too much on bookkeeping, not management accounting. I MBA Acctg Student 9
don't remember which side of a "T" account does what! The MBA accounting
knowledge is more about how to record revenues / expenses, how to read
financial statements, assess a company's financial health, etc. It felt like there
were too many questions about LITERALLY recording these in the books.
No Never learned FIFO/LIFO; I remember overall accounting concepts relevant to MBA Acctg Student 25
doing business, but not the formats for journal entries very well anymore
No Some of the questions were poorly worded. It was not clear what I was MBA Acctg Student 1
supposed to answer.
No There's much more to it than remembering easy-to-reference formulas by heart. MBA Acctg Student 2
48
Smartly Accounting participants comments
Yes All questions were very similar to the Smartly modules and seemed to be a fair overall test of Smartly Acctg
knowledge. Student 31
Yes Because honestly, with my education and experience, I didn't know a damned thing about Smartly Acctg
accounting before this course. Student 22
Yes Covered full range of topics Smartly Acctg

Student 14
Yes Everything tested in the quiz was taught in the lecture series. Thank you, this was a very helpful tool Smartly Acctg
to give me a basic understanding of accounting! Student 12
Yes Having no background knowledge, I have applied the learning from the 2.5-3 hours of online lessons Smartly Acctg
only. Hence, the assessment was fair Student 28
Yes I completed the assesment alone and without the use of study materials Smartly Acctg
Student 23
Yes I did not refer to external resources and thought carefully about each question. Smartly Acctg
Student 25
Yes I have not had a thorough accounting lesson in many years, so going through the course and taking Smartly Acctg
this post-exam shows my current understanding of what I retained. Student 2
Yes I learned a lot from the course and applied to my best of ability. Smartly Acctg
Student 29
Yes I recognized all topics I was asked about. Smartly Acctg

Student 9
Yes I think it hit on most of the topics that I had learned throughout the classes, and the classes Smartly Acctg
comprised most of my knowledge. Student 4
Yes I think it tested the concepts well. I don't really get the framing of the LIFO/FIFO question, though. Smartly Acctg
Student 7
Yes It covered most of what the courses covered, and I improved a lot from start to finish. Smartly Acctg
Student 8
Yes It covered the concepts covered in the accounting courses I took on Smartly. Smartly Acctg
Student 18
Yes It focused on the highlights of what we learned in class. Smartly Acctg

Student 16
Yes It refers to some concept of accounting and some examples. Smartly Acctg
Student 33
Yes It was comprehensive and included realistic examples, similar to what one working in accounting Smartly Acctg
would have. Student 32
Yes Most of the questions were straightforward Smartly Acctg

Student 3
Yes Questions were based on topics learnt. Smartly Acctg

Student 30
49
Smartly Accounting participants comments (cont’d)
Yes Relevant and challenging questions were asked to test knowledge gained from the courseware. Smartly Acctg
Student 21
Yes Similar to course questions which is my only basis for assessing accounting knowledge Smartly Acctg
Student 24
Yes Simple and concise Smartly Acctg

Student 1
Yes The classes through Smartly are my only exposure to accounting. I recognize the concepts in the Smartly Acctg
assessment from the blocks on instruction, however I think it is safe to say I didn't master the Student 15
concepts.
Yes The questions asked exactly what we learned from the course. Smartly Acctg
Student 26
Yes The questions covered a wide range of accounting topics. Smartly Acctg
Student 19
Yes The questions were based on the fundamental accounting coursework that I learnt Smartly Acctg
Student 27
Yes The questions were not too tricky and accurately corresponded to the knowledge learned during Smartly Acctg
the lessons. Student 6
Yes The Smartly course was comprehensive and prepared me well Smartly Acctg
Student 34
Yes This was a decent assessment, but my only comment is that I have already seen these questions Smartly Acctg
either in the Smartly material or in the pre-assessment material. New questions might be a more Student 10
fair assessment, but a challenge would be assessing whether improvement was made.
No I have not had to apply it in a human-to-human interaction. My knowledge is circumstantial and I Smartly Acctg
would not feel confident sharing this knowledge in real life. Student 5
No I only wrote no because I don't like yes or no questions. I think a "rating" question would be Smartly Acctg
better. I thought for the most part it was a fair assessment. However, there's a lot more of Student 11
material that was not covered.
No I think it's pretty specific to the Smartly test environment. I think there are a lot of things that I Smartly Acctg
know through experience that weren't explored or captured here, but some more technical things Student 17
like LIFO vs. FIFO that I was unfamiliar with that are.
No I'm still a bit confused on the concepts. Smartly Acctg

Student 20
No Yes and no. While this assessment addressed the basics, I do not believe it tackled the more Smartly Acctg
challenging lessons learned in Part 2 of the study materials. Student 13
50
MBA Finance participant comments
Yes Basic knowledge MBA Finance

Student 52
Yes Covered decent amount of topics MBA Finance

Student 49
Yes I agree with result. Questions were formulated well, perhaps too easy for a finance major MBA Finance
Student 44
Yes I honestly don't really remember the test MBA Finance

Student 29
Yes It assessed basic building concepts of finance that should be learnt during introductory courses. MBA Finance
Student 23
Yes It captured the essentials any finance professional should know.. Of course, this only scratches the MBA Finance
surface, but you managed to capture the essentials. Student 1
Yes It covered a great deal of information that was actually relevant to what has been learned. deal MBA Finance
Student 14
Yes It covered basic principles. MBA Finance

Student 38
Yes It tested a variety of concepts covered in mba coursework MBA Acctg

Student 6
Yes It was representative of what I learned in my finance courses. MBA Finance

Student 27
Yes Seemed to ask the same types of questions that we were asked to learn during the MBA. However, MBA Finance
sometimes the questions were phrased a little confusingly. Those questions were probably less Student 25
accurate assessments of my knowledge.
Yes Seemed to ask very specific questions targeted at assessing the level the responder understands or MBA Finance
does not understand finance, ranging from easy to hard questions, so yeah i thought it was fair. Student 54
Yes Similar questions demonstrated in course material MBA Finance

Student 30
Yes The assessment evaluated basic financial concepts. MBA Finance

Student 24
Yes The question itself was simple but require certain knowledge about finance. MBA Finance
Student 16
Yes The questions have covered most of the corporate finance MBA Finance
Student 18
Yes The questions made sense to me and I think should be a base leavel MBA Finance
Student 42
Yes The questions spanned a broad range MBA Finance

Student 37
No Finance is more about conceptual knowledge than just numbers.. This assessment concentrated more MBA Finance
on quantitative stuff. Student 35
51

MBA Finance participant comments (cont’d)
No I felt the question's were really basic. If it's supposed to measure basic finance skills, spot on. But MBA Finance
overall for someone in finance it doesn't give proper assessment as it was too basic and easy. Student 7
No it asked theories and definitions MBA Finance

Student 46
No It looked more at specifics instead of general principles MBA Finance

Student 4
No It was too Basic MBA Finance

Student 39
No Question could have been harder MBA Finance

Student 43
No To be honest, I can't remember. I do think that Stanford's finance curriculum is wholly inadequate MBA Finance
and stultifying. Student 50
No Too easy MBA Finance

Student 9
52
Smartly Finance participant comments
Yes All concepts were covered in the lessons Smartly Finance

Student 25
Yes Because I tried as hard I could. But I am really bad at this stuff. I still learned a bit. Smartly Finance
Student 39
Yes Covered the topics, major concepts and methodologies Smartly Finance
Student 34
Yes I performed better than the first test, and Smartly was my only influence on the difference in my Smartly Finance
knowledge. Student 43
Yes I really liked how I was able to get hints, use my keyboard entirely and be exposed to helpful and Smartly Finance
relatable examples. Student 13
Yes I recognized a lot of the terms. The only issue was memorizing the formulas, which I didn't do! Smartly Finance
Student 11
Yes I recognized all of the terms and equations that were presented in this quiz. I didn't necessarily Smartly Finance
study my notes diligently before taking this post-assessment, though; I would have done better Student 6
had I studied as if it were an actual class or exam.
Yes I think giving us the same questions before and after the courses is a useful way to test what we Smartly Finance
learned in the middle Student 22
Yes I think it accurately covered most of the material that was taught through Smartly! Smartly Finance
Student 19
Yes I was a blank slate for finance knowledge. So anything I have or have not learned came directly Smartly Finance
from this platform! Student 10
Yes I was pleasantly surprised that the answers I had zero clue about in the pre-assessment, I now Smartly Finance
understood. Student 36
Yes it adequately tested all the topics and formulae I just learned through Smartly Smartly Finance
Student 15
Yes It asked questions directly related to the Smartly material. Smartly Finance
Student 1
Yes It reviewed the material exactly as presented. Smartly Finance

Student 4
Yes It tested the formulas I learned. I do think that you should test what the concepts actually mean. Smartly Finance
Student 3
Yes It was definitely fair. I think that the Smartly courses did not ask for enough answers to questions Smartly Finance
without their "trick"support... ie : green /yellow font left room for guessing... Student 7
Yes I've gone through the courses in a short time and completed the test with no help from outside Smartly Finance
sources. Student 20
53
Smartly Finance participant comments (cont’d)
Yes Questions covered the material in the lessons Smartly Finance

Student 21
Yes Questions were based on topics covered in Smartly courses. Course was knowledgeable and Smartly Finance
enjoyable. Student 16
Yes Similar question format to the lessons. Smartly Finance

Student 23
Yes Since I didn't specifically prepare for it, it reflects accurately the extent to which I would be able Smartly Finance
to rely on this training in the future. Student 14
Yes The assessment allowed me to show which equations whose underlying relationships I Smartly Finance
understood. I probably got the answers wrong for questions where I didn't fully understand the Student 8
concept taught through Smartly.
Yes The assessment covers what I learn from the courses. Smartly Finance
Student 38
Yes The assessment questions sum up the Smartly courses Smartly Finance
Student 37
Yes The content was generally the same. Smartly Finance

Student 2
Yes The courses were good but at times they went quickly Smartly Finance
Student 40
Yes The material was covered in the lessons. Smartly Finance

Student 17
Yes The questions are based on the online courses I learned on Smartly. Smartly Finance
Student 41
Yes The questions pertained to the material learned through the online modules. These questions Smartly Finance
were not too difficult, but also did not give away the answers. Student 31
Yes The questions were similar to the ones in the training Smartly Finance
Student 29
Yes This covered parts from each of the different chapters/courses. Smartly Finance
Student 35
Yes went over what the courses it Smartly Finance

Student 5
Yes Yes because all topics questioned were covered. I think I got a few of the problems wrong Smartly Finance
where I had to apply real numbers to formulas I had previously learned as I am still not 100% Student 12
familiar with each formula.
Yes Yes, it was less difficult than I thought it would be Smartly Finance
Student 28
54
Smartly Finance participant comments (cont’d)
Yes yes. I learned a lot Smartly Finance

Student 9
Yes Review of lessons Smartly Finance

Student 42
No Because the questions seems very simple and similar to Smartly content. It felt like the course Smartly Finance
taught to the test. Student 18
No I was able to figure some out from the answers, but honestly would not be able to think of it Smartly Finance
and everything a formula means without the answer. Student 26
No I worked in investment banking for almost two years Smartly Finance

Student 33
No More calculation examples, as we had trough classes could have been included here, like "real Smartly Finance
case" examples. Student 30
No Simply taking a course online without any extra study time does not accurately reflect my Smartly Finance
ability to take in and learn information. The questions in the final assessment were very Student 24
detailed. It is hard to remember these details from just taking an online course once.
No Some of the assesment questions were purely commonsense. In Part 3, I would have loved to Smartly Finance
solve one of the cases again which would have trained me further on practical applications. Student 27
Also, Part 3 was exactly the same as Part 1 which means I already knew the common sense
answers to many problems.
No There should be case based questions Smartly Finance

Student 32
55

Smartly Report - Final 08-10-2016

Uploaded by

Copyright:

Available Formats

Smartly Report - Final 08-10-2016

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Smartly Report - Final 08-10-2016

Uploaded by

Copyright:

Available Formats

Foundational

MBA accounting and finance

Assessment Design Study Design and Data Analysis

Tables and Figures

Improvement (pre-test to post-test): Smartly participants

Accounting 34 57% 86% 29 points

Finance 43 49% 82% 33 points

Post-test performance: Smartly versus MBA

Mean Post- Number of Mean Post-test Number of

Accounting 86% 34 75% 50

Finance 82% 43 81% 54

Time Spent in Smartly Courses

Participant evaluation of Smartly custom assessment

MBA (Accounting) “It asked the fundamentals that I learned in Accounting I”

Recruitment and eligibility

Evaluation and pilot testing

Final study sample

MBA Smartly MBA Smartly

Number of Number of Number of Number of

Women 39% 20 44% 15 43% 23 57% 24

Men 44% 31 53% 18 57% 31 43% 18

MBA Smartly MBA Smartly

Number of Number of Number of Number of

18-24 4% 2 44% 15 4% 2 37% 16

25-34 92% 47 53% 18 93% 50 58% 24

Smartly pre-course performance vs. MBA performance

Table 3. Comparison of Smartly pre-test results with MBA group

Number of Mean Score Lowest

Smartly pre-test 34 57% 54% 27% 80%

MBA post-test 51 75% 76% 41% 93%

t-statistic for test of

Smartly pre-test 43 49.2% 53% 12% 82%

MBA post-test 54 82.4% 8% 29% 100%

t-statistic for test of

Course effectiveness: Pre-test to post-test Improvement for Smartly participants

Including participants completing both the pre- and post-tests

t-statistic for test of

Table 5. Explaining score improvement

Smartly Post-test performance vs. MBA performance

t-statistic (with p-value in

Accounting 86% 34 75% 51 3.95 3.97

Finance 82% 43 81% 54 0.44 0.46

Feedback from participants

Smartly participant satisfaction (Net Promoter Score)

Limitations of the study

Possible future studies

Comparisons to other online, self-study platforms

SMARTLY MBA SMARTLY MBA

SMARTLY MBA SMARTLY MBA

Smartly Accounting group results (N = 34)

Mean Median Mode Minimum Maximum

Pre-test scores 57% 54% 54% 27% 80%

Post-test scores 86% 89% 98% 49% 100%

% Change 59% 46% 38% -9% 191%

Smartly Accounting individual participant results (N = 34)

Smartly Acctg Student 1 34% 61% 79% 27 135 Yes

Smartly Acctg Student 2 61% 88% 44% 27 173 Yes

Smartly Acctg Student 3 78% 98% 25% 20 126 No

Smartly Acctg Student 4 46% 90% 95% 44 99 No