I've been skeptical of claims of so-called brain-based learning for years. Someday perhaps we'll be able to link analysis of brain function to behavior and learning, but we're not there yet.
Here is a brilliant YouTube video by Daniel Willingham, professor at the University of Virginia, that does a great job explaining the problems with brain-based learning. His audience seems to be schools and universities, but his arguments hold for work-learning as well.
Smile sheets (the feedback forms we give learners after learning events) are an almost inevitable practice for training programs throughout the workplace learning industry. Residing at Donald Kirkpatrick's 1st level—the Reaction level—smile sheets offer some benefits and some difficulties.
On the plus side, smile sheets (a) show the learners that we respect their thoughts and concerns, (b) provide us with a customer satisfaction ratings (with the learners as customers), (c) hint at potential bright spots and trouble spots, and (d) enable us to make changes to improve later programs.
On the minus side, smile sheets (a) do not seem to correlate with learning or behavior (see meta-analysis by Alliger, Tannenbaum, Bennett, Traver, and Shotland, 1997, that showed very weak correlations), (b) are often biased by being provided to learners in the learning context immediately after learning, and (c) are often analyzed in a manner that over-values the data as more meaningful than it really is.
Based on these benefits and difficulties, I recently developed a new smile sheet (one that shows its teeth, so to speak. SMILE) for my workshop Measuring and Creating Learning Transfer.
It has several advantages over traditional smile sheets:
1. Instead of asking learners to respond globally (which they are not very good at), it asks learners to respond to specific learning points covered in the learning intervention. This not only enables the learners to better calibrate their responses, it also gives the learners a spaced repetition (improving later memory retrieval on key learning points).
2. The new smile sheet enables me to capture data about the value of the individual key concepts so that changes can be made in future learning interventions.
3. The smile sheet has only a few overall ratings (when lots of separate ratings are used in traditional smile sheets, most of the time we don't even analyze or use the data that is collected). There is space for comments on specifics, which obviates the need for specific ratings, and really gets better data as well. The average value is highlighted, which helps the learners compare the current learning intervention to previous learning interventions they have experienced. (You should be able to click on the image to see a bigger version).
4. The smile sheet asks two critical questions related to how likely the information learned will be utilized on the job and how likely the information will be shared with others. In some sense, this is where the rubber hits the road because it asks whether the training is likely to have an impact where it was intended to have an impact.
5. The smile sheet shows some personal touches that encourage the learners that the learning facilitator (trainer, professor, etc., or me in this case) will take the information seriously.
6. Finally, the smile sheet is just a starting point for getting feedback from learners. They are also sent a follow-up survey 2 weeks later, asking them to respond to a few short questions. Here are a few of those questions. Again, you might need to click the image to see a bigger version.
The learners get the following question only if they answer a previous question suggesting that they had not yet shared what they learned with others.
Why I Like My New Smile Sheet
I'm not going to pretend that I've created the perfect assessment for my one-day workshop. As I've said many times before, I don't believe in "perfect assessments." There are simply too many tradeoffs between precision and workability. Also, my new smile sheet and the follow-up survey are really only an improvement on the traditional smile sheet. So much more can be done, as I will detail below.
I like my new evaluation sheet and follow-up survey because they give me actionable information.
I also like the evaluation practice because it supports learning and performance.
Limitations
Learning Measurement is a Critical Leverage Point
Learning Measurement provides us with a critical leverage point in the work that we do. If we don't do good measurement, we're not getting good feedback. If we don't get good feedback, we're not able to improve what we're doing.
My workshop smile sheet and follow-up survey attempt to balance workability and information-gathering. If you find value in this approach, great, and feel free to use the links below to download my smile sheet so you can use it as a template for your evaluation needs. If you have suggestions for improvement, send me an email or leave a comment.
Download the Smile Sheet
References
Alliger, G. M., Tannenbaum, S. I., Bennett, W. Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341-358.
This Friday online, I'm leading a discussion on measurement best-practices during my Brown Bag Learning event. It's a 30-minute review of the a new job aid I've created to help you do better measurement. As usual, it's research-based!!
Click to get the job aid.
Click to register (or learn more) about the webinosh.
I've developed three separate several job aids in regards to learning measurement over the last few years, and I decided recently that 1 was enough, so I've integrated the wisdom from all three into one job aid, which I now make available to you for free.
This job aid has several advantages:
Click here to download the job aid now.
In a provocative article today in the New York Times (June 6, 2008), Paul Krugman makes the case that all media (books, music, articles, software, etc.) providers/creators will be forced to lower prices significantly or give away their products for free. The new business model will involve selling ancillary services or products.
This will not only produce a profound shift in how the world works, but it will affect the learning-and-performance industry as well. For example, no longer will I be able to earn practically nothing for producing some of the best research-to-practice stuff on the planet. Instead, I'll be able to earn nothing at all for the research reports I produce. No worries, I'm all right with that. I've got my consulting and speaking to support me.
But really, all you training providers ought to peer deeply into the future. How will this affect you?
Here's some wild ideas:
Okay, what do you think?
As for my research/consulting practice, don't worry, I'm already there. I basically make 80% of my money doing consulting, speaking and keynotes, workshops, learning audits, and providing instructional-design help. My research is done for love, because it's the right thing to do, and because it makes me—in my not-so-humble opinion, extraordinarily good at providing consulting, workshops, etc.
This is the first draft of a chapter from my forthcoming book (don't ask when, it's a labor of love), modified somewhat to avoid references to other parts of the book (to make it whole here). Your comments, criticisms, and good ideas will be gratefully explored.
It's helpful to have an overall understanding of what we're trying to do in the learning-and-performance profession. I offer the following model as a way to frame our discussion, as a way to provide a deep conceptual map of our world—at least the world to which we should be aspiring.
I've been in the learning-and-performance field for almost a quarter century. I've been an instructional designer, a trainer, a university adjunct, a simulation architect, a project manager, a project manager, a business leader, a researcher, and a consultant. And yet, even recently, I have found myself wanting to build a better model of what we do. In the book, I'm going to offer several models that provide a good starting place for deep understanding. The first of these models I call, The Learning Landscape. I've been gradually building this model for years, and I recently added some additional complexity that completes the picture of what we do—or what we should aim to do.
Of course, all models are simplifications in the interest of understanding and usability. Early versions of The Learning Landscape have resonated with my clients, and I think this latest version provides additional value.
I'm going to unveil the model a piece at a time, adding complexity as this blog post progresses.
Look at the bottom of the following diagram. You'll notice three labels there. The learning landscape I'm describing is one in which we build a learning intervention to help our learners perform in their future performance situations (for example on the job) in an attempt to create certain beneficial learning outcomes. So for example, if we build a course to teach creative-thinking skills, we do it to help learners be more creative in their jobs and produce more innovations for their organizations.
Look at the diagram below. It shows how a learning intervention creates performance that lead to results. In the learning intervention (box A), the learners learn—they build an understanding of the learning content. Later, in the performance situation (box C), the learner retrieves from memory the information that they learned. They also apply what they learned (box E). This successful retrieval and application enable the learner to get from the learning what they hoped to get (box F) and the organization gets the learning results it wanted to get (box G) from the investment it made.
The diagram above shows the minimum requirements for a successful learning intervention. The learners have to learn (box A), retrieve (box C), and apply (box E) what they've learned in order to create beneficial learning outcomes.
It would be pollyannaish for us to believe that this process always works as diagrammed above. When our learners fail to learn (box A), the whole process breaks down. You can't retrieve what you never learned and you can't apply what you can't retrieve. Even when our learners fully learn a topic, at a later time they may fail to retrieve what they learned (box C). People forget information. They may forget something permanently or they may suffer temporary or contextually-induced forgetting. Learners can also learn and retrieve successfully, but not apply what they've learned (box E). There are many reasons that learners fail to apply what they are able to retrieve. The learning intervention might not have been sufficiently motivating to prompt the learners to apply what they learned. The incentives in the performance situation may discourage application. The learners may not have time, resources, or other competencies that enable them to be successful.
The diagram above is very helpful in understanding how our learning interventions create learning results. It highlights the obvious importance of the quality of the learning intervention itself. It also highlights the criticality of retrieval. Later we will explore in-depth how we can build learning interventions to specifically support retrieval. Finally, the diagram above highlights the importance of the performance context in supporting learners in applying what they learned. Later we will talk about how we can gain more influence in the performance situation. We'll also discuss how to enroll learners' managers to improve the likelihood of successful application.
The above diagram is missing a few key elements. While it nicely highlights the importance of creating retrieval, it doesn't account for other working-memory triggers. In the diagram below, I've added another box (box D "Learner Responds") to represent working-memory processing not directly related to what was previously learned in our original learning intervention.
While working-memory "retrieval" and "responding" are overlapping and often interdependent processes, I wanted to distinguish between the retrieval of information from the learning intervention and responding generated by other means, for example job aids, performance support, management prompting, and other guidance mechanisms.
While often box C and box D work in concert (as when a job aid provides guidance and also supports retrieval of what was previously learned), we need to remember that we can get our learners thinking productive thoughts without necessarily relying on course-like learning interventions.
To reiterate, there are two ways to trigger working-memory processing related to our learning efforts. First, retrieval cues can trigger our learners to remember what they've learned in the original learning intervention. Second, other triggers (i.e., performance-support tools) can also stimulate working-memory responding. As you probably know, sometimes it is more effective to rely on retrieval, while other times it is more effective to rely on performance support interventions. Sometimes it is better to utilize both in concert.
We also should recognize that learners can learn in their performance situations. The full model below adds this performance-situation learning, what some people call informal learning.
Performance-situation learning (box B) belongs in the model because it is a powerful force in real-world performance. Our learners do a large part of their learning on the job. Whether they receive formal training or not, learners learn through their experiences at work, at play, just living their lives.
Performance-situation learning (box B) can support the learning-intervention learning (box A) by reinforcing what was originally learned, taking the learning deeper, determining real-world contingencies, and creating fluency in retrieval, among other things. One thing we need to realize as learning professionals is that we don't necessarily have to get learners up to speed completely in our formal learning interventions—in fact it is difficult to do so. Instead of trying to cram every bit of information into our training programs, we would be far better off to design our programs with an eye to what can be learned on the job and a plan for how to support that later on-the-job learning.
From the opposite direction, learning-intervention learning (box A) can be designed specifically to help learners learn in their performance situations. We'll talk more about this later, but briefly, we can help our learners learn by helping them notice cues they might not readily notice, by providing them with relevant mental models of how the world works, and by influencing the performance-situation itself—for example by providing reminding mechanism and getting learners' managers involved. To highlight this point again, box A learning can influence and support box B learning and box B learning can reinforce and extend box A learning.
All models and all metaphors have limitations, even if they are brilliantly clear in simplifying complex realities into workable conceptual maps (think e=mc2). I've been working on this learning landscape model for years and though it seems complete and potent to me now, I imagine that in the years to come I and others will find chinks in its armor or improvements that can be made.
I admit the possibility of the model's limitation partly because it is true (that it is likely to have limitations) and partly to model useful thought processes in learning design. Too often, our instructional-design programs have taught models as gospel, unfortunately creating instructional designers who are only able to follow recipes—and who particularly (a) are unable to be creative when unusual situations confront them, (b) are unable to create new models of increasing or context-specific usefulness, and (c) are unable/unwilling to listen and learn from the wisdom of other people and other disciplines.
The learning landscape model is intended to make the following points:
I've decided to offer free webinars from time to time on very short subjects. I did my first one last week, and it went pretty well, except for a bit of operator error on my part.
In these 30-minute sessions, I'm going to focus on some of the most important issues in learning design.
Each session includes:
Upcoming Online Sessions:
(Click below to sign up)
Friday June 6th
Feedback on Correct Answers
Friday June 20th
Minimizing Forgetting
Previous Online Sessions:
Friday May 30, 2008
The Learning Landscape
