Meanwhile, On Our Other Flank...

Lots of online commentary and reposting of OpEd ("The Trouble with Online Education") by M Edmundson of UVA in last week's New York Times.  For an interesting response see N Jurgenson on the Cyborgology blog.

Personally, I’m pretty sure that we (non-elite liberal arts colleges) ARE sunk. Not so much BY online education as by our response to it. As far as I can tell, almost all the reaction from liberal arts quarters is just that: reaction.  And mostly reactionary (that’s how I’d characterize Edmundson’s piece, even if I agree with him that a great face-to-face class can be a great thing – all one can say is “of course it is”).  Our usual approach, trying to preach the demons away, does not have a good track record.

But the big mistake made by little institutions that DO take a look is to think that the way to go is to “get into” offering online courses.  Terrible idea, in part due to the simple problem of scale.  I pray that we at my own institution don’t even waste time on it. If we venture down the fool’s path of looking for (yet) another cash cow, we’ll be a footnote in the history books in no time at all.

Which is not to say online instruction and instructional resources are a bad thing.  Au contraire!  What we should be thinking about is how we can use these tools to increase productivity and effectiveness and repertoire (and thereby lower the cost of we do well so that we can attract more enrollment).  Coursera and EdX (and P2PU et al.) are amazing ventures generating all manner of raw material for the bricolage of creative minds; liberal arts colleges should be thinking like innovators – learning all about them with an eye to how to use them to our benefit – not like a ostriches, fundamentalists, Alamoistas or Masadaists.

Alas, every indication I’ve seen over the last few years in our approaches to budgets, revenue, innovation, technology, and enrollment is very discouraging in this regard.  Circumstances force something like 90% of our attention to be on today and we tend to squander the last 10%  on yesterday; tomorrow is our neglected child.

The Path from "Running a University Like a Business" Leads Where?

A blog post with some insightful comments about UVA situation (http://bit.ly/PD6WNv) notes that the folks who pressured the president to resign favor something they call "strategic dynamism."  The blogger notes that it's pretty much all about dynamism and not much about strategy.  That is, it's about action, not planning.

This had me thinking about Karl Mannheim's ideal types of conservative thinking (ranging from bureaucratic conservatism to fascism).  Mannheim characterizes the latter as “active and irrational,” noting that at “the very heart of its theory and its practice lies the apotheosis of direct action, the belief in the decisive deed, in the significance attributed to the initiative of a leading elite.  The essence of politics is to recognize and grapple with the demands of the hour” (Ideology and Utopia 1936 : 134 http://bit.ly/PD6NJT). 

Indeed, it's a real danger in higher education when administrators and government boards think that "running a college like a business" means adopting the deference to swashbuckling command and control oriented decisive simplifiers.  The very fact that they cannot see how many different ways there are to be "like a business" is what makes them most dangerous.

Know (and be smarter than) Your Enemy

This post is not specifically about assessment, but it relates to the larger conversation of which assessment is but one component : the future of American higher education.  Thanks to tweet from Cedar Reiner for turning me onto it.

You've possibly already seen this D Levy opinion piece in the Washington Post from March (or certainly other examples of the genre) example of what “they” are saying and reading (spoiler: it's the standard “we pay them 100 grand and they only work 15 hours a week” tirade): " Do college professors work hard enough?

It's a tired bit of rhetoric, to be sure, but sung over and over like church hymns, it comes to define reality for a certain set.  That needs to be countered by smart talk widely repeated; smirking won't do.  Here’s one reasoned rebuttal by Swarthmore's Tim Burke that casts the problem in terms of larger arc of private capture of value through de-professionalization: "The Last Enclosures."

The real challenge here is that most representatives of “the other side” (e.g., administrators, trustees, legislators) have not actually thought things through carefully but have bought into a well-crafted rhetoric and catchy simplifications, while “our side” takes a fundamentally conservative approach (same as it ever was) and puts its finger in its ears and goes “la la la la I cannot hear you….”  Higher education has a broken economic model, but too many of us are content to just demonize those with really bad ideas about how to fix it.  I agree with most of Burke's critique, but I think we need to move beyond critique.  There is a romantic valor in identifying the corruption in the current wave of education reform, but it won't be stopped by mere resistance.  Bad new ideas need to be defeated by good new ideas (as can be found in some of Burke's other posts).

What If Administrator Pay Were Tied to Student Learning Outcomes

The recent negotiation in Chicago ("Performance Pay for College Faculty") of a tie between student performance and college instructor pay brought this accolade from an administrator:  it gets faculty "to take a financial stake in student success."

It got me wondering why we don't hear more about directly tying administrator pay to student success.  If we did, I'll bet the students would have a lot more success.  At least, that's what the data released to the public (and Board of Trustees) would show.  There'd be far less of a crisis in higher education.

Thought experiment. What would  happen if we were to tie administrator pay to student success -- much the way corporate CEOs have their pay packages designed -- especially administrators of large multi-campus systems.

Prediction 1.  The immediate response to the very proposal would be "oh, no, you can't do that because we do not have the same kind of authority to hire and fire and reward and punish that a corporate CEO has."  But think about this...

1. Private sector management has a lot less flexibility than those looking in from the outside think.  Almost all of the organizational impediments to simple, rational management are endemic to all organizations.
2. Leadership is not primarily about picking the members of your team. It's about what you manage to get the team you have to accomplish.
3. Educational administrators do not start the job ignorant of how these educational institutions work. It is tremendously disingenuous to say "if only I had a different set of tools."  People who do not think they can manage with the tools available and within the culture as it exists should not take these jobs in the first place.
4. This, it turns out, is what some people mean when they say that schools should be run like a business. The first impulse of unsuccessful leaders is to blame the led. The second one is to engage in organizational sector envy: "if I had the tools they have over in X industry...."  What this ignores is the obvious evidence that others DO succeed in your industry with your tools.  And plenty of leaders "over there" fail too.  It is not the tools' fault.

Prediction 2.  Learning would be redefined in terms of things produced by inputs administrators had more control over.  And resources would flow in that direction too.

Prediction 3. Administrators would get panicky when they looked at the rubrics in the assessment plans they exhort faculty to participate in and that are included in reports they have signed off on for accreditation agencies.  They'd suddenly start hearing the critics who raise questions about methodologies.  They would start to demand that smart ideas should drive the process and that computer systems should accommodate good ideas rather than being a reason for implementing bad ones.

Prediction 4. In some cases it would motivate individuals to start really thinking "will this promote real learning for students" each time they make a decision.  And they'll look carefully at all that assessment data they've had the faculty produce and mutter, "damned if I know."

Prediction 5. Someone will argue that the question is moot because administrators are already held responsible for institutional learning outcomes.   Someone else will say "Plus ça change, plus c'est la même chose."

Better Teaching Through a Financial Stake in the Outcome

In an Inside Higher Ed article this week ("Performance Pay for College Faculty") K Basu and P Fain describe how the new contract signed between City Colleges of Chicago and a union representing 459 adult education instructors links pay raises to student outcomes.

Administrators lauded the move in part because it gets faculty "to take a financial stake in student success." The details of the plan are not clear from the article, but the basic framework is to use student testing to determine annual bonus pay for groups of instructors working in various areas. That is, in this particular plan it does not sound like the incentive pay is at the level of individual instructors.

Still, should the rest of higher education be paying attention? Adult education at CCC is, after all, a markedly different beast than full time liberal arts institutions or 4 year state schools or research universities. One reason we should because it's precisely the tendency to elide institutional differences that is one of the hallmarks of the style of thought endemic among some higher education "reformers." Those who think it's a good idea for adult education institutions are likely to champion it elsewhere.

But most germane for the subject of this blog is the question of what data would inform such pay for performance decisions when they are proposed for other parts of American higher education. Likely it will be something that grows out of what we now know as learning assessment. I ask the reader: given what you have seen of assessment of learning outcomes in your college, how do you feel about having decisions about your pay check based upon it?

But, your opinion aside, there are several fundamental questions here. One is whether you become a more effective teacher by having a financial stake in the outcome. The industry where this incentive logic has been most extensively deployed is probably the financial services industry, especially investment banking.  How has that worked for society?  It would be easy to cook up scary stories of how this could distort the education process, but that's not even necessary to debunk the idea.  The amounts at play in the teacher pay realm are so small that one can barely imagine even a nudge effect on how people approach their work.

But what about the data?  Consider the prospect of assessment as we know it as input to ANY decision process, let alone personnel decisions.  Anyone who has spent any time at all looking at how assessment is implemented knows that the error bars on any datum emerging from it dwarf the underlying measurement. The conceptual framework is thrown together on the basis of dubious theoretical model of teaching and learning and forced collaboration between instructors and assessment professionals.  The process sacrifices methodological rigor in the name of pragmatism, a culture of presentation (vis a vis accreditation agencies), and the tail of design limitations of software systems that wags the dog of pedagogy and common sense.  At every step of the process information is lost and distorted. But it seems that the more Byzantine that process is, the more its champions think they have scientific fact as product.

It could well be that the arrangement agreed to in Chicago will lead to instructors talking to one another about teaching, coordinating their classroom practices, and all sorts of other things that might improve the achievements of their students.  But it will likely be a rather indirect effect via the social organization of teachers (if I understood the article, the good thing about the Chicago plan is that it rewards entire categories of instructors for the aggregate improvement).  To sell it at the level of individual incentive is silly and misleading.  And, if we think more broadly about higher education, the notion that you can take the kinds of discrimination you get from extremely fuzzy data and multiply it by tiny amounts of money to produce positive change at the level of the individual instructor is probably best called bad management 101.

College Presidents and Bonuses

A never posted draft dusted off and posted now.

While doing some research on a completely unrelated topics I came across a few articles on the question of whether college presidents should get performance bonuses.  One is from Insider Higher Ed and asks whether there should be bonuses for improved US News ratings (basically no, but with some dissenting voices) and another in Trusteeship, published by the Association for Governing Boards (also, mostly no). 

This is actually relevant to assessment since in the most general terms, assessment is about institutional accomplishment of stated mission goals.

So, let's look at how obviously corrupt it is to give college presidents performance bonuses.

Why does one give performance bonuses?  To motivate behavior that rewards the institution.  But that is just the president's job.  To counteract self-interest?  The most basic values governing a position like college president already rule this out.  Anyone who needs their self-interest balanced is corrupt to start with (in fact, the opposite is the problem for governing boards: they must be vigilant in ensuring that administrators do not use their position to feather their own career nest at the expense of furthering the institution's mission).

Or maybe a governing board would want to show its appreciation for a job well done.  But who is actually doing the job?  Any president worth her/his stuff, knows the success of an institution of higher education depends on hundreds of individuals toiling away in vocational dedication to a task.  A president who accepted a financial bonus based on increased enrollments, higher selectivity, student learning outcomes or student success would be cynical in the extreme.

This model, ported uncritically and uncreatively from the private sector, assumes far less ambiguous goals and measurements of outcomes than exist in higher education. And it assume far more command, control, and executive power than exists in the private sector. And most of all, it probably inflates the influence of presidents on important outcomes.

Supporters likely only want to go half-way. It is unlikely that they would be willing to propose cuts when performance lags or when bad decisions reduce the performance of subordinates.

There could be a satisfying, if tragic, irony when such bonuses are offered: if it becomes widely known, the resulting demoralization of faculty and staff might undo the results that won the bonus.  But alas, the lag time in such things (and the fact that boards that give performance bonuses rarely administer failure penalties) would probably mean the chief executive could take the money and run (which would, perhaps, fulfill the goal of making higher education more like business).

The only ethical and rational thing for a board motivated to give performance bonuses to actually do is to reward the entire institution (and probably in a creatively progressive manner, not with an across the board percentage of salary).

See also

• Dane, Roger and Allen E. Koenig. 1981. "A Good President Rates a Cash Bonus." AGB Reports, v23 n5 p30-32 Sep-Oct 1981
• Hall, Holly. 2010. College President Receives Controversial Fund-Raising Bonus." Chronicle of Philanthropy, January 21, 2010.
• LÓPEZ-RIVERA, Marisa. 2007. "Presidential Bonuses, Often Secret, Are Wide Open at Some Public Universities," Chronicle of Higher Education, November 16, 2007
• NACUA/ACE/NACUBO. 2009. "A Federal Income Tax Guide for College & University Presidents"
• Stripling, Jack and Andrea Fuller. 2011. "Presidents Defend Their Pay as Public Colleges Slash Budgets." Chronicle of Higher Education, April 3, 2011.

White House College Scorecard Suggestions

The White House asked for some feedback on their proposed "scorecard" for higher education cost and value which is intended "to make it easier for students and their families to identify and choose high-quality, affordable colleges that provide good value." Below are their questions and my (quick, off the top of my head, answering-an-online-survey level of analysis) responses.


What information is absolutely critical in helping students and their families choose a college:

You shouldn't be asking this question here. It's a researchable, empirical question. First, on what basis DO people decide? Then, to what degree do they have the appropriate information to do so?

As someone who studies things like this, I don't think the info presented here as it is here presented will provide much added value or better decisions. In terms of presenting information, probably better to summarize in simpler terms: "On metric one, college X is above/at/below average for it's sector." But then don't just stop there -- we also need to global comparison because people don't get how the sectors vary.

Note that costs are in fact a distribution and presenting average after grants still leaves family very much in the dark if they've no way to know where they're likely to fall on the distribution.

Graduation rates does not suffer from this problem.

Percent of loan repayment is too crude to be useful. It's useful for a banker who may want to finance loans for a student at a given school, but very unclear how this number helps student/family shopping for a college.

Average loan amount is useful.

As important as earning potential is, it's a really stupid number here. Just do a tiny bit of due diligence and you'll see screamingly wide variations across majors, careers, and even within majors. Lawyers, for example, have a certain average starting salary, to be sure, but really big range of variation. Frankly I think putting a single number or even a distribution of incomes next to the name of a school would be nothing but phony quantification. Either that or have a really big footnote explaining statistical significance of differences in means.

What other information would be helpful:


Rather than average loan amount and discount rate what would be useful would be ratios. Tell me (1) list price cost of attendance is X; (2) distribution of discounts is ... and (3) range of debt at graduation is ...

Interesting that you don't really have any room for general comments on doing this at all. You are going to end up diverting an incredible amount of resources toward a project that will in all likelihood produce at best some only moderately useful numbers with huge error bars on them. You will feed into the illusion that choice produces improvement (can you cite any actual evidence?). And you will do absolutely nothing that actually lowers or controls costs, increases graduation rates or lowers indebtedness. In short, not a drop of innovation here. Lots of window dressing, but very little that deserves the name policy.

I'm left wondering why this administration is so confident that "better than the alternative" will continue to be a reason people like me support you.

Does the scorecard cause you to think about things you might not have otherwise considered when choosing a college:

Not in the slightest. It makes me think that whoever made it up has never actually been through the process. It reads more like it is informed by a need to respond to conservative activists who are trying to make hay about higher education. As an Obama supporter and contributor I have to admit it's really a little bit embarrassing to read this as part of the administration's policy proposal. If you can't do better than this, I wonder how bad it would really be to have a republican in the WH as well as in control of congress.


How should this version be modified for 2-year colleges:

Look, it's pretty obvious that there are two issues with two year colleges: (1) to what degree does it lead to successful and timely completion of a four year degree, OR (2) to what degree does it yield serious, usable job training.

So, a start would be to provide rate of students who seek admission to four year who actually graduate from a four year. But really easy to get garbage data on this if you don't set up the categories and the tracking really smart.

On the job side, again, gonna be really serious data quality problems that will likely as not make the information worthless (mostly because you are going to see massive variation from program to program WITHIN schools). That said, let's start with simple "how many people are working in a full-time non-temporary job in or related to the field of their AA degree within X years?"

How should comparison groups for colleges be made? What are important things to consider in grouping institutions together that serve similar students:

Catch-22 here. You are asking people to choose -- if you separate it out too well, the really important thing gets lost: we want people to better understand what the different "rungs" represent. One of the big crimes in higher education is that crappy institutions with minimal value added get to promise people a college degree. And if you only compare within groups each one gets to, in a sense, set the standards. What you need is a tool that more clearly lets people see the payoff differences between the tiers (to the degree there are some).

A most important thing that you'll probably leave out is the effect of what you bring to college on the college outcomes. Huge naivete in college assessment world that the college output has only to do with what the college did. Gigantic effects of origins still at work in higher education. Just be sure your new tool doesn't simply do more to perpetuate the myth.

What search and comparison features would you like the online tool to have:

Something that shows schools in context and behind that groups in context (where does this school sit within its group and where does its group sit in the larger picture).

What should we call this tool? Would a different name better explain the service being provided:

One name would be "Republican Higher Education Policy as Adopted by Obama Administration."