SACEUR Inspires Me: Wherein I Repeat Myself

I Blame Foust

So I’ve been on vacation for a little while, and the urge to do anything rather than document another lap around the drain that is 2014 was pretty much overpowering. And I do mean anything. You should see my socks. They have never been so organized.

This also means I’ve been avoiding things like news, blogs, etc., and this blog in particular. Mainly because the folks that usually post here make sense, and just reinforce what I’m seeing here. Which, is likely going to drive me to drink. So in all of that I missed this excellent post by Joshua Foust. Why is it excellent? Because of his excellent choice in references. That’s right, I’m that good. And humble. And not one to toot my own horn. Ever.

I’m going to reserve my commentary specifically for the Asia Foundation survey he references, which I referenced in a comment on a blog post, which he then posted in his post. (See “excellent choice in references” above). So there’s a lot referencing and posting going on here. Here’s what I had to say on this post in reference (see? lots of referencing) to the Asia Foundation survey completed in 2010. Well, sorta. The italicized portions are what I said re: the 2010 survey. Below that are the comparisons to the 2011 survey, along with the occasional rabbit trail and pursuit of something shiny.

Caveat: Here’s the link to the 2011 survey. It’s bigger…better…shinier! Seriously, it’s an infographics surge of Santorum proportions this time around. Obviously, someone at USAID coughed up some serious dough for the website.  And here’s the links to all of the Asian Foundation surveys dating back to 2006 via Scribd. Just in case you want some fun of your own. Personally, I’d recommend smashing your own fingers with a hammer first, then if that doesn’t hurt enough, get your survey read on:

For the first time, as part of the survey, we are unveiling an online data mapping tool,Visualizing Afghanistan and a photo slideshow.

Who doesn’t live a good slideshow? I know I can’t say no.

Back to the back and forth. Again, this is me referencing the 2010 survey.

So a survey that interviewed 6,467 people in a country of 28 million, conducted 23% of its surveys in the Central/Kabul region, and 14% of its surveys in Kabul province itself (the next nearest closest percentage was 7% in Herat), should be classified as “one of the broadest public opinion polls in the country?”

Here’s where I thought I’d been talking about the 2011 survey in my original comment, simply because the numbers are so close. As in freakishly close. This time around Asia Foundation conducted 22% of its 6,348  surveys in the Central/Kabul region, and 14% of its surveys in Kabul province itself (the next closest once again being Herat). Again, not exactly “broad.”

The survey reportedly asked 70 questions. In a single survey. The issue with this is that it’s too broad, covers too much ground, and takes too much time to complete per person being surveyed. As a means to collect information, a 70 question survey is a deeply flawed instrument, as it covers too many topics to adequately demonstrate a specific set of data.

Yup, they did this in 2011, too. And I still stand by this statement.

Additional notes on methodology: they indicate they would use a Kish grid in order to determine who would be asked the questions. This works well in an environment where both men and women are allowed to speak equally. This is not the case in many areas in Afghanistan, and conducting a survey using Western methodologies without taking into consideration the realities on the ground would cause the results to be suspect.

The “Kish grid” is one of those red herrings that survey organizations toss out to show that no kidding, they really do know what they’re doing. It’s a word they use in proposals a lot. Especially really thick proposals. I read a proposal that laid out how they were going to use the “Kish grid” in the “qadas of Afghanistan.” Neat trick, since that’s an Iraqi government district structure, but whatever…either way, successfully implementing a Kish grid is a messy proposition in any area that’s fairly village-centric. Like, I dunno, Afghanistan.

Um…Do You Know Where Your Village Is?

One of the things that usually jumps out about the Asia Foundation surveys is how many sample points had to be moved for one reason or the other, mainly security. Once again, they do explain the adjustment of interview points. First, from 2010 (in italics) followed by the 2011 statements (in not-italics):

In 2010, the situation continues to deteriorate. One hundred and thirty eight of the 885 random sampling points were inaccessible to survey researchers due to security problems (16% of all sampling points).

In 2011, the situation has improved somewhat. Ninety-five of the 876 sampling points had to be replaced due to security reasons (11% of all sampling points).

So that’s a plus, yes? Well, except for what they term the Kabul/Central region: in 2010 only 10 points had to be changed due to security. This year? 17 points, a 70% increase. It’s also the most number of sampling points that have ever had to be moved in the Kabul area. As in Kabul, theoretically the most secure area in Afghanistan. The capital. Yeah, that Kabul.

Also what that means is that  the survey, rather than being done in an area that might not be as secure, is focused in an area that’s safer. Therefore more secure. And consequently more likely to be just plain happier with ISAF and GIRoA overall.

For example, in looking at Ghazni province, which promises to be one of the key areas of focus for ISAF in the coming year or so (as it’s been for a couple of minutes already), every single sample point had to be changed due to “the district is controlled by the Taliban.”

Not the town…the entire district. Which was also true in 2010. In 2009, the changes are done in the same province due to what’s termed “security problem” without being specific. The reasons for changes in Ghazni get less specific in 2008.

Sidebar: Point of interest — 18% of the survey point changes were made due to the fact that they couldn’t find the village. One would think they’d go back to the same places to see what kind of progress was being made, but instead of tracking the progress over time in a particular area, they move to another village, like one they can find, and ask the same questions in a different place. They do explain their methodology as being random based on population, but the value of a survey over time is that you keep asking the same questions of the same people to better gauge their responses.  Of course, I’m not a smart researcher who knows how to “Kish” stuff, so I’ll leave that for those with brains larger than my own.

I did find something interesting. Well, interesting to me…keep in mind I sort my socks: in further reference to how the sampling was done:

However, in 2011, in two provinces, Paktika and Zabul, the deteriorated security situation restricted the freedom of movement, making it unsafe to employ female interviewers. This meant that no women were included in the sample in these provinces.

Samangan, Farah, Zabul, Panjshir and Uruzgan were the only provinces where sampling points did not have to be replaced.

Maybe it’s a different Zabul, but in the one paragraph the security situation was bad enough that they couldn’t interview any women, but then later they state that they didn’t need to replace any sampling points. Which means they knew the situation was bad to begin with, and didn’t actually select random sampling points, but rather selected sample points in areas where it was safe to travel (just not for women). I might be reading too much into it, but the contrast in statements is intriguing.

Sidebar: In the 2008 survey, apparently the universe was the focal point of the survey…seriously. In two different places:

The universe is divided into eight geographical regions consisting of 34 provinces.

Due to the local cultural traditions, the universe at the outset was divided into male and female sub-samples.

But I Don’t Want to Talk About the Survey

I figured that would be enough, that I could just copy and paste some additions to my original comment on my own post (narcissistic…is that still a thing?), and be done with it. But today, Admiral James Stavridis, SACEUR posted this courtesy of NTM-A and…we’re off:

The third key is simple: pressuring the insurgents. We’ve made a lot of progress, and it is reflected in the attitudes of the Afghan people. For example, the recent Asia Society annual poll of the people of Afghanistan concluded that over 85% of Afghans approve of their Army and over 75% of their Police forces — strong improvements over the past five years.

Most of my comments are going to focus on the ANP, since, well, the Asia Foundation has been surveying that particular set of questions consistently, and it’s the arm of the ANSF that the average citizen is more likely to see and have any kind of interaction with on a regular basis.

Sure, they approve of the ANP overall, but  according to the survey, 48% of respondents were afraid when encountering ANP officers, which is down… from 52%. On average, since 2007, 49.8% of respondents stated that they had (some/a lot of) fear when encountering the ANP. So they approve, but they’re afraid of them. And as for the “strong improvements” he mentions, well:

It’s not the greatest graphic ever, but…with all due respect to SACEUR, just taking a look at the top line…five years ago 86% of the people surveyed felt that the ANP is honest and fair. And while it’s now better than it was, it’s still not as high as it was before NTM-A took over training all the ANSF forces.

Another point with regard to this confidence in the ANP: in the Central/Kabul area, in 2010 56% of respondents said that they were confident that the government would punish the guilty party if the respondent was the victim of a crime. In 2011, that number is actually down…to 49%.

Unfortunately, SACEUR isn’t alone in trumpeting this survey, as Mr. Foust pointed out. Even a publication with the usual gravitas of The Economist jumps on the bandwagon:

According to the Asia Foundation, satisfaction with the government’s performance is at 73%.

Which really, is incredible. Except that it’s the exact same percentage as it was last year. And it’s actually down 7% from the 2007 survey, when 80% of respondents felt that the central government was doing a “very good or somewhat good job.” Seriously. Here’s the graph:

I apologize for the quality, but um, yeah…it’s like in colors and stuff. And, here’s some more tables.

The one below is from the 2010 survey:

This is from the 2011 survey.

ANP is unprofessional and poorly trained? (20% vs. 20%) Oh…ouch. Unchanged. This is one of the key metrics that LTG Caldwell kept hammering out (see my post referenced above), and was often fond of pronouncements like this one:

“People’s perception of the Afghan forces is two years old,” he said.

After over two years of the best efforts of the NTM-A, whose sole purpose for existence is training the Afghan National Security Forces (ANSF) to include the Afghan National Police (ANP), existence, 20% of respondents still believe that the ANP are unprofessional and poorly trained. In and of itself that seems fairly minor, until one realizes that the largest respondent pool was in the Central/Kabul area, theoretically the area with the greatest police presence, consequently with the greatest level of police exposure, and opinions are still unchanged.

But, Everybody’s Feeling Good. Right? Please Say Yes

I wish that I could write something different. Like Agent Mulder, I want to believe. Desperately. But, thank you, Asia Foundation, for making this part really easy.

Overall, 35% of Afghan citizens in 2011 say the country is moving in the wrong direction – an increase of 8% from 2010. The main reason cited for pessimism is insecurity, reported by 45% of the respondents who say that the country is moving in the wrong direction. This is followed by corruption (16%), bad government (15%), and unemployment (13%)

And it’s the highest this metric has been since 2007 when they started doing this particular survey. In other words, more people than ever before actually feel the entire country is moving in the wrong direction.

More from the survey:

This year, despite economic, political, and security challenges facing Afghanistan, nearly half of the respondents (46%) say that things in the country are moving in the right direction, a slight decline but still higher than previous years.

Insecurity (attacks, violence, and terrorism) is identified as the biggest problem in Afghanistan by over a third of respondents (38%), followed by unemployment (23%), and corruption (21%).

I’m Worth Repeating

And, in conclusion, I say again:

So the survey focused too many questions across an extremely small sample for a national-level survey, focusing that sample even further into the Central/Kabul area of Afghanistan, which is generally more secure than the rest of Afghanistan. With apologies to the Asia Foundation and the really shiny report they put together, the methodology for this survey is so deeply flawed that any conclusions drawn from the survey should be suspect.

The message from ISAF is clear: as sure as the Chicago Cubs never have to worry about playing baseball in October, the great Afghanistan experiment will be drastically altered from its current iteration by 2014. In fact, in the last few days, the security responsibility for Marjah district, Helmand province, was handing over to the Afghans without a single release from ISAF. In case you’re thinking Marjah wasn’t of any real interest to ISAF, I submit this, and this, and finally this review of the HBO documentary for those of us for whom “reading” sounds a lot like “work” and might involve “words.”

So the good word must go forward, even if that means quoting from survey instruments so intellectually dishonest they wouldn’t pass muster in support of a good Kim Jong Il propaganda film. It’s the kind of survey even Assad would look at and say, “Yeah, that’s not really accurate.”

In a war that’s light on metrics and long on message, it’s critical to the political endgame being played here that ISAF appears to be  turning over this country as soon as possible. And doing so successfully. And it’s making Afghans happy. Here’s to you, 2014: when you get here, I hope you’re way better than I’m afraid you’ll be.

This post was written by...

Dan – author of 18 posts on Registan.net.

Having spent time in Iraq as part of Uncle Sam's 9/11 World Tour as an infantry and civil affairs officer, Dan is currently employed in Afghanistan as a civilian. His day job is way cooler than he deserves, and while he wants (desperately) to believe in the ISAF mission, he's pretty certain that all we're doing is lapping the drain. However, there are bright spots here, and he tries to find them when he can.

For information on reproducing this article, see our Terms of Use