Measurement and evaluation

Questions we have, topics of interest:
What metrics should I being looking at? What indicators should I react to?
Reporting on what users do, it's overwhelming and less meaningful than I'd like, what are others doing for evaluation?
Not just what the users are doing, but what is the company investing in the community? What value are users deriving?
How do value the interactions on the site, how does the community site influence the overall goal of the org?
We have a lot of manual data, need to get away from it. A lot of the good stuff is manual.
Need metrics to prove that community mgrs are doing their jobs
Use a combo of tools--from vendor, Google Analytics, still looking for how to measure engagement. Trying to look at unique posters, unique visitors. Looking over time.
International work--scaling measurements. Member engagement, satisfaction. Are members happy?
Metrics for ad revenue
Hard to gauge how people feel about the product, not mining the data properly. Overly anecdotal community feedback.
Trying to come up with good metrics on how well the community is doing. Something meaningful to the company. Satisfaction surveys of users.
Struggled with having quant data, but it didn't get to helping community managers knowing where the next flame war was going to erupt. Had to do subjective 'friendliness' measure.
Figuring out how to judge online community resources and satisfaction with those vs. other forms of support
Engagement, how it's measured.
Identifying key metrics. So far tracking user traffic, want to take it a step further. What factors determine growth and success of a group?
How do you measure if a group is healthy?
Ways to measure how we can show profit, so it seems like a good way to spend marketing $
How do you define engagement? What metrics?
What are the metrics for success of a community? How is it different from success of a website?
Engagement, health of community
Measuring impact on the organization--did they use the info to change something in their organization?

Defining Engagement
How many posts this month? How many ppl added a pic to their bio? compare month to month
Looking at quality of posts? Not much, it's a lot of work.
Rubric for looking at what the facilitators are doing, the type of conversation they are generating
Page views and contributions. Not measuring quality, just measuring number.
Satisfaction surveys
Defining good and bad participation? Good vs. bad posts? Bad is porn/inappropriate. Good is anything else?
Asking people to rate posts as helpful or not as a way of defining what a good post is.
Good and bad depends on the goal of the community.
Weighting of different kinds of posts.
Defining relevance rather than good or bad.
What about the 'me too' posts? They are building community, but are they quality posts?
Age of community--do posts decline? How many characters per post? How often do posts happen--what's the interval? Linking to time--when the iPhone comes out, there's a lot on the iPhone, but you'd expect it to drop off.
User rankings--great but it's not many people who are doing the ranking. Also some people try to game the ratings system.
Looking at numbers of characters in a post
Defining social engagement--a presentation here last year--web that is generated as people speak to each other
Maybe you should survey the members of the community--is this something you want to continue to be a part of? Ask the users if they think this is a good community experience.
Engagement as the likelihood of doing something again, returning to take that action again. We predict based on current activity the likelihood of people's future activity.
Character length--problem is that if someones comes bad and says thank you--that's great.
I try to include quotes and notes of when people return to a post in my reports.
Others think that's not enough; need more numbers. Huge silent community who may be affected by what we do and we don't know.
Set of users, randomly chosen, whom we are using to get information from
Monthly contests for teachers to participate--if they posted, they got points, etc. Other teachers would come on and see that, would give them a model.
Want qualitative data, not just page views, but need to quantify it--that's where rubrics come in
Interviews of participants to get the perspective of the silent majority
Facilitators working too hard--answering questions too quickly--keeps people from contributing
Surveys--some have had success but with smaller groups. did a survey, 67% said they benefited from the discussions but definitely 67% weren't participating. So you don't necessarily HAVE to get the silent majority to participate; it's not necessarily bad. Making sure your content is lurker-friendly.

Key Takeaways:

1) Surveys, interviews, rubrics to look at quality/depth content
2) Defining engagement
3) User rankings, #'s of characters in posts
4) Making sure your content is lurker-friendly. Lurkers are ok.