Research Associate - AI Evaluations

About Karya:

Why was Karya on the cover of the Time Magazine , highlighted by Satya Nadella , and chosen by Google as its partner for Project Vaani?

In part, because Karya is on a mission to provide AI enabled earning and learning opportunities to economically underserved communities, thereby building a pathway out of poverty for them. Karya achieves this while also delivering high quality, timely, and price competitive data to its clients.

Karya's workers make at least 20 times the Indian minimum wage and through our one-of-a-kind digital work platform, we have delivered over 40 million digital tasks and are poised to positively impact over 100 thousand workers by the end of the year. In the coming years, our goal is to rapidly scale our impact by bringing economic opportunities to millions of underserved users in India.

About The Role

At Karya, we’re building AI evaluation systems that measure and understand whether AI systems actually work for the communities they’re meant to serve. We’re looking for a Research Associate to help design and run community-facing research that shapes how AI systems are evaluated. You’ll contribute to research workstreams end to end: shaping the research design, building toolkits, running fieldwork, and turning what you learn into evaluation frameworks and datasets. You’ll contribute to a range of evaluation projects across the team. These could include building a gender-bias benchmark for AI chatbots in domains like agriculture; conducting grounded research into what people in India actually use AI for, in which languages and in what contexts, to build evaluation categories from real use; or contributing to conceptual and methods work on what “community” and “participatory” should mean in AI evaluation, pressure-tested against fieldwork with Karya’s worker community. 

The role suits someone with roots in social science who has worked where technology, impact and AI meet, and who cares about gender equity and doing research with communities.

Concretely, this means:

• Contribute to research design, including research questions, methods, site selection and sampling, working closely with project leads
• Build and adapt research toolkits: discussion guides, oral consent scripts in local languages, training materials and feedback tools
• Plan and conduct focus group discussions, interviews and pilots with communities, including participant recruitment and field logistics
• Coordinate with field teams during data collection, keeping research consent-first and respectful of participants’ time and context
• Clean, cull and analyse qualitative and quantitative data through a gender and intersectional lens
• Synthesise insights into findings that strengthen evaluation frameworks, datasets and toolkits
• Contribute to desk research, reports and external outputs
• Support project management by tracking timelines, deliverables and decisions across internal teams (Ops, Tech) and external partners

Why This Role

You’ll be embedded with the Evaluations team within Karya’s tech team, where social science and engineering work as one practice. We believe you can’t judge whether an AI system works for people without understanding both the people and the system, and how social and technical structures shape each other. So researchers and engineers on our team work closely together, each bringing what the other can’t. Your research will shape what g