What to do when your grant is rejected
The day after she submitted a grant proposal last November, Sarah McNaughton listed all the tactics she could think of to boost her chances of success next time. “I expect to be rejected,” says McNaughton. “It is the exception to get funded, not the rule.” Publishing key papers and forging new collaborations were on her list, as was collecting preliminary data.
McNaughton, a nutrition researcher at Deakin University in Melbourne, Australia, studies dietary patterns to find ways to improve public health. For the next phase of her work, she wants volunteers to use wearable cameras to capture what influences their food choices in real life, so she can determine how those vary depending on a person’s nutrition knowledge and cooking skills.
After McNaughton had sent off her grant application to Australia’s National Health and Medical Research Council (NHMRC), top of her to-do list was launching a pilot study. “If we can show that people will wear the cameras, and they capture the data we need, that would really strengthen the application,” she says.
A good idea is no guarantee of grant success. At the US National Science Foundation (NSF) in 2017 — the most recent year for which data are available — proposals worth a total of almost US$4 billion were rejected simply because they were beyond the organization’s budget, even though reviewers had rated them as very good or excellent. At the US National Institutes of Health, the aggregate success rate for research grants was 20.5% in 2017 (the most recent data available). At the biomedical-research funder Wellcome in London, roughly 50% of applications make it through the preliminary stage. Of those, around 20% were funded in 2017–18. And the NHMRC Investigator Grant category that McNaughton applied for had a success rate of just 7% in the previous round in 2019.
“Given the low success rates of funding around the world, the odds are stacked against you in winning that one proposal,” says Drew Evans, an energy researcher at the University of South Australia in Adelaide, and former deputy chair of the Australian Early- and Mid-Career Researcher Forum. “Work towards a portfolio of activities,” he says. Aiming for different strands of funding to cover various aspects of a researcher’s work is a safer bet than seeking one major grant, he adds.
McNaughton applies the same strategy to any research for which she is seeking funding. “I think about how I can split it up and target it to other organizations,” she says. It’s the first step towards applying to different funders without having to start from scratch each time — and you can work on it while waiting for the outcome of one application. “Rather than writing eight different grants, you are building an area — calling on the same literature and on your same publications,” McNaughton says.
Planning for rejection is a crucial part of the granting process, say those who have been through the wringer (see ‘More on rejection recovery’). The limited pot of research funds worldwide means that competition is fierce. “We receive many more proposals — many more very good proposals — than we can possibly fund,” says Dawn Tilbury, a mechanical engineer at the University of Michigan in Ann Arbor who is head of the NSF Engineering Directorate, which funds basic research in science and engineering.
MORE ON REJECTION RECOVERY
• You’re not alone. Average success rates are around 20% among large funders, so grant rejection is common. “Don’t lose heart,” says Shahid Jameel, chief executive of IndiaAlliance, a biomedical-research funder in New Delhi and Hyderabad. Rejection doesn’t mean that your work is flawed.
• Give yourself time. Allow a week or so to recover, says Candace Hassall, head of researcher affairs at the biomedical funder Wellcome in London. “When people are turned down, they are angry and upset. Let that play out,” she says. Put the application to one side for a few days before you consider your next steps.
• Share your setback. Discussing the grant rejection with colleagues, mentors and others can provide emotional support in the short term, and give you constructive feedback to help you to reapply for the grant when you are ready. “People whose grants have been rejected might not want to tell anybody, but getting advice and input can really help,” says Karen Noble, head of research careers at Cancer Research UK, which funds scientists and health-care professionals working on cancer treatments.
• Look for ways to improve. Tackling the concerns of the reviewers who rejected your grant is important. “But don’t assume that just by addressing the issues outlined, you will necessarily be successful next time,” says Noble. It is unlikely that the same reviewers will see your application again, so look at it holistically and strengthen it for the next round. This might involve incorporating key new data, learning a crucial technique or forming a fresh collaboration.
• Get feedback. Your revision needs review by a broad, diverse group of people, including colleagues, mentors and members of your network. You should also circulate the revision to scientists who don’t specialize in your field.
Rejection can be a bruising experience, say veteran grant-writers, and applicants need to give themselves at least a week to get through the initial pain. “Take a deep breath, close your computer, go home. Talk to your partner, or pet your cat,” says Tilbury. It’s a rollercoaster that Evans has ridden plenty of times. “You go through the various stages of emotions — anger, disappointment, despair, grieving almost,” he says. “Having time to digest, to get upset and angry — you need to go through that process, because you need a clear mind to come back to it constructively.”
But grant-seekers can develop tricks to handle rejection better, says McNaughton. “Part of the reason I make a to-do list is to pull back my expectations,” she says. “Once it might have taken me a week or two to bounce back. Now, it’s 24 hours.”
During the emotional recalibration process, researchers should share the setback with others, including colleagues and other professional contacts, says Evans. “It is your network that is going to give you the support and encouragement to keep going,” he says. Peers and mentors can help to put the rejection into context. They might also know of other funding opportunities that can help to bridge an immediate financial shortfall, or of potential collaborators who might be able to bring a researcher into a larger funding opportunity.
Ask the funder
After working through the emotional component, applicants should next seek feedback from the granting organization. The level of feedback sent out with rejection letters varies drastically, depending on the organization, the scheme applied for and the stage the application reached before rejection.
For smaller funders, feedback might not be provided as a matter of course. “That takes a bit of effort to put together,” says Kristina Elvidge, research manager at the Sanfilippo Children’s Foundation in Australia. The charity, based near Sydney, funds up to Aus$700,000 (around US$472,000) annually on research into treatments for the rare genetic disorder Sanfilippo syndrome, which causes fatal brain damage.
“I always give feedback to rejected applicants if they ask — but they very rarely do,” Elvidge says. For researchers whose work might align closely with the mission of a small foundation, seeking feedback can be the first step in starting a dialogue and building a relationship with a potential long-term funder. Megan Donnell, the foundation’s executive director and founder, says that the organization welcomes such efforts.
Discussing grant rejections with peers can help to put them into context, advises Drew Evans (left), shown talking to early-career researcher Nasim Amiralian.Credit: Drew Evans
For applicants to a larger organization or agency, such as the NSF, a rejection typically comes with some feedback — but that doesn’t mean the researcher can’t seek more, Tilbury says. “The programme director might be able to fill in some of the blanks,” she says. The feedback can contain many comments, criticisms and suggestions, and often the grant reviewers do not agree with each other. The programme director can help the applicant to peel away superficial concerns and make sure that she or he understands the proposal’s underlying weaknesses so as to address them in a potential revision, Tilbury says. “It’s one of the things programme directors enjoy doing — mentoring junior faculty members and trying to help them in their grant writing.”
Some funders will not have the resources to provide feedback. But researchers should not fear tainting their reputation if they ask, says Candace Hassall, head of researcher affairs at Wellcome. “A funding agency won’t think badly of anyone contacting them for advice, even if we can’t give it.”
Get feedback on the feedback
Once a researcher has gathered constructive criticism, he or she should candidly appraise the strengths and weaknesses of their application. It is important to avoid taking feedback personally, says Shahid Jameel, chief executive of IndiaAlliance, a large research funder in New Delhi and Hyderabad. It supports biomedical and health research in India, and is itself funded by Wellcome and the Indian government’s Department of Biotechnology. “You have to get out of this mindset that there is either something wrong with you, or that people are against you,” Jameel says. “Reviewers really want you to do well — that is why they are spending their time reviewing your grant and providing feedback.”
Reviewer feedback often seems less negative over time, McNaughton says. “I often colour code my reviewers’ comments — green for good and red for bad — and then realize that actually, there are a lot of good things in there as well,” she says. “These little things can make the process a bit easier.” And getting reviewer feedback is certainly preferable to not getting any, she adds. For her most recent rejection, she received only numerical scores on various components of her grant. “Then it is very difficult to know how to improve the application,” she says.
Unsuccessful applicants should also seek input from colleagues and others whose opinions they value. “Talk to your peer group and your mentors — they will have been through the process and they can help you interpret the letter,” says Karen Noble, head of research careers at Cancer Research UK in London, which funds work on cancer treatments. Researchers can ask colleagues whether they agree with the feedback, whether they think that the reviewers missed an important point because it was not fully explained in the proposal, or whether they consider the proposal’s argument to be flawed.
Researchers also need to determine whether they should reapply to the same funding scheme or seek alternatives (see ‘Rejection resources’). If an application fell at the first round of screening — in which reviewers assess the overall suitability of an applicant and proposal for that particular scheme — an alternative funder could be a better fit. For example, some government-supported agencies, such as the NSF, give grants for only basic research, whereas others, such as the US Department of Energy, are mission-focused and fund more-applied projects. “It is also important to consider funders that are not in one’s own nation,” says Jameel.
• Seek help from your peers. Blogs run by academic researchers often contain useful career advice and information about the challenges of winning funding. Examples include The Research Whisperer in Australia and US-based blog The Professor Is In. Some are dedicated to research funding in specific regions, such as Research Fundermentals, which covers UK grant news.
• Find another grant scheme. It might be that your chosen funder wasn’t the best fit for your proposal. Searching portals such as www.grants.gov in the United States and the funding-opportunities database SPIN (run by US firm InfoEd Global) could reveal schemes you hadn’t previously encountered.
• Consider different funders. If your application for a government grant was unsuccessful, try obtaining funds from industry. Also look at small foundations — their remits vary widely (see, for example, http://fdnweb.org/eppley) but your work might align perfectly with one foundation’s mission.
• Do some training. Look for short courses aimed at writing grant resubmissions so you can learn the most effective ways to reapply.
• Network. Join a group of early- to mid-career researchers to gain advice and support. This might be a national organization or one at your institution.
Grant-writers should keep industrial funders in mind, Evans says. He notes that applicants might be able to reshape a proposal to show its value to a particular business, adding that scientists who engage with businesses can diversify their grant portfolio and boost the resilience of their research income stream. Exploring potential applications of one’s work to industry could keep a researcher going until the next round of funding agency grants. “Industry partnerships are now one of the hot topics around the water cooler,” he says.
Nailing the details
Rejection also lurks after the preliminary screening stage when a grant application enters peer review. “If there’s a particular approach the reviewers don’t like, sometimes you may just need to explain it better — but sometimes there’s a mismatch,” Tilbury says. She adds that many early-career scientists seek to apply a technique or expertise they honed during a postdoc to a new area of research. If the reviewers weren’t sold on the idea, the grant-writer needs to think carefully about the proposal, Tilbury says. “Are the reviewers right? Am I using the wrong hammer to pound this nail?”
If a grant-seeker is certain that their proposal — and their expertise — do fit the grant scheme, they need to make that clear to reviewers. “A common reason for rejection is that the applicant has made assumptions about what the reviewers know about them,” Hassall says. “If a technique or method is critical to what you are proposing, you have to include it. Make it easy for people to get the information that they need.”
Similarly, if referees rejected a grant because the applicant had no experience in a particular technique, it pays to get it and include that information in the next round. Before reapplying, researchers should seek collaborators who are experts in that area or technique, or spend a week working in the collaborator’s lab to gain experience.
It is the applications that just miss out on funding that can be the hardest to revise, Noble says. “Sometimes there wasn’t anything inherently wrong with somebody’s application. It just didn’t make it to the top of the list. Those can be the harder ones to try to repackage and put in again.”
Yet perseverance is key, says Mariane Krause, a psychologist at the Pontifical Catholic University of Chile, and president of the National Commission for Scientific and Technological Research (CONICYT) in Chile, which funds research in the country. She encourages researchers to refine their applications and continue to apply. “I have many young researchers who get a grant the third time,” she says.
Reapplying to the same organization for funding can work if the funder allows it. “The success rate of reapplications is significantly higher than for first-time applications,” says Alex Martin Hobdey, head of the unit at the European Research Council (ERC) that coordinates project calls and follow-ups. For example, new applicants to ERC grants have a 9–10% success rate. “For people reapplying, the success rate goes up to 14 to 15%. We have people who got their first grant on their seventh application,” he adds (see go.nature.com/2vrfugk).
Some schemes impose a specific hiatus period before accepting applications, or have an annual or biannual application deadline. Others, including Cancer Research UK, don’t impose specific limits. But programme officers recommend resisting the temptation to rush in a revised application as quickly as possible. “Take time — don’t knee-jerk,” Noble says. “Will you really be in a better position to reapply in a month?”
Nature 578, 477-479 (2020)
Originally posted on nature.com on 18th February 2020 - https://www.nature.com/articles/d41586-020-00455-0
- Career Advice
Secrets to writing a winning grant
When Kylie Ball begins a grant-writing workshop, she often alludes to the funding successes and failures that she has experienced in her career. “I say, ‘I’ve attracted more than $25 million in grant funding and have had more than 60 competitive grants funded. But I’ve also had probably twice as many rejected.’ A lot of early-career researchers often find those rejections really tough to take. But I actually think you learn so much from the rejected grants.” Grant writing is a job requirement for research scientists who need to fund projects year after year. Most proposals end in rejection, but missteps give researchers a chance to learn how to find other opportunities, write better proposals and navigate the system. Taking time to learn from the setbacks and successes of others can help to increase the chances of securing funds, says Ball, who runs workshops alongside her role as a behavioural scientist at Deakin University in Melbourne, Australia. Do your research Competition for grants has never been more intense. The European Commission’s Horizon 2020 programme is the European Union’s largest-ever research and innovation programme, with nearly €80 billion (US$89 billion) in funding set aside between 2014 and 2020. It reported a 14% success rate for its first 100 calls for proposals, although submissions to some categories had lower success rates. The commission has published its proposal for Horizon Europe, the €100-billion programme that will succeed Horizon 2020. In Australia, since 2017, the National Health and Medical Research Council has been funding less than 20% of proposals it receives. And the US National Science Foundation (NSF) received 49,415 proposals and funded 11,447 of them in 2017 — less than 25%. That’s tens of thousands of rejections in a single year from the NSF alone. Being a renowned scientist doesn’t ensure success. On the same day that molecular biologist Carol Greider won a Nobel prize in 2009, she learnt that her recently submitted grant proposal had been rejected. “Even on the day when you win the Nobel prize,” she said in a 2017 graduation speech at Cold Spring Harbor Laboratory in New York, “sceptics may question whether you really know what you’re doing.” To increase the likelihood of funding success, scientists suggest doing an extensive search of available grants and noting differences in the types of project financed by various funding bodies. Government agencies such as the NSF tend to be interested in basic science that addresses big, conceptual questions, says Leslie Rissler, programme director at the NSF’s Division of Environmental Biology in Alexandria, Virginia. A private foundation, however, might prioritize projects that inform social change or that have practical implications that fit into one of its specific missions. Pitching a proposal Before beginning an application, you should read descriptions and directions carefully, advises Ball, who recently pored over 200 pages of online material before starting a proposal. That effort can save time in the end, helping researchers to work out which awards are a good fit and which aren’t. “If you’re not absolutely spot on with what they’re looking for, it may not be worth your time in writing that grant,” she says. Experienced scientists suggest studying successful proposals, which can often be acquired from trusted colleagues and supervisors, university libraries or online databases. A website called Open Grants, for example, includes more than 200 grants, both successful and unsuccessful, that are free to peruse. Grant writers shouldn’t fear e-mailing or calling a grants agency to talk through their potential interest in a project, advises Amanda Stanley, executive director at COMPASS, a non-profit organization based in Portland, Oregon, that supports environmental scientists. For six years, she worked as a programme officer for the Wilburforce Foundation in Seattle, Washington, which supports conservation science. At this and other private foundations, the application process often begins with a ‘soft pitch’ that presents a brief case for the project. Those pitches should cover several main points, Stanley says: “‘Here’s what I’m trying to do. Here’s why it’s important. Here’s a little bit about me and the people I’m collaborating with. Would you like to talk further?’” She notes that a successful proposal must closely align with a foundation’s strategic goals. Each organization has its own process, but next steps typically include a phone conversation, a written summary and, finally, an invitation to submit a formal application. “Once you’ve gotten that invitation to submit a proposal from the programme officer, your chances of getting funded are really, really high,” Stanley says. Grants manager Cheryl Smythe (left) allows for IT glitches when submitting grant proposals.Credit: Dr Louisa Wood The write stuff Applicants should put themselves in the shoes of grant reviewers, who might need to read dozens of applications about complicated subjects that lie outside their own fields of expertise, often while juggling their own research. “Imagine you’re tired, grumpy and hungry. You’ve got 50 applications to get through,” says Cheryl Smythe, international grants manager at the Babraham Institute, a life-sciences research institution in Cambridge, UK. “Think about how you as an applicant can make it as easy as possible for them.” Formatting is an important consideration, says Aerin Jacob, a conservation scientist at the Yellowstone to Yukon Conservation Initiative in Canmore, Canada. White space and bold headings can make proposals easier to read, as can illustrations. “Students are tempted and sometimes encouraged to squeeze in as much information as possible, so there are all kinds of tricks to fiddle with the margin size, or to make the font a little bit smaller so that you can squeeze in that one last sentence,” Jacob says. “For a reviewer, that’s exhausting to read.” Ball advises avoiding basic deal-breakers, such as spelling errors, grammatical slips and lengthy proposals that exceed word limits. Those kinds of mistake can cast doubt on how rigorous applicants will be in their research, she says. A list of key words, crucial for indexes and search engines, should be more than an afterthought, Ball adds. On a proposal for a project on promoting physical activity among women, she tagged her proposal with the word ‘women’. The descriptor was too broad, and her application ended up with a reviewer whose expertise appeared to be in sociology and gender studies instead of in exercise or nutrition. The grant didn’t score well in that round of review. To prevent a reviewer’s eyes from glazing over, Jacob says, use clear language instead of multisyllabic jargon. When technical details are necessary, follow up a complex sentence with one that sums up the big picture. Thinking back to her early proposals, Jacob remembers cramming in words instead of getting to the point. “It was probably something like, ‘I propose to study the heterogeneity of forest landscapes in spatial and temporal recovery after multiple disturbances,’ rather than, ‘I want to see what happens when a forest has been logged, burnt and farmed, and grows back,’” she says. Grants can be more speculative and more self-promotional than papers are, Rissler adds. “A grant is about convincing a jury that your ideas are worthy and exciting,” she says. “You can make some pretty sweeping generalizations about what your proposed ideas might do for science and society in the long run. A paper is much more rigid in terms of what you can say and in what you must say.” Getting some science communication training can be a worthwhile strategy for strengthening grant-writing skills, Stanley says. When she was reviewing pitch letters for a private foundation, she recalls that lots of scientists couldn’t fully explain why their work mattered. But when she received pitches that were clear and compelling, she was more willing to help those scientists brainstorm other possible funding agencies if her foundation wasn’t the right fit. Scientists who sent strong — albeit unsuccessful — applications were also more likely to get funding from the foundation for later projects. Science storytelling To refine project pitches and proposals, Stanley recommends that scientists use a free communication tool from COMPASS called the Message Box Workbook, which can help to identify key points and answer the crucial question for every audience: ‘So what?’ Scientific conferences often provide symposia or sessions that include funders and offer helpful tips for writing grants. And development officers at institutions can help scientists to connect with funders. “A good development officer is worth their weight in gold,” Stanley says. “Make friends with them.” Jacob has taken science-communication training through COMPASS, The Story Collider (a science-storytelling organization) and from other such organizations. She has learnt how to talk about her work in the manner of a storyteller. In proposals and interviews, she now includes personal details, when relevant, that explain the problems she wants to address and why she decided to speak out about conservation — an example of the kind of conflict and resolution that builds a good story. Jacob senses that the approach strikes a chord. “As a reviewer, you remember somebody’s proposal just that little bit more,” she says. “If you have a stack of proposals, you want to find the one that you connect with.” A clear focus can help to boost a grant to the top of a reviewer’s pile, Ball adds. In one of the first large grants that she applied for, she proposed collecting information on the key factors that prevent weight gain as well as designing and implementing an obesity-intervention programme. In retrospect, it was too much within the grant’s two-year time frame. She didn’t get the funding, and the feedback she received was that it would have worked better as two separate proposals. “While it’s tempting to want to claim that you can solve these enormous, challenging and complex problems in a single project,” Ball says, “realistically, that’s usually not the case.” Teaming up with collaborators can also increase the chance of success. Earlier this year, Ball was funded by the Diabetes Australia Research Program for a study that she proposed in collaboration with hospital clinicians, helping disadvantaged people with type 2 diabetes to eat healthy diets. Earlier in her career, she had written grants based on her own ideas, rather than on suggestions from clinicians or other non-academic partners. This time, she says, she focused on a real-world need rather than on her own ideas for a study. Instead of overreaching, she kept the study small and preliminary, allowing her to test the approach before trying to get funding for larger trials. It is acceptable — even advisable — to admit a study’s limitations instead of trying to meet preconceived expectations, Jacob adds. In 2016, she had a proposal rejected for a study on spatial planning on the west coast of Canada that would, crucially, be informed by knowledge from Indigenous communities. She resubmitted the same proposal the next year to the same reviewers, but with a more confident and transparent approach: she was straightforward about her desire to take a different tack from the type of research that had been tried before. This time, she made it clear that she wanted to listen to Indigenous peoples and use their priorities to guide her work. She got the funding. “I saw that if I tried to change it to meet what I thought funders wanted, I might not be accurately representing what I was doing,” she says. “I just wanted to be really clear with myself and really clear with the interviewers that this is who I am, and this is what I want to do.” What not to do Writing is hard, and experienced grant writers recommend devoting plenty of time to the task. Smythe recommends setting aside a week for each page of a proposal, noting that some applications require only a few pages while major collaborative proposals for multi-year projects can run to more than 100 pages. “It can take months to get one of these together,” she says. Scheduling should include time for rewrites, proofreads and secondary reads by friends, colleagues and family members, experts say. Working right up to the deadline can undo weeks to months of hard work. At the last minute, Jacob once accidentally submitted an earlier draft instead of the final version. It included sections that were bolded and highlighted, with comments such as, “NOTE TO SELF: MAKE THIS PART SOUND BETTER.” She didn’t get that one, and has never made the same mistake again. Add an extra buffer for technology malfunctions, adds Smythe, who once got a call from a scientist at another organization who was in a panic because his computer had stopped working while he was trying to submit a grant proposal half an hour before the deadline. She submitted it for him with 23 seconds to spare. “My hand was shaking,” she says. That proposal was not successful, although the scientist sent her a nice bottle of champagne afterwards. Grant writing doesn’t necessarily end with a proposal’s submission. Applicants might receive requests for rewrites or more information. Rejections can also come with feedback, and if they don’t, applicants can request it. Luiz Nunes de Oliveira, a physicist at the University of São Paulo, Brazil, also works as a programme coordinator at the São Paulo Research Foundation. In this role, he sometimes meets with applicants who want to follow up on rejected proposals. “We sit down and go through their résumé, and then you find out that they had lots of interesting stuff to say about themselves and they missed the opportunity,” he says. “All it takes is to write an e-mail message asking [the funder] for an interview.” Jacob recommends paying attention to such feedback to strengthen future proposals. To fund her master’s programme, she applied for a grant from the Natural Sciences and Engineering Research Council of Canada (NSERC), but didn’t get it on her first try. After requesting feedback by e-mail (to an address she found buried on NSERC’s website), she was able to see her scores by category, which revealed that a few bad grades early in her undergraduate programme were her limiting factor. There was nothing she could do about her past, but the information pushed her to work harder on other parts of her application. After gaining more research and field experience, co-authoring a paper and establishing relationships with senior colleagues who would vouch for her as referees, she finally secured funding from NSERC on her third try, two years after her first rejection. Negative feedback can be one of the best learning experiences, Rissler adds. She kept the worst review she ever received, a scathing response to a grant proposal she submitted to the NSF in 2003, when she was a postdoc studying comparative phylogeography. The feedback, she says, was painful to read. It included comments that her application was incomprehensible and filled with platitudes. After she received that letter, which is now crinkled up in her desk for posterity, Rissler called a programme officer to ask why they let her see such a negative review. She was told that the critical commenter was an outlier and that the panel had gone on to recommend her project for the grant, which she ultimately received. “I learnt that you do need to be tough,” says Rissler, who now helps to make final decisions on funding for other scientists. She emphasizes that whereas reviewers’ opinions can vary, all proposals undergo multiple independent expert reviews, followed by panel discussions and additional oversight by programme directors. Grant writing tends to provoke anxiety among early-career scientists, but opportunities exist for people who are willing to take the time to develop ideas and push past rejections and negative feedback, she says. “We can’t review proposals that we don’t get. Originally posted on nature.com on 20th December 2019 - https://www.nature.com/articles/d41586-019-03914-5
- Sponsored Content Article
Working Scientist podcast: Inside the NIH grant-review process
Julie Gould and Elizabeth Pier discuss how the US National Institutes of Health grant review process works. Your browser does not support the audio element. In this first episode of a six-part weekly series about funding, Julie Gould outlines the US National Institutes of Health (NIH) grant-review process and the extent to which reviewers evaluating the same applications agree or disagree. Is the current system the best way, she asks Elizabeth Pier, lead author of a March 2018 paper published in Proceedings of the National Academy of Sciences, Low agreement among reviewers evaluating the same NIH grant applications. Paid content This episode concludes with a slot sponsored by the European Research Council. Jean-Pierre Bourguignon, its president, outlines the organization's role and remit as a grant funder. TRANSCRIPT Julie Gould and Elizabeth Pier discuss how the US National Institutes of Health grant review process works. Julie Gould: Happy 2019! I hope you’ve all managed to take some time to celebrate. As it’s a new year and new years often come with a makeover in one form or another, the Nature Careers team decided to give the podcast a makeover. As well as a new name, we’ve also got a new format. So, instead of our monthly episodes, we’re going to be producing more episodes in 2019 and grouping them together into different series, featuring six weekly episodes followed by a short break. So, here’s series one – funding – and as an added extra, each episode in this series will end with a ten-minute sponsored slot from the European Research Council. So, without any further ado, let’s go.... (Theme music) Hello, I’m Julie Gould and this is Working Scientist, a Nature Careers podcast. Grant funding plays such an overwhelming role in the career of an academic scientist, and the funders are all too aware of it. Now, I know that all researchers spend many sleepless nights and cups of coffee writing grant proposals, so when I first started doing the research for this series, I wanted to find the best experts to give you the best tips on how to write the best grant proposals to make things a little bit easier for you. But then I came across a research paper that made me stop and reflect. In March 2018, Elizabeth Pier – who was then a PhD student at the University of Wisconsin, Madison, in the Educational Psychology department – published a paper as part of her thesis in PNAS. The paper was entitled "Low agreement among reviewers evaluating the same NIH grant applications." When I first read this title, I thought, "Wait a minute, I thought the idea of the whole funding process was that the top proposals were being funded, the ones where everyone in the peer review system agreed that these were the best ideas supported by the best researchers to do the work." But clearly, this title shows that there’s something else going on in the background, so I wanted to find out more. The research was funded by the National Institutes of Health, which commissioned an independent study to examine the potential for bias to enter into the peer review process. The overarching goal of the whole project was to look for evidence of gender or racial bias, based on the characteristics of the PI or the application, and where in the process these biases might enter. Now, this particular piece of research from Pier is just one of the studies. It recreated a peer review panel to see how these meetings unfold and how they affect the decision-making process. Using previously accepted NIH project proposals, Pier explored this as part of her research. But before we go any further, it’s worth me outlining some of the basic steps of how the NIH proposal review system works. Now, just so you’re clear, these steps are the bare bones and they miss out a lot of the details, but they should give a flavour of what happens once you hit submit. So, the NIH uses a two-stage review process. In the first stage, between two and five reviewers individually evaluate each grant application, and they rate them using the NIH’s nine-point scale, with one for exceptional and nine for poor. They also record what they feel are the application’s strengths and weaknesses. The reviewers will then meet for what’s called a study section meeting to discuss their preliminary ratings. The discussion only looks at the top half of all the applications they have evaluated. The study section members then collectively assign a final rating, and this is averaged into a final priority score. So, that’s stage one. Then in the second stage, members of the NIH advisory councils use this priority score and the written critiques from the reviewers to make funding recommendations to the Director of the NIH institute or centre that awards the funding. So, given all that, I spoke to Elizabeth Pier who now works as a research manager at Education Analytics to find out more about her research. Elizabeth Pier: In this particular study, I was really interested in looking at the degree of agreement between different reviewers and what is even happening before the reviewers come together and how are reviewers going about scoring these applications based on their assessments. So, another way of putting that is: Are the reviewers agreeing not only on the score that they assign, but are they also identifying similar strengths and weaknesses in the critiques that they write prior to the meeting, and also what’s the relationship between that numeric score and the written evaluation? Julie Gould: So, there were some sobering results... Elizabeth Pier: We found that numerically speaking, there really was no agreement between the different individual reviewers in the score that they assigned to the proposals. We also found that when we were looking at the relationship between the strengths and weaknesses, written proposal, and the score that was assigned, we did see a relationship between the number of weaknesses that a reviewer would identify in their critique and the score that the reviewer assigned, but that relationship between the weaknesses and the score doesn’t hold up between different reviewers. Julie Gould: Another way of saying this is that the individual reviewers were really consistent – the more weaknesses they identified in the proposal, the lower the score awarded. But unfortunately, it appeared that each reviewer had a different idea of what a weakness is and what score that meant the proposal would ultimately be given. So, what this means is… Elizabeth Pier: ...we can’t really compare the evaluations of different reviewers and the degree of disagreement that we see in the scores seems to be a reflection of a different sense of calibration in what constitutes a bad score versus a good score. Julie Gould: The reviewers do come together for a meeting to discuss the papers based on the initial reviews, and in the meetings that Elizabeth Pier recreated… Elizabeth Pier: As you would predict and as people told us based on their intuition participating in these kinds of meetings, the range of scores does get smaller after discussions, so there’s a degree of consensus building within individual peer review panels, but the agreement between different panels actually got wider after discussion, and we had a unique opportunity here because we had four different panels that were evaluating the same applications. So in practice each application is only evaluated by one study section but for the purposes of this study we exploited that we had these four different groups looking at the same proposals. And so, in the process of building consensus within a given panel, different panels actually went further apart. Julie Gould: So really the outcome that you’re coming to is that it’s potentially better that these reviewers don’t meet? Elizabeth Pier: Our studies haven’t indicated any value or benefit in the sense of improving the consistency or reliability of the process. Julie Gould: But what about the variability in the quality of the proposal being discussed, doesn’t that make a difference? Elizabeth Pier: But we had to ask people to donate their applications and the summary statement that they received to us and the donations that we received just happened to be funded. And so, we tried to say that above a certain quality threshold, our results suggest that it’s essentially a random process and the meeting doesn’t seem to remove that randomness. However, I will say a caveat to that caveat is that the applications that get discussed in the meeting have already gone through triage, so only the top 50% of applications based on their preliminary score even get discussed in the meeting. So, what we are talking about is given that top 50% of proposals, after you’ve already excluded the ones that really have no chance of being funded initially, there really is a lot of randomness, but even more so, there’s already randomness such that the applications that have been weeded out so to speak and don’t get the opportunity to be discussed in the meeting might actually have a lot of merits. # Had it been assigned to a different panel with different reviewer it very well could have gone on to be discussed. Julie Gould: So, what you’re saying really is that luck plays a very large role in whether or not your research gets funded. Elizabeth Pier: Yes, that is what our results suggest. Above a certain degree, if you have a relatively competitive application, there aren’t any major issues that would immediately disqualify it to any kind of representative effort in the field, there’s a great deal of randomness and luck that we find in determining who does and does not get funding. Julie Gould: So, what does that mean for all those people who are spending all these hours and hours and hours and hours and hours on getting their funding applications organised and sorted and written up? I mean my heart goes out to them... Elizabeth Pier: Yes, my heart does as well. I mean it’s one of the reasons I studied this for my dissertation because it’s incredibly important for individual careers and also incredibly important for the progress of science, right? We want to make sure that the most deserving ideas are getting rewarded and funded and that it’s not just picking out of a hat. So, I mean there are a couple of pieces of silver lining. I think that we see evidence, especially as grants get resubmitted, that being responsive to reviewers’ critiques can play a strong role in conveying to reviewers improvement over time, and so there is something to be said for if you get rejected or you don’t get funded, having some tenacity and resubmitting that application and doing everything you can to address the reviewers’ critiques and feedback. It can make a potential difference. I also think that it’s important for folks not to take it personally. As academics, we definitely are used to rejection and used to plenty of times when we think we have really great ideas and reviewers of manuscripts or of grant applications don’t seem to agree with us, so I would encourage people to take a little bit of solace in that it’s not necessarily a reflection of the quality of the ideas but it’s more kind of a feature of the process. Julie Gould: How would you suggest then that the process is improved? Elizabeth Pier: There should be some assessments of whether what some scholars have called a "modified lottery system" could do. So, the idea being that there’s some initial screening process that experts do conduct to make sure, like I said, they’re kind of weeding out any really problematic proposals, things that are just wildly out of left field in terms of being feasible to complete given the budget or things like that. And then after that kind of initial screen then it really is just a random selection. And the reason I think that would be an improvement is because if the process is already random above a certain quality threshold, which our study suggests it is, we might as well save the money and the time involved to convene thousands of people and spend millions of dollars to have these meetings if the outcome is essentially the same as a random process. Julie Gould: Now, we’ll touch on the idea of a lottery-style funding system later on in the series, but what we can say now is that change is going to be slow – it always is in academia. Is there anything that can be done in the meantime, before this lottery style system or something completely different is created? Elizabeth Pier: Starting to accept the fact that it’s not a completely objective process, that humans are fallible, they are subjective, and when you’re asking experts to make very complex judgements about the potential likelihood of success of a project, that’s a really difficult decision that’s going to bring in a lot of heuristics and biases that go into their decision making. Julie Gould: My final question to Elizabeth was what advice have you got for anyone who’s currently writing a grant proposal to the NIH. Elizabeth Pier: One piece of advice, which is probably pretty obvious but I will say is backed by our findings, is that weaknesses are much more predictive of the score that reviewers will assign, rather than strengths. So, what that means is minimise as many weaknesses as you can. Julie Gould: So, after all of that, I’m intrigued. What do you think? How would you feel about a more lottery-style funding system? Please send in your thoughts to the Nature Careers team which you can do via Twitter, Facebook and LinkedIn. And over the next two episodes, I’ll be speaking to different experts on how to minimise the number of weaknesses within your funding application, in the hopes – fingers crossed – that you’ll have a bit more success and a bit more luck. Now, that’s all for this section of our Working Scientist podcast. We now have a slot sponsored by and featuring the work of the European Research Council. Thanks for listening. I’m Julie Gould. (Theme music) Jean-Pierre Bourguignon: So, my name is Jean-Pierre Bourguignon and my title is President of the European Research Council, which of course is supported by the European Union as through the European Commission. I’m a French mathematician, I should say. I spent most of my career in the CNRS (Centre National de la Recherche Scientifique). My field was differential geometry, but did a lot of work actually at the boundary of theoretical physics, general relativity and Dirac operators and these kind of topics, but still always as a mathematician. The European Research Council is actually an interesting story. It was created in 2007, so it’s now 11 years of age, and it was a long process. Myself, the first time I heard about the possibility of having an ERC was 1995, and it was a long effort by the scientific community, and step after step we had to convince people in the Commission, people in the European Council – namely the countries – that they should support such a project, but still it has a lot of very specific characteristics, particularly the power which has been given to its scientific council is considerable. It really was an innovation and the council has the responsibility of deciding on how to spend the money and how to do the evaluation. This is unique in the setting of the European Commission, that a group of 22 scientists are given such a responsibility and of course as President of the European Research Council, I have some very specific ones, which is to confirm the list of people who are granted and really guarantee the quality of the work done. The mission of the ERC was really to make Europe more attractive, to be a place where science can develop really in the most ambitious way and to push the ambition, particularly of young people, upward, that is to make them independent early enough and to take their vision on board. You know, we are at the stage of giving 1100 research grants this year, which is of course a very significant amount of money. The budget is now really over €2 billion per year, and we are covering all fields of science – that is physical sciences, engineering, including maths, computer science, and so on, life sciences, social sciences and humanities. If you want to know what you are doing, you need to talk and meet and discuss with the people you are funding and so I do travel a lot, particularly in Europe, to meet the people we call our grantees – the people who get the grants from the ERC – and this part of my job is really extremely worthwhile and extremely rewarding because the selection process is a very tough one – the typical success rate at this moment is 13%. And it means that people have all proposed very ambitious projects that are conditioned to be successful at the ERC, high risk again, we need to encourage the panels who are selecting people to really accept to take risks. And that’s one thing I hear regularly from grantees, telling me, ‘I submitted a very similar project to my national agency but then I was not funded, it was considered too risky, then I submitted to the ERC, and then the ERC funds me,’ so it makes a big difference. Another component which is very important in our strategy is the fact that the clause, which has been put in place by the European Research Council. Really, we have three categories at the moment for the individual grants, which is the starting grants, consolidator grants and advanced grants, and it means timed to PhD. So, starting grants – to 2 to 7 years, consolidator grants – 7 to 12 years, and advanced grants – there is no condition, it just means people who are already confirmed, and while doing that it means that in the end we are dedicating typically two thirds of our budget to the younger people, people who are typically below 40 years of age. Very often people get the belief that really if you are not from one of the leading research institutions in Europe, then you have no chance. This is not the case. I mean the institution which is your host institution is not part of the evaluation, which is really the key for the evaluation is the project. You have to show that you have thought of what kind of resources will be needed, and you describe them, but this is not the institution as such which is very important. So, it means that in particular in terms of the support we give, part of it could be also buying expensive equipment if you need them and if it’s not available in your institution. So, we consider the project, not just as helping the people, but helping the people also to set up the environment which will make it possible to get the project through. So, this is sometimes one misconception that people have, that they get the feeling that if they are coming from a smaller place, they have no chance. The number of institutions the ERC has been signing with is close to 800 now, so of course it’s quite a significant number of institutions based in Europe and of course some of the leading ones got more grants than others, but definitely even small institutions have been very successful at the ERC. One of the key things that the ERC is doing is empowering researchers. This is something very, very important for us, and a very good example for this is one of the specific characteristics of the ERC programme which is called portability, but the host institution is not part of the selection criteria – it’s just here to make sure that there is a legal body that is able to receive the contract and it gives a lot of power to the researchers. And one of the typical powers, what I mentioned, portability, which means that the researcher can change the host institution if he or she feels that it’s not given the proper treatment, or maybe they could give other personal reasons to do that. This is the whole philosophy behind it – we really want the researchers in the driver’s seat at the level of the council but also at the level of how they run their contract, and of course there is an institution behind it because you want to be sure that there is a legal basis for this, but we really want the researchers to be able to do their research in the best possible conditions. The map I have in front of me, which is the map of the world, has on top of it one of our mottos which is ‘Open to the world.’ One of the conditions to be funded is that you have to spend at least 50% of your time in Europe, but you can be from any country, and we want to be sure that Europe is the leader to tackle some of the most challenging scientific problems. At the very beginning, we could notice that the percentage of women who were applying to the ERC was less than the percentage of women in the scientific community, and we felt this was definitely not adequate. Also, we had for the ones who applied, the success rate was definitely lower than the success rate for men. Through very sustained efforts, and identifying the issue from the very beginning, I think we made very significant progress. So, first of all, the percentage of women applying to the ERC has been steadily growing. We are now basically at the level where the percentage of women applying to the ERC is very similar to the level of the percentage of women in the age group of the different goals we have. So, from that point of view, I think we really achieved something which means that there’s no some kind of resistance or reluctance of women to apply, so this is one step. And then, of course very important but I think the two are linked – the fact that in recent years, the ERC women have been on average more successful than men. It’s a very slight difference but since we started the situation shows the opposite. We are very pleased that all the efforts we made, particularly to tackle complicit bias or various other things have been more or less successful. I’ve been visiting many, many countries in Europe, in particular countries in the "EU13," because I feel you need to understand the real situation people are exposed to in the various countries, and they are the ones who joined Europe the most recently, and most of them located in the eastern part of Europe and it’s very important to realise that actually that situation can be quite different from one country to the next. It has to do with teaching load, it has to do with the power structures in the institutes, it has to do of course with the support which is available to people, so for me it was very, very important to meet the researchers because that’s for me the key point. Also, to meet the authorities in these countries and to understand in which environment they are operating because I think that’s the very best way. We, at the level of the European Research Council, have also introduced some help, in particular by encouraging the various countries, could be also regions, to support possibilities of researchers in typically underrepresented regions (it could be EU13 countries) – to really give them some possibility of spending some time with the support of their countries or their region, in ERC teams, so that they understand what it takes to submit a proposal, but also to understand better, to really also test their ideas with other people so that then they have a much better idea what it takes to submit a proposal and therefore they are better prepared personally, not just intellectually, to really submit a proposal in good conditions because they have seen what a difference it makes and also what also kind of effort you have to put in if you want to be successful. Originally posted on Nature Careers - 04 January 2019 - https://www.nature.com/articles/d41586-019-00016-0
- Career Advice
Early-career funding sources: you will not find what you do not seek
During my postdoctoral training at the University of Cambridge, UK, I reached the final round of applications for a Sir Henry Wellcome Postdoctoral Fellowship, offered by the research charity Wellcome in London — but my research proposal was ultimately rejected. After getting over my initial disappointment, I chose to seek out less obvious funding sources. I realized that the major sources of financial support for early-career researchers are hugely competitive: success rates usually range from 10–40%, and failure is deflating and time-consuming. So, I looked elsewhere. Over the next few years while still at Cambridge, I managed to secure funding from a wide variety of sources, including smaller charitable bodies, pharmaceutical companies, life-science publishers, university departments and research societies. I received travel grants from several organizations — Thrombosis UK, a charity based in Llanwrda, west Wales; the University of Cambridge School of Biological Sciences; The Company of Biologists, a charity in Cambridge; and Cayman Chemical, a biotechnology company in Ann Arbor, Michigan. I also won early-investigator awards — including the Thrombosis & Haemostasis society of Australia and New Zealand in Darling, Australia; the Frontiers in Cancer Science conference; Wolfson College at the University of Cambridge; and the British Society for Haematology in London. These individually modest awards generated a cumulative body of evidence of my ability to find and secure funding — eventually just more than £50,000 (US$66,000) over 6 years — to support my progression from unfunded, newly qualified scientist to funded research-faculty member. There are several reasons to apply for money from unusual sources. Perhaps you don’t have enough time to complete a full proposal for a major funding scheme. Or maybe you are attempting to bolster your CV before applying for a promotion. There doesn’t have to be a specific rationale; your choice might just be a good place to apply. But this is not to say that established funding sources should be ignored, or that it is easier to gain funding through less obvious sources. In fact, by no means were all of my attempts successful. Applications to alternative funding sources can provide valuable training in grant writing; in my case, my improved grant-writing skills contributed to the receipt of a career development award from the American Heart Association, based in Dallas, Texas. It is difficult to say whether early-stage research awards from less apparent sources will change the course of a career. Regardless, in my experience, these four principles could improve your chances of a successful career: 1. Be transparent. Talk to your supervisor(s) and give them with details about the application. Perhaps offer to provide the information that you would like to be included in a letter of recommendation. If necessary, politely remind them of the reasons you want to apply, and that your success would reflect favourably on their laboratory. Fortunately for me, my supervisors have been supportive of my applications — but many group leaders might not have enough time to go through all your applications in detail, and could even question the value of applying for such awards on a regular basis. Hopefully, they will at least be willing to provide you with permission to apply. 2. Cast a wide net. Funding schemes can be identified using online search engines, such as Research Professional and Funding Institutional. Researchgate also offers a search engine for funding competitions for US-based members. Other useful resources for identifying non-standard funding schemes include the acknowledgements sections of academic papers and presentations, as well as your colleagues and peers. Universities and their departments often provide schemes for seed funding, research exchange placements or other internal funding. Pharmaceutical companies might advertise collaborative grant opportunities aimed at researchers in translational sciences (such as the opnMe schemes from Boehringer Ingelheim in Ingelheim am Rhein, Germany) or travel awards that can be used when presenting work at academic conferences (the Cayman Chemical travel grants, for example). Financial support to attend international meetings is commonly available through competitions from the organizing body of the meeting (such as the travel scholarships from the Keystone Symposia in Silverthorne, Colorado), and even from publishers or individual journals (for instance, the Disease Models & Mechanisms travel grant). 3. Pay attention to details. Identify the funding opportunities that are appropriate for your career stage. Funders’ eligibility guidelines for their schemes will help with this. It is usually possible to find a published list of award recipients, whose positions you can compare with your current career stage. And note that, the layout and presentation of your application is almost as important its scientific content. An audience will be impressed by a tidy, well-organized and well-presented piece of writing, as well as by what the words themselves say. 4. Learn from your mistakes. Peer-reviewed funding schemes are competitive; researchers will probably experience rejections more often than successes. Although these rejections can be disheartening, don’t let them demolish your confidence. The peer-review process is subjective, and the opinions of one reviewer do not necessarily represent those of the scientific community. At the same time, do not ignore your reviewers. In fact, carefully read the reviewer comments and incorporate their suggestions into your next submission — which should improve your proposal. Finally, remember that the same, or a slightly tweaked, proposal can sometimes be used to apply for more than one award. In the end, most investigators are likely to obtain the majority of their funding from the larger national funding bodies. However, the lesser-known funding sources can be useful when attempting to accumulate evidence of scholarly performance and productivity, even for those who will eventually leave academia. FINDING FUNDING There are many places to check for funding sources apart from major national agencies. Examples of search engines for funding sources: Research Professional Funding Institutional ResearchGate Examples of fellowships and research grants: Boehringer Ingelheim opnMe collaboration proposals Cambridge British Heart Foundation Pump-Priming Grants Parke Davis Exchange Fellowship Wolfson College Cambridge Junior Research Fellowship Wolfson College Oxford fellowships British Society for Haematology grants Lurie Children’s Hospital internal funding opportunities Examples of travel grants: Thrombosis UK grant opportunities Disease Models & Mechanisms Conference Travel Grants Cayman Chemical Conference Travel Grants Thrombosis & Haemostasis society of Australia and New Zealand travel grants American Association for Cancer Research travel grants American Heart Association travel grants Keystone Symposia scholarships European Respiratory Society grants Originally posted on nature.com on 20th December 2019 - https://www.nature.com/articles/d41586-019-03873-x