A new procurement panel for AI-enabled research performance data and analytical services will be accessible to all government agencies as a part of the Australian Research Council’s reform of its own evaluation programs.
Last week, the Australian Research Council (ARC) issued a request for tender seeking a supplier with capabilities in providing research output metadata and citation information, as well as analytical services and advice.
Experience in the “ethical and effective application of artificial intelligence in research evaluation, policy development, and scientometrics” is a requirement, according to tender documents. This supports the development of a data curation algorithm under the ARC’s new data framework.
The panel has been established as the ARC transitions to a research quality assessment program that employs a “modern data-driven approach”. It replaces the agency’s long-standing Excellence in Research for Australia (ERA) program, which has relied on resource-intensive university submissions.
Education minister Jason Clare called on the ARC to develop an ERA transition plan in August 2022 and confirmed the assessment program would not continue in its current form in August 2023, as a part of the government’s response to a review of the ARC Act.
While the ERA was deemed to be “robust” and is given credit for uplifting “competition and quality across the sector,” it placed a heavy reporting burden on universities, the ARC’s ERA transition plan says.
In the 2018 round of the ERA, 42 universities made submissions. This included over 650 explanatory statements, 3,000 research statements for non-traditional research outputs, and made 60,000 full-text outputs available for peer-review.
The ARC wants a new approach that can “use data harvesting from commercial publishers and third parties to gain information about 75 per cent of the research publications previously submitted”, although this is mostly from STEM disciplines.
According to the ERA transition plan, the ARC has been working with data providers to develop an AI algorithm to automatically curate data into meaningful categories, replacing the need for university staff to manually review individual research outputs. It plans for this to be ready to use in the second or third year of the reform program.
This is a key support for the development of a new data framework that aims to remove the need for university submissions and enable flexible performance evaluation. This could support two to four deep-dive evaluations every year in priority areas like indigenous studies, quantum computing, climate change, or food and agriculture.
From the fourth year of the program, the ARC wants research data to be collected through “full smart harvesting and curation” and be connected with other government data sources such as Australian Research Data Commons, Department of Industry, and IP Australia.
The new data framework will also feature new engagement and impact indicators, standardised metadata practices across all universities, and the development of data capabilities and workflows to collect non-harvestable data from universities.
Standing offers for suppliers of the services will last between financial year 2024-25 to 2026-27, with an option to extend for an additional three years depending on future requirements. It is expected to be operational in the third quarter of 2024, according to an ARC spokesperson.
The tender will close to applications on March 14 and there is no limit to the number of providers that can sit on the panel.
Work to improve research evaluation and assessment metrics has been ongoing across the federal government. This includes the release of a report on ‘modernising research assessment’ by the chief scientist last November. The report highlighted the need for new metrics that didn’t focus solely on publications and citations. Similar work is also a priority of the Trailblazer Universities program.
Do you know more? Contact James Riley via Email.