Information in this document referring to "Cyprus" refers to the southern part of the island. Note from all Member States of the European Union, the OECD and the European Union.
Conclusions
According to the evaluation of experts, AI's potential performance in literacy is close to that of adults with proficiency at Level 3. Across the OECD countries in PIAAC, on average 90% of adults are at or below Level 3 in literacy and only 10%.
1 Setting the stage: Approaches to assessing AI’s impact
The study assesses artificial intelligence (AI) capabilities by collecting expert judgments on whether AI can perform tests from the OECD's Survey of Adult Skills of the Program for International. Assessing AI capabilities in these domains is indicative of AI's potential impact on work and life, as literacy and numeracy are relevant in most social contexts and work situations.
Previous studies that measure AI capabilities and impact
Some studies draw information from patents to measure the usefulness of artificial intelligence and robotics in the workplace. The demand for AI experts in companies can serve as a proxy for the use of AI in the workplace.
Objective for the exploratory assessment of AI capabilities
By filling these gaps, measures that rely on expert judgment can contribute to a more complete assessment of AI capabilities. Both the pilot and this follow-up aim to explore the assessment of AI skills with the Adult Skills Survey using expert judgement.
Plan of the report
Seamans (2019), “The Changing Impact of Artificial Intelligence on Work: The Role of Complementary Skills and Technologies”, SSRN Electronic Journal, https://doi.org/10.2139/ssrn.3368605. OECD (2021), AI and the Future of Skills, Volume 1: Skills and Assessments, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/5ee71f34-en.
Notes
2 Evolution of human skills versus AI capabilities
This chapter first analyzes the skill level of adults aged 16 to 65 in the areas of literacy and numeracy and shows how it changes over time. The chapter then provides an overview of recent technological developments in the fields of natural language processing (NLP) and AI quantitative reasoning.
Changes in skills supply
Comparison with the IALS results shows that literacy skills of the working population have declined slightly over time. Meanwhile, the margins of the skill distribution - the shares of workers with the lowest and the highest numeracy - have increased.
Recent developments in AI capabilities
Performance also improved in Natural Language Inference, the task of "understanding" the relationship between sentences, e.g. For example, the word "rose" would be represented differently in the sentences "The roses are red" and "The sun rose."
The importance of measuring AI capabilities
Supplementary tables
4 The leaderboard for SuperGLUE can be found at: https://super.gluebenchmark.com/leaderboard (accessed 21 January 2023). It first provides an overview of the PIAAC test, the skills it measures, and the test questions used to measure them. The chapter concludes with a summary of the methodological challenges encountered in the study and the attempts to resolve them.
3 Methodology for assessing AI capabilities using the Survey of
The chapter then describes the methods used to select experts, to collect expert judgment, to develop the questionnaire and to construct composite measures of artificial intelligence (AI) abilities in literacy and numeracy. In 2016, the OECD asked a group of computer scientists to assess the capabilities of computers in relation to the core skills measured in the Survey of Adult Abilities within the Program for International Assessment of Adult Abilities (PIAAC) (Elliott, 2017[1] ). . This chapter describes the approach used to assess AI capabilities and the methodological improvements made during the course of the work.
Overview of the Survey of Adult Skills (PIAAC)
Sometimes this is relatively easy, as the required information is stated directly and clearly in the text. These relationships may be explicitly indicated (eg the text says that "the cause of X is Y") or may require the reader to make inferences. Mathematical information can be further expressed in text form (eg "the crime rate has increased by half").
Identifying a group of computer scientists
Department of Intelligent Systems, University of Memphis Natural language processing, natural language-based knowledge representations, semantic similarity, question answering, intelligent guidance systems. Cognitive Systems Group, IBM Artificial Intelligence, Cognitive Systems for Holistic Service Systems Michael Witbrock, Professor, School of Computer Science,. University of Auckland Artificial Intelligence, AI for Social Good, AI Entrepreneurship, Natural Language Understanding, Machine Reasoning, Knowledge.
Collecting expert judgement
Computational Linguistics, Discourse Analysis, Artificial Intelligence, Parsing, Syntax, Semantic Interpretation, Information Retrieval, Knowledge Representation, Coding Common Sense Knowledge Aviv Keren*, Senior Applied Scientist, Anyword Artificial Intelligence, Philosophy of Mathematics, Mathematics. In such a case, these experts should be able to communicate this information to the team at any point in the evaluation process. Alternatively, they can share their views with the group throughout the data collection process.
Developing the questionnaire
Rather than endorsing the "two-stage" solution, the survey included an additional open-ended question: "If you think AI can't do the whole task or you're not sure about it, would you say AI can do part duty. If so, which parts?" This question was intended to specify the elements of the task that are easy and difficult for the machines to perform.At the end of the literacy and numeracy parts of the test, the experts could report any difficulty in understanding or answering the questions on the domain or leave any comments or suggestions.
Constructing aggregate measures of AI literacy and numeracy performance
However, such an exercise would require experts to agree on a set of categories to describe different types of capabilities and to identify those required for each task. Finally, a follow-up survey asked all experts to predict AI capabilities against each PIAAC question in 2026. A pilot study asked experts to predict technological improvements ten years into the future.
Challenges and lessons learned
To reach agreement, experts needed clarification on the generality of the AI capabilities being evaluated. This chapter describes the results of the follow-up assessment of computer skills with the Survey of Adult Skills (PIAAC). It first presents the results of the literacy assessment and then the results for numeracy.
4 Experts’ assessments of AI capabilities in literacy and
Participants rated the potential performance of current artificial intelligence (AI) in relation to each of the questions in the literacy and numeracy domains of the PIAAC. The chapter first discusses the results of the literacy assessment and then the results for numeracy. In terms of literacy, the results suggest that current computer technicians can perform about as well as adults at level 3 proficiency on the test.
Evaluation of AI capabilities in the domain of literacy
The AI is expected to solve all questions at level 1 and below and 93% of questions at level 2, by simple majority vote. At Level 3 and Level 4 and above, the AI is expected to answer about 70% of questions correctly. They show that the average judgments of all experts are in the upper middle of the AI performance scale.
Evaluation of AI capabilities in the domain of numeracy
Supplementary tables
The chapter analyzes the changes in the estimated literacy and numeracy capabilities of artificial intelligence (AI) between 2016 and 2021. To this end, it compares the majority responses of the expert groups that completed the pilot and follow-up assessments. The chapter also examines the level of expert agreement and the prevalence of uncertain responses in both assessments to compare the quality of group assessments obtained in 2016 and 2021.
5 Changes in AI capabilities in literacy and numeracy between 2016 and
It then analyzes expert predictions of how AI capabilities will develop through 2026 to gain information on the likely direction of AI advancement in the near future. Regular assessment of AI capabilities can provide information on the direction, speed and content of AI technological developments. This chapter compares the results of the 2016 and 2021 assessments to examine how AI capabilities in literacy and numeracy have developed over this period.
Change in AI literacy capabilities over time
Comparisons of AI performance ratings over time can be biased by changes in the composition of the expert groups that provide the ratings. First, Figure 5.4 provides an overview of the average literacy scores of all experts in the 2016 and 2021 measurements. Experts in the follow-up study expected that AI would be able to answer all literacy questions by 2026.
Change in AI numeracy capabilities over time
Supplementary figures
Figures A5.1 and A5.2 in Annex 5.A provide additional analyzes comparing the results of both assessments according to the aggregation rule used in 2016. At each level of question difficulty, AI performance in literacy was assessed with the average expert rating higher in 2021 than in 2016. Similar to the results relying on the majority of experts' ratings, the results relying on averages show lower AI numeracy performance in 2021 than in 2016.
6 Implications of evolving AI
To this end, it analyzes the use of literacy and numeracy skills at work and the skills of workers who use these skills on a daily basis.
It considers the potential impacts of computer skills development on employment by looking at the use of literacy and numeracy skills at work. This focuses on the need to develop skills in the population that are beyond those of technology and the importance of diversifying people's skill sets to enable them to compete, but also work together, with AI.
Summary of results
This suggests that AI could potentially outperform a large portion of the population on the PIAAC literacy test. The evaluation approach used in the follow-up study is comparable to that of the pilot. The latter allow models to capture relationships in longer paragraphs and "learn" the meaning of words in context.
Policy implications of evolving AI capabilities in literacy and numeracy
Percentage of workers with different skill levels who use literacy and numeracy skills on a daily basis at work. Daily use of literacy and numeracy practices at work Percentage of workers who report using the practice on a daily basis. Everyday use of literacy and numeracy at work together with other skills Percentages of all workers.
A new approach to assessing AI
Additionally, the project will map AI performance measures to information about worker skills. Successive volumes in this series will describe the development of the AI suite of measures and the project's explorations into their use. 1 The use of each skill is assessed by a number of variables from the PIAAC: writing – the frequency of writing letters, memos or mails; articles; reports; or filling out forms; digital skills – frequency of internet use for mail; to find information about work; for carrying out transactions; frequency of using tables;
Is Education Losing the Race with Technology?
AI’S PROGRESS IN MATHS AND READING
