LLM Benchmarking: Amazing Task Complexity Benefits

The main objective of many large language models (LLM) is providing compelling lessons which is as close as possible to be unlikely from human writing. And it is a major reason to gauge the relative performance of LLM using the traditional benchmarks so hard: the quality of writing is not necessary with the matrix used to traditionally measure the performance of the processor, such as the instruction performance rate.

Connected: Large language models are improving rapidly

But researchers from Berkeley, California. Think tank meter (for) Model evaluation and danger research) Came with a simple idea. First of all, identify a series of tasks with different -complications and record the average time for a group of humans to complete each task. Then various versions of LLM have completed similar tasks, in cases in which a version of LLM successfully completes the task with some level of reliability, says 50 percent of the time. The resulting data plots confirm that as the time moves, the gradual generations of an LLM can complete longer and longer (greater complex) functions.

No surprise. But the shock was that this improvement in LLM’s capacity has been firmly done to complete difficult tasks InconsequentWith a dual period of about seven months.

IEEE spectrum Reached out Megan KinnamementOne of the authors of A METR research paper Describing this work and its amazing implications.

Evaluation of LLM Performance Matrix

Did you suspect that you would get these results?

Megan Kinnamement: I, at least individually, we did not expect we explain it as an incomplete as we did. Models are definitely getting better quickly, though. Therefore, some fast rates of progress were not completely unpredictable.

As you indicate in paper, it is always dangerous to see and extreme extracts in the future. However, you suggest that this continuity is likely, which means that by 2030 we will be looking at the work of the month within the most advanced capacity. Big language model.

Kinniment: Let’s take a look at him. Up to a month, we mean about 167 working hours, so the number of hours (human) work in a month. And it is at 50 percent reliability. But prolonged tasks require high reliability usually to be useful. So this is something that can create in-act, real world, economic impact is not as intense as it is predicted.

There are many things that have to continue for this prediction. The hardware will have to continue to improve the rate that is improving it; Software has to be improved. You will have adequate training data and the availability of that training data to continue training on breathtaking clips in recent years.

Kinniment: The forecasts and the dates we have found are only extracting the trend that we see on our task suit. (There are trends) Do not take into account real-world factors or calculation-scaling changes.

If a large language model can somehow achieve the ability to complete 167-hour types with 50 percent credibility, then there are types of things that now put the ability for a big language model?

Kinniment: Well, the big we often think is accelerating the AI ​​R&D Research. To the extent that you can create models that accelerate your company’s ability to make better models, you can end in a situation where AI abilities actually develop quite rapidly.

What you are describing reminds of this idea Eccentricity, where you have AIS, which is making other AIs in itself, is not aided by humans.

Kinniment: I think you can achieve acceleration that is quite intense and make things more difficult to control things as a result of this mass explosive increase. It is due to thinking that you may have various hurdles that slow down things in practice. Even if it was the case that we had a lot of, very clever AIS, then this pace of progress could still end the bottleneck on things like hardware and Robotics. But yes, eccentricity is definitely an idea that is relevant to this entire region of things.

Things can go very quickly, but it’s not that it is Eccentricity or nothing. (AI-development rate) which were lighter than a uniqueness, can still be quite intense for how the world needs to be adapted.

Kinniment: I think this is really a relatively serial thing Chat, and potentially before that. They are less likely to get stuck. When things are not working, they are a bit better in changing strategies, but it is a little hit or miss. And they are definitely much better in doing things, as they were better and better to use devices. But it seems that there are some fundamental aspects that have not changed much. One thing I like to see when I get a new model, on each task, we give a number to the model TokenMany words that can say this. And if you can imagine giving them more and more time or more time to do a task, how does it affect how much they are likely to succeed? And basically, what we see is quite firmly. There is a point on which you give them more tokens and it does not really help. And for each new model, that plateau gets slightly higher.

Megan Kinnime was in the team at Metr who published the results of a study of LLM performance.Megan Kinnamement

I imagine that I imagine there is low returns. But if you give a human lot and much time to do something, they will probably do a better job, especially if you have many human beings. And I think I would be very impressed by a big language model, even if its full score was low, it seemed that it could only keep doing things and improve. This can be a big thing.

You found that the models performed poorly on the tasks that had high “mess” scores. Was there any indication that you exited the data that this situation was changing? In other words, that model can gain more ability to handle tasks that had high disturbances?

Kinniment: Messays was a solution that I created to try and get quantitative measurements to some extent how our tasks were compared to the real world. And most of our tasks are not that mess. This is a 16-point scale. Meaning is about 3, and the dirty tasks are about 8 out of 16.

So what will happen in case of a 16 task mess?

Kinniment: Something like this Steering, where you have a lot of resource limitations. This is very punished. You have agents that are actively adapted against you. It is easy to mess. It is a novel.

Kinniment:Openai published O3And O3 was a little more capable than anticipated given the tendency. So we are following some amounts in terms of measuring other models. We want to focus on informing the world about AI development and frightening risks from the AI ​​system.

Horrific risk from advanced AI

What are the most potential horrific risks from AI? I mean, people coming to my mind have massive dislocation in employment if and when AI becomes supremely capable.

Kinniment: When we are talking about terrible risks, we are not only talking about unemployment on a large scale. We are talking about things that are more than this: If everyone became unemployed or you did not need human workers for most things, then you may not need human workers to maintain your army, or very few humans. This can be easy for someone to make a coup, essentially. Or, if you have a large amount of talents in the data center, it will make you a very powerful person. If you use it to produce military hardware, it is possible that we can achieve concentration of power, and you may no longer have a democratic state.