# Jinja Template in `tokenizer`



## Base

åœ¨`transformers`ä¸­çš„`tokenizer`å¾€å¾€ä¼šæœ‰ï¼Œä»¥`Qwen2.5-7B-Instruct`ä¸ºä¾‹ï¼Œå…¶`tokenizer_config.json`ä¸­æœ‰å­˜åœ¨`chat_template`çš„é”®ï¼Œå…¶å€¼å¦‚ä¸‹ï¼š

```plaintext
"chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
```

è¯¥æ¨¡ç‰ˆæ˜¯ç”¨äº�`tokenizer.apply_chat_template`å‡½æ•°ä¸­ï¼Œé€šè¿‡è°ƒç”¨è¯¥æ¨¡ç‰ˆï¼Œå°†å¤šè½®å¯¹è¯�çš„`messages`å�˜æˆ�å­—ç¬¦ä¸²ï¼Œå¦‚ä¸‹æ‰€ç¤ºã€‚

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("/data1/HF-Models/Qwen/Qwen2.5-7B-Instruct")

chat = [
  {"role": "user", "content": "Hello, how are you?"},
  {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
  {"role": "user", "content": "I'd like to show off how chat templating works!"},
]

tokenizer.apply_chat_template(chat, tokenize=False)
```
è¾“å‡ºå¦‚ä¸‹ï¼š
```plaintext
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello, how are you?<|im_end|>
<|im_start|>assistant
I'm doing great. How can I help you today?<|im_end|>
<|im_start|>user
I'd like to show off how chat templating works!<|im_end|>
```
å¦‚æ�œè®¾ç½®ï¼Œ`add_generation_prompt=True`ï¼Œåˆ™æ‰“å�°ç»“æ�œå¦‚ä¸‹ï¼š

```plaintext
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello, how are you?<|im_end|>
<|im_start|>assistant
I'm doing great. How can I help you today?<|im_end|>
<|im_start|>user
I'd like to show off how chat templating works!<|im_end|>
<|im_start|>assistant
```

## Advanced: Adding and editing chat templates

å¾ˆç®€å�•ï¼Œå�ªéœ€ç¼–å†™ä¸€ä¸ª jinja æ¨¡æ�¿å¹¶è®¾ç½® tokenizer.chat_templateã€‚æ‚¨å�¯èƒ½ä¼šå�‘ç�°ï¼Œä»�å�¦ä¸€ä¸ªæ¨¡å�‹çš„ç�°æœ‰æ¨¡æ�¿å¼€å§‹ï¼Œç„¶å��æ ¹æ�®æ‚¨çš„éœ€è¦�ç®€å�•åœ°å¯¹å…¶è¿›è¡Œç¼–è¾‘ä¼šæ›´å®¹æ˜“ï¼�ä¾‹å¦‚ï¼Œæˆ‘ä»¬å�¯ä»¥é‡‡ç”¨ä¸Šé�¢çš„ LLaMA æ¨¡æ�¿ï¼Œå¹¶å°† â€œ[ASST]â€� å’Œ â€œ[/ASST]â€� æ·»åŠ åˆ°åŠ©æ‰‹æ¶ˆæ�¯ä¸­ï¼š

```python
template = tokenizer.chat_template
template = template.replace("SYS", "SYSTEM")  # Change the system token
tokenizer.chat_template = template  # Set the new template
tokenizer.push_to_hub("model_name")  # Upload your new template to the Hub!
```

# References

- HF-Chat Templates: https://huggingface.co/docs/transformers/chat_templating#advanced-extra-inputs-to-chat-templates
- Jinja: https://jinja.palletsprojects.com/en/stable/templates/

Jinja Template in tokenizer

# Writing Perfect Papers


æ��è‘†æ˜¥æ•™æ�ˆçš„åˆ†äº«ï¼Œå�—ç›Šè‰¯å¤šï¼Œæ„Ÿè°¢ã€‚æœ¬æ–‡ä¸»è¦�æ˜¯æ‘˜æŠ„æ��æ•™æ�ˆæ��ä¾›çš„ Slides ä½œä¸ºå��ç»­æŸ¥æ‰¾å¤�ä¹ èµ„æ–™ã€‚

Video: https://www.bilibili.com/video/BV18v411n7mr/

Slides: https://iqua.ece.toronto.edu/papers/writing-perfect-papers-2021.pdf



Our goal today: improve both the skills and the process.

# 1 Writing perfect papers

A top-down approach: from the big picture to tiny details.

## 1.1 The story

How do I make a great movie? A great movie starts from a powerful and convincing story.

**Three essential elements of a great story:**
Your paper is not really the best place to show your theoretical prowess. It's about advancing the state-of-the-art.
- **The problem**
    - Don't have to be something "**trendy**".
    - But it has to be something you really **enjoy** working on.
    - The "**trend**" may not last forever.
    - Following the trend may lead to more "**incremental**" results â€”â€” less exciting and less important.
    - Be a **contrarian** and working against the trend may be good idea.
- **What's new?**
    - In the context of what's already done in the related work, **what's new** in your work?
    - First, briefly introduce the **state-of-the-art** in the literature, solving the same problem you wish to solve.
    - Then write about what is **new** in you work that is different from previous work.
    - If the problem is **different** from previous work â€”â€” different assumptions, for example â€”â€” provide justifications **why** this is **important** and **necessary**.
- **How is it better?**
    - Now that your ideas are new, they should also lead to **bettter** results than before.
    - Show **how** your solution is **better** than previous work: such as stronger theoretical properties or better experimental results.

## 1.2 The work

Two things that you must do: **Read** and **Write**.

Read:
- Goal: become an expert on the problem.
- But how?
    - Start from one paper.
    - Perhaps a highly cited **seminal** paper.
    - Then do an **expanded-ring** search.
        - Papers that cited this paper.
        - Papers that it cited.
        - Papers authored by the same researchers.
    - Try not to depend only on a search engine.
    - Read **quickly** first.
    - Read more carefully if necessary **later**.

Write:
- Write about what you understood.
- Write every day.
- **idea -> write paper -> do research**
- If I don't have an idea what do I write about?
    - You write **anyway**.
    - Write about related work.
    - Write about what's challenging.
    - Write about why the problem is important.
    - Write about what needs to be fixed.
    - Write a survey paper.
    - Submit your survey paper for publication.
    - Write about **what's new** in the context of related work.
- Creating ideas interacts with **writing** closely.
- **Writing** is the best way to force yourself to **think clearly** and be **focused** and to **crystallize** what you don't quite understand yet.
- **Writing** also opens a dialogue.
- For others to read what you wrote.
- To stimulate discussions with others.
- That's why **writing** is a **slow** and **painful** process.
- Write in a **crystal clear** way.
- Write about **one** problem and **one** (potential) solution.
- Write with a **flow of ideas** that's easy to follow.
- Keep your readers engaged **throughout** the paper.
- Write your paper so that it's as easy to understand as **absolutely possible**.
- Your readers don't have to work hard.
- Write your paper **slowly**, so that your readers can read **quickly**.

## 1.3 The art

Pay attention to the title, abstract, introduction, and flow of ideas.

Tital: **2** lines, **10,000** readers.
Abstract: **10** lines, **1,000** readers.
Introduction: **100** lines, **100** readers.
The rest of the paper: **1,000** lines, **10** readers.

### The title

What's good title?

It attracts a reader to read the abstract.

It reflects the essence of the new idea.

It is as simple as possible
- does not have to be a precise summary of the paper.
- does not need to include all the keywords.

The best title is one that is the easiest to understand **at a glance**.

:::note{title="Example"}

Let's say we have a new idea that uses a new coding technique to reduce the latency when delivering data over the Internet. Which title is the best?

- LLRC: A Low-Latency Rate-Controlled System for Fast Data Delivery over the Internet
- Coded Information Distribution: New Content-Delivery Protocols for the Internet
- **Coding Reducces Latency over the Internet**

:::

### The abstract

The abstraction is a very important tool to attract readers to read the introduction.

Again, it conveys essential information about the paper.

It should be concise, and does not need to be long.

Remember the three essential elements: The problem, What's new? and How is it better?

A typical structure

- One sentence to state the background.
- One sentence to state **what the proble is**.
- Two to four sentences to state the original contribution in the paper â€”â€” **what's new?**
- One sentence to state **how the solution is better**, validated using analyses, simulations or experiments.

:::note{title="Abstract Example"}

(**Background**) With cloud storage services, it is commonplace for data to be not only stored in the cloud, but also shared across multiple users. (**Problem**) However, public auditing for such shared data -- while preserving both data and identity privacy -- remains to be an open challenge. (**Contribution: Overview**) In this paper, we propose the first privacy-preserving mechanism that allows public auditing on shared data stored in the cloud. (**Contribution: Highlights**) In particular, we exploit ring signatures to compute the metadata needed to verify the integrity of shared data. With our mechanism, the identity of the signer of each data block is kept private from a third party auditor (TPA), who is still able to publicly verify the integrity of shared data without retrieving the entire file. (**Contribution: Additional features**) In addition, we utilize homomorphic MACs and homomorphic hash function to mitigate the overhead of storing our signatures. (**Validation by experiments**) Our experimental results demonstrate the effectiveness and efficiency of our proposed mechanism when auditing shared data.

:::

### The introduction

If your title and abstract can get a reader to read the introduction, you are half way there.

The remaining job: impress the reader with an exciting and clear introduction.

The introduction is so important that I rewrote the introduction in many of papers.

It is important because we wish our reader to understand our ideas, even if they only read the introduction and noting else.

**Write a good introduction**
- It should be self-contained, so that a reader short on time doesn't need to read the rest of the paper.
- It should be clear **what the problem is**.
- It should be easy to understand **why our solution is new**, and **how it's better** than previous work.
- Don't make it too long.
- Don't spend too much space on the background and related work.
    - There can be a separate "Related Work" section.
- Don't make your work sound more original than it really is.

**Introduciont: a typical structure**
- Frist (opening) paragraph:
    - A **general overview** of the research field -- basic facts needed to "warm up" the reader and to prepare for the problem statement.
    - Not too long -- 2-3 sentences are good enough.
- The second paragraph: **problem statement**
    - A clear description of the challenges to be addressed and the **problems** to be solved.
    - Appeal to the intuition of the reader.
    - Describe at a high level.
    - It is fine (and even a good idea) to include an **intuitive solution** to the problem.

Unnecessary details inhibit the **natural** and **brisk** flow of the main ideas.

Returning to the typical structure of the introduction:
- General overview to warm-up the reader (the opening paragraph)
- State the problem (challenge) and existing solutions (1-2 paragraphs)
- Limitations of existing solutions that motivate this paper (2-3 paragraphs)
- Proposed solution: main idea (1-2 paragraphs)

**Write about what's new**
- Why is the proposed solution different from and better than existing solutions?
- State 1-2 most impressive highlights, not all of them.
- Make the originality of the paper crystal clear and stand out.
- You may use the sentence: "Highlights of our original ocntributions in this paper are as follows. First, ... Second, ... Finally, ..."

The list of original contributions drives the **entire paper** -- the rest of the paper substantiates the claims you have made.

Your readers may think: "Hey, if they can really deliver this, that'll be very exciting! I'd better read the rest of the paper."

It is a good idea to include a **table** to compare important properties of the proposed solution with its "direct competitors" in the existing literature, highlighting your **advantages**.

Use an example:
- It's an even better idea to show an intuitive **example**.
- Your **example** shows how your **main idea** works in a **special case**.
- Continue with **more** examples throughout the paper.
- After each theorem is proved -- or each algorithm shown -- explain their **implications** or **intuition** with an example.
- Your examples need to be really **simple**.
- They helps readers to understand your solution well.
- It'll be a greate idea to include a well-designed **figure** to illustrate your example.
- Includes as many figures as you can throughout the paper.

The **flow of ideas:**
- Just like the storyboard when making a movie, the **flow of ideas** needs to be carefully **designed**.
- Make sure that each paragraph contains one **complete** idea - **split** a paragraph that contains more than one idea, and merge short paragraphs that cover the same idea.
- Where is the section on **related work**?
    - Some prefer to place it after the introduction
        - Rationale: the section can be used to "warm up" the reader.
    - Some others prefer to place it before the conclusion
        - Rationale: After the introduction, the readers don't understand your main ideas yet, there's no point in talking about their differences from related work.
    - It depends on the design of your flow of ideas.
- Where do you write about **experimental results**?
    - Most papers collect all the experimental results and put them **at the end**.
    - But you don't have to.
    - Some **negative** or **preliminary** experimental results may be important for motivating the main idea in the paper.
    - Your flow of ideas can then be: "**initial results - idea to improve - more results to show better performance**"
    - You can even interleave experimental results with descriptions of your idea, if this provides the best flow.
- How do you allocate space across sections?
    - The short answer is: no one knows it better than you, because it depends on the flow of ideas.

**General rules of thumb:**
- Don't write a very long abstract (200 words), introduction (one page), related work (half a page), or concluding remarks (1-2 paragraphs)
- Throughout the paper, make it self-contained, but don't use a lot of space for unnecessary background.
- Keep the motivation short and concise.
## 1.4 The details

**Use a git repository to manage your workflow.**

Get repositories help you to track all histories and collaborate with others.

Place everything - source code, reference - into your git repository.

Use any of the public cloud services such as GitHub.

**Eliminate typographical and grammatical mistakes.**

First, eliminate all spelling mistakes by running your paper through a spell checker.

Use a UNIX command-line tool such as spell to check spelling - not Microsoft Word.

Then fix grammar and usage problems by proofreading.

**Proofreading** also helps you to fix the remaining spelling mistakes that a spell checker cannot catch.

You cannot rely on someone else - certainly not the reviewers - to proofread for you, it's your paper!

**Use transitional words, phrases, and sentences**

Use **transitions** across the boundary of sentences, paragraphs and sections.

**Important** for readers to follow your **flow of ideas**.

If you don't add transitons, readers will need to add them subocnsciously in their mind, anyway. You are asking readers to do the hard work!


:::note{title="Examples of transitions"}
Connecting two halves of a sentence:
- as, since ,or else

Connecting sentences:
- However, In addition, Further, Nevertheless, Fortunately, Unfortunately, Suprisingly,
- To make matters worse, to further exacerbate the problem, The bad new is,
- The implications are two-fold, It turns out that, As as example, To take ... a step further,

Connecting paragraphs and sections:
- It only remains to see ...
- The simple answer to this question is,
- The only challenge that remains now is,
- To address this challenge,
- We first present...
- Next, we evaluate...
- We are now ready to...

:::

**Use correct English**

- Keep punctuations marks inside the closing quotation mark.
- Don't use long sentences
    - Don't use long sentences with more than one comma in the middle of the sentence - and **abuse** "where", "in which", "whose", 'so that", "such that".
- Don't be too colloquial
    - Don't use words that are too informal and colloquial.
    - "a lot of" is more colloquial than "a large number of".
    - Instead of "big" use "substantial" or "large".
- Don't be too formal either
    - Overly formal words and phrases sometimes feel awkward.
    - "we endeavour to ascertain that..." - "We show that..."
    - "It can be ascribed to..." - "It is due to..."
    - "The overwhelming quantity of..." - "The exceedingly large number of..."
- Don't use words that are too emotional
- Countable vs. uncountable nouns
    - If a word is countable, when using its plural form, remember to use "a few", "a number of", "fewer". Instead of "less bits," use "fewer bits".
    - If not, do not invent its plural form (such as "performances," "advices," "equipments," "informations"),
    - and use "less" or "lower" rather than "fewer"
- Agreement of the verb with the subject.
- Watch out on common words and phrases.
    - It almost takes a lifetime to master the usage of common words, such as "with", "of", "for", "at", "against", "in", "on"
    - Or to master the large number of phrases in English "in other words" or "in another word"?
- Articles as determiners
    - The articles "a" / "an" (the indefinite article) and "the" (the definite article) are frequently used in correctly by new students
    - Plural nouns are typically used without an article:
        - "The source node receives acknowledgements" (not "the acknowledgments")
    - The indefinite is weaker than the definite article:
        - "a large portion of" (not "the large portion of")
    - But there's no need for "the" in section titles - instead of "The System Model," just say "System Model."
    - Articles can be tricky to use, but there are too many of them in paper - pay attention!

Good news: most English problems are not hard to fix - just proofread every sentence with plenty of time!

**Typest your paper correctly and beautifully**

1. Use LaTex, no matter what.
2. Draw figures using a vector-based application: Image-based drawing applications produce images that become fuzzy when zoomed in on a reader's iPad.
3. Proofread your bibliography and make it consistent: Use BibTex to typeset the bibliography in your paper. After downloading BibTex entries from the web, proofread to get them consistent. Keep a consistent style of abbreviating journal and conference titles throughout the bibliography. 

Try your best to find out the venue where the paper was published in, rather than citing arxiv.org.

# 2 Writing good papers fast!

Writing good papers is hard. That's why you are advised to write slowly. In the good old days, academics can take years to write one research paper. We also have the "publish or perish" culture, where the number of papers is an indication of quality. I need to graduate - tell me how to write papers quickly.

http://faculty.hitsz.edu.cn/hubaotian

![æˆªå±�2024-10-19 10.02.46.png](/static/img/083df19c1637a02c1271. 5cfe4c8f96e32.%C3%A6%C2%88%C2%AA%C3%A5%C2%B1%C2%8F2024-10-19%2010.02.46.png)

**Know your deadline**

At all times, always work towards a paer deadline.

Deadlines are all equal: It doesn't matter what kind of deadline it is - 
- a conference deadline
- a deadline from a journal special issue
- a deadline you set up yourself for submitting a journal paper

1. You will have to make it no matter what happens.

Set your deadline realistically. Admit that you are not able to write a good paper in a month or two.

But a deadline six months later may not carry a lot of weight either - unless you are extremely disciplined.

2. Make it public knowledge

Tell as many peoples as you can when your deadline will be.

So that it feels like a public disgrace if you fail to meet the deadline. As a researcher, your sure wish to be seen as responsible and dependable.

You work around the clock to make sure that others who depend on you will have the peace of mind that the work will be done and done by the deadline.

3. Allocate your time well.

Design a rough timeline at the very beginning. Assign deadlines to each milestone in your timeline.

If necessary, ervise your timeline along the way. But think seriously why you missed your milestone deadlines. Don't procrastinate and delay all the work to the days before the deadline. It may be feasible, but it will be stressful and it degrades the quality of your work

4. Don't follow, lead!

You are the first author of the new paper, right? Then you are the leader, the boss, and the person who cares most about this work.

Start your team at the beginning of your project. Only invite trustworthy collaborators when building your team. Your collaborators include your advisors - get them to work for you. Get them to commit their time. Warn them of the tough work ahead along the way. Promise them an exciting outcome. Assign your collaborators to milestones - they are your "resources". Remind them about their milestone deadlines often. Persist if they forget about your requests. Cancel their "membership" in your team if they consistently fail. Have backup plans if they fail to deliver by the deadline.

**Choose a problem with the right size**

It must be feasible to be completed before the deadline.

Goal: make solid improvements over on paper, and one paper only.

Choose a paper that you enjoyed reading or even better, a paper with source code. If its source code is not on GitHub, ask the authors. If you don't have source code, estimate the amount of time to reproduce the work. If it may take a long time, give up and choose another paper.

Read the paper along with its source code. The source code contains secret ingredients in its "recipe". It helps you to understand the paper completely and deeply. And only with a complete and deep understanding can you create something new. And create something new quickly. To do it, you will have to a deadline in your timeline for creating something new.

A deadline forces you to think long and hard. Revise the source code and do a feasibility test for your idea. If its feasibility is promising, think about its theoretical properties - anything provably correct? Your little problem must fit into the scope of the conference you'll submit to.

Goal: get the paper accepted.

Many conferences have their favoured research topics.

Writing Perfect Papers

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2311.07911 |
> | Code & Data | https://github.com/google-research/google-research/tree/master/instruction_following_eval |
> | Public      | arXiv |
> | Date        | 2024.09.13 |


# Summary Overview

å¯¹äº� LLM çš„ Instruction Following èƒ½åŠ›çš„è¯„ä¼°æ²¡æœ‰æ ‡å‡†åŒ–ï¼šäººå·¥è¯„ä¼°æˆ�æœ¬é«˜ã€�é€Ÿåº¦æ…¢ï¼Œè€Œä¸”æ— æ³•å®¢è§‚åœ°é‡�å¤�ï¼Œè€ŒåŸºäº� model judge çš„è‡ªåŠ¨è¯„ä¼°åˆ™å�¯èƒ½å­˜åœ¨å��å·®æˆ–å�—åˆ° judge model èƒ½åŠ›çš„é™�åˆ¶ã€‚ä½œè€…ä¸ºæ­¤å¼•å…¥äº†æŒ‡ä»¤è·Ÿè¸ªè¯„ä¼°(Instruction-Following Eval, IFEval)ã€‚IFEval æ˜¯ä¸€ç§�ç®€å�•æ˜“è¡Œçš„è¯„ä¼°åŸºå‡†ï¼Œå®ƒä¾§é‡�äº�ä¸€ç»„â€œå�¯éªŒè¯�çš„æŒ‡ä»¤â€�ï¼Œä¾‹å¦‚â€œwrite in more than 400 wordsâ€�å’Œâ€œmetion the keyword of AI at least 3 timesâ€�ã€‚ä½œè€…ç¡®å®šäº†25ç§�å�¯éªŒè¯�çš„æŒ‡ä»¤ï¼Œå¹¶æ�„å»ºäº†çº¦500ä¸ª promptï¼Œæ¯�ä¸€ä¸ª prompt åŒ…å�«ä¸€ä¸ªæˆ–å¤šä¸ªå�¯éªŒè¯�æŒ‡ä»¤ã€‚

# Main Content

ç�°æœ‰çš„è¯„ä¼°æ–¹æ³•å�¯åˆ†ä¸º3å¤§ç±»ï¼Œæ¯�ä¸€ç±»éƒ½æœ‰å�„è‡ªçš„ç¼ºç‚¹ï¼›
1. **Human evaluation** is time consuming, expensive and relies on a set of humman annotators, leading to potential bias and inconsistencies for reproducibility.
2. **Model-based evaluation** involves using an internal or external model to assess the performance of the target model. However, this approach heavily rely on the correctness of the evaluator model, which is not guaranteed. If the evaluator model has significatn limitations, it yields misleading evaluation signals.
3. **Quantitative benchmarks** provide a standardized and scalabel evaluation approach.

æˆ‘ä»¬æ€»å…±åˆ›å»ºäº† 25 ä¸ªå�¯éªŒè¯�æŒ‡ä»¤åˆ—è¡¨ã€‚æˆ‘ä»¬è¿˜åˆ›å»ºäº†ä¸€ç»„ 541 ä¸ªæ��ç¤ºï¼Œæ¯�ä¸ªæ��ç¤ºåŒ…å�«ä¸€ä¸ªæˆ–å¤šä¸ªå�¯éªŒè¯�çš„æŒ‡ä»¤ã€‚

æ€»ä¹‹ï¼Œæˆ‘ä»¬æ��å‡ºäº† IFEvalï¼šInstruction-Following Evalï¼Œè¿™æ˜¯ä¸€ä¸ªä½¿ç”¨ä¸€ç»„åŒ…å�«å�¯éªŒè¯�æŒ‡ä»¤çš„æ��ç¤ºæ�¥è¯„ä¼° LLM é�µå¾ªæŒ‡ä»¤èƒ½åŠ›çš„åŸºå‡†ã€‚è¿™äº›å�¯éªŒè¯�çš„æŒ‡ä»¤æ˜¯å�Ÿå­�æŒ‡ä»¤ï¼Œäººä»¬å�¯ä»¥ç”¨ä¸€ä¸ªç®€å�•ã€�å�¯è§£é‡Šå’Œç¡®å®šæ€§çš„ç¨‹åº�æ�¥éªŒè¯�ç›¸åº”çš„å“�åº”æ˜¯å�¦é�µå¾ªäº†æŒ‡ä»¤ã€‚

![image.png](/static/img/e4d0bce01e86dcf5f2783d5ac65e9ba5.image.png)

![image.png](/static/img/005384b24cb4f672439d3ac89787ebae.image.png)

![image.png](/static/img/1573e3aacc129c48d03d56e92c251a38.image.png)

é€šè¿‡å››ä¸ªæ­¥éª¤æ�¥åˆ›å»ºæ��ç¤ºèƒ½å¤Ÿç¼“è§£ä¸€äº›é—®é¢˜ï¼š
1. ç”Ÿæˆ�ä¸€ç»„åŸºæœ¬ä½“å¼�ï¼Œå¹¶åœ¨æ¯�æ�¡æ��ç¤ºçš„æœ«å°¾é™„ä¸Šä¸€è‡³ä¸‰æ�¡éš�æœºé€‰æ‹©çš„å�¯éªŒè¯�æŒ‡ä»¤ï¼›
2. ä½¿ç”¨å°‘é‡�æ��ç¤ºæ�¥è¯†åˆ«ä¸�å�ˆé€»è¾‘çš„æ��ç¤ºï¼Œå¹¶å°†å…¶åˆ é™¤
3. é‡‡ç”¨å�¦ä¸€ç§�åŸºäº�å°‘é‡�æ��ç¤ºçš„æ–¹æ³•ï¼Œå¯¹æ¯�ä¸ªæ��ç¤ºè¿›è¡Œé‡�æ–°æ�ªè¾�ï¼Œä»¥å¢�åŠ  prompt çš„å¤šæ ·æ€§
4. å¯¹é‡�æ–°ç”Ÿæˆ�çš„ prompt é€�ä¸€è¿›è¡Œäººå·¥æ£€æŸ¥å’Œç¼–è¾‘

**strict metric**

$$\text{is followed}(resp,inst)=\begin{cases}\text{True},&\text{if instruction is followed.}\\\text{False},&\text{otherwise.}\end{cases}$$

**loose metric**

$$\text{is followed}_{{\mathrm{loose}}}(resp,inst)=\mathrm{Any}{\left(\mathrm{is_followed}(\mathrm{transform}_{t}(resp),inst\right)}\mathrm{~for~}t=1,2,\cdots$$

å…¶ä¸­ï¼Œ$\mathrm{transform}_t(resp)$æ˜¯ç¬¬$t$ä¸ªè½¬æ�¢å��çš„å“�åº”ã€‚ä½¿ç”¨ä»¥ä¸‹è½¬æ�¢å‡½æ•°å¯¹æ¯�ä¸ªå“�åº”è¿›è¡Œè½¬æ�¢ï¼š
1. åˆ é™¤ markdown è¯­æ³•ä¸­å¸¸è§�çš„å­—ä½“ä¿®é¥°ç¬¦ï¼Œå°¤å…¶æ˜¯ "*"å’Œ "**"ã€‚
2. å�»æ�‰å›�å¤�çš„ç¬¬ä¸€è¡Œï¼Œè¿™æ ·æˆ‘ä»¬å°±å�¯ä»¥è·³è¿‡ "sure, here it is:"è¿™æ ·çš„å¼€åœºç™½ã€‚
3. åˆ é™¤å›�å¤�çš„æœ€å��ä¸€è¡Œï¼Œè¿™æ ·æˆ‘ä»¬å°±å�¯ä»¥è·³è¿‡ "Hope it helps."è¿™æ ·çš„ç»“æ�Ÿè¯­ã€‚

![image.png](/static/img/7555555957519e05f30c9ee8d7717a00.image.png)

![image.png](/static/img/28a88d4f10ed1526d0f2a6448dd5a90f.image.png)

# ğŸ¤–ChatGPT

:::info{title="ChapGPT"}
1. è®ºæ–‡çš„åˆ›æ–°ä¹‹å¤„ä¸�ç‹¬ç‰¹æ€§ï¼š
   - **åŸºäº�å�¯éªŒè¯�æŒ‡ä»¤çš„è¯„ä¼°æ–¹æ³•**ï¼šè®ºæ–‡æ��å‡ºäº†ä¸€ä¸ªå��ä¸ºIFEvalçš„è¯„ä¼°æ–¹æ³•ï¼Œä¸“é—¨ç”¨äº�è¯„ä¼°å¤§å�‹è¯­è¨€æ¨¡å�‹(LLMs)çš„æŒ‡ä»¤è·Ÿéš�èƒ½åŠ›ã€‚è¿™ç§�è¯„ä¼°æ–¹æ³•ä½¿ç”¨äº†25ç§�å�¯éªŒè¯�çš„æŒ‡ä»¤ï¼ˆå¦‚å­—æ•°é™�åˆ¶ã€�ç‰¹å®šå…³é”®å­—ä½¿ç”¨ç­‰ï¼‰ï¼Œå¹¶ä¸”è®¾è®¡äº†çº¦500ä¸ªæ��ç¤ºç¬¦ï¼Œé€šè¿‡ç¨‹åº�åŒ–éªŒè¯�æ¨¡å�‹çš„å“�åº”æ˜¯å�¦ç¬¦å�ˆè¿™äº›æŒ‡ä»¤ã€‚è¿™ä½¿å¾—è¯„ä¼°è¿‡ç¨‹è‡ªåŠ¨åŒ–ã€�å�¯é‡�å¤�ä¸”å®¢è§‚ï¼Œé�¿å…�äº†äººç±»è¯„ä¼°ä¸­çš„ä¸»è§‚æ€§å’Œæˆ�æœ¬é—®é¢˜ã€‚
   - **ç®€åŒ–å¤�æ�‚è¯„ä¼°çš„æ¨¡å�‹**ï¼šIFEvalé€šè¿‡è®¾è®¡æ˜�ç¡®çš„è§„åˆ™å’Œâ€œå�¯éªŒè¯�æŒ‡ä»¤â€�è§£å†³äº†ä»¥å¾€è‡ªç„¶è¯­è¨€å¤„ç�†æ¨¡å�‹è¯„ä¼°ä¸­ç”±äº�è¯­è¨€æ¨¡ç³Šæ€§å’Œå¤šä¹‰æ€§å¯¼è‡´çš„å¤�æ�‚æ€§ï¼Œä½¿å¾—è¯„ä¼°æ ‡å‡†æ›´åŠ ç»Ÿä¸€å’Œæ¸…æ™°ã€‚
   - **å®½æ�¾å’Œä¸¥æ ¼çš„å�Œé‡�è¯„ä¼°æ ‡å‡†**ï¼šä¸ºäº†è§£å†³å�•ä¸€è¯„ä¼°æ ‡å‡†çš„å±€é™�æ€§ï¼Œè®ºæ–‡è®¾è®¡äº†â€œä¸¥æ ¼â€�å’Œâ€œå®½æ�¾â€�ä¸¤ç§�æŒ‡æ ‡ï¼Œå‰�è€…åŸºäº�å®Œå…¨åŒ¹é…�ï¼Œå��è€…å…�è®¸ä¸€å®šçš„å�˜é€šï¼ˆå¦‚å¿½ç•¥markdownæ ‡ç­¾ç­‰ï¼‰ï¼Œè¿™æ��å�‡äº†è¯„ä¼°çš„ç�µæ´»æ€§å’Œç²¾åº¦ã€‚

2. è®ºæ–‡ä¸­å­˜åœ¨çš„é—®é¢˜å�Šæ”¹è¿›å»ºè®®ï¼š
   - **æŒ‡ä»¤ç±»å�‹çš„å±€é™�æ€§**ï¼šå°½ç®¡è®ºæ–‡æ��å‡ºäº†25ç§�å�¯éªŒè¯�æŒ‡ä»¤ï¼Œä½†è¿™äº›æŒ‡ä»¤å¤§å¤šå±€é™�äº�æ ¼å¼�ã€�å­—æ•°ã€�è¯­è¨€ç­‰æµ…å±‚é�¢ï¼Œç¼ºä¹�å¯¹å¤�æ�‚ä»»åŠ¡æˆ–å¤šæ¨¡æ€�ä»»åŠ¡ï¼ˆå¦‚å›¾åƒ�ç”Ÿæˆ�ã€�è§†é¢‘å¤„ç�†ç­‰ï¼‰çš„æ‰©å±•ã€‚æ”¹è¿›å»ºè®®æ˜¯è¿›ä¸€æ­¥æ‰©å±•æŒ‡ä»¤ç±»å�‹ï¼Œä½¿å…¶æ¶µç›–æ›´å¤šå¤�æ�‚çš„å®�é™…åº”ç”¨åœºæ™¯ã€‚
   - **æ¨¡å�‹å¯¹è¯„ä¼°ç»“æ�œçš„ä¾�èµ–**ï¼šç›®å‰�çš„è¯„ä¼°ç»“æ�œä¾�èµ–äº�å½“å‰�å�¯ç”¨çš„ä¸¤ä¸ªLLMï¼ˆGPT-4å’ŒPaLM 2ï¼‰ï¼Œç„¶è€Œæ¨¡å�‹é—´çš„å¯¹æ¯”å¹¶ä¸�å…·å¤‡å�¯æ�¨å¹¿æ€§ï¼Œå› ä¸ºè¿™äº›æ¨¡å�‹çš„å�‚æ•°å·®å¼‚è¾ƒå¤§ï¼Œæ— æ³•è¿›è¡Œç›´æ�¥æ¯”è¾ƒã€‚å»ºè®®æœªæ�¥å¢�åŠ æ›´å¤šæ¨¡å�‹çš„æµ‹è¯•ï¼Œå¹¶åœ¨ä¸�å�Œå¤§å°�ã€�æ�¶æ�„çš„æ¨¡å�‹ä¸Šè¿›è¡Œå¹¿æ³›è¯„ä¼°ã€‚
   - **æŒ‡ä»¤å†²çª�é—®é¢˜**ï¼šè®ºæ–‡æ��åˆ°å¤šä¸ªæŒ‡ä»¤ç»„å�ˆæ—¶å�¯èƒ½ä¼šäº§ç”Ÿå†²çª�ï¼Œå°½ç®¡å·²ç»�é€šè¿‡few-shotæ–¹æ³•å‡�å°‘äº†è¿™ç§�æƒ…å†µï¼Œä½†åœ¨æœªæ�¥çš„å·¥ä½œä¸­ï¼Œå»ºè®®æ›´ç³»ç»Ÿåœ°å¤„ç�†æŒ‡ä»¤å†²çª�ï¼Œå�¯èƒ½é€šè¿‡ä¼˜åŒ–æŒ‡ä»¤ç”Ÿæˆ�é€»è¾‘æˆ–å¼•å…¥å†²çª�æ£€æµ‹æœºåˆ¶æ�¥è§£å†³è¿™ä¸€é—®é¢˜ã€‚

3. åŸºäº�è®ºæ–‡çš„å†…å®¹å’Œç ”ç©¶ç»“æ�œï¼Œæ��å‡ºçš„åˆ›æ–°ç‚¹æˆ–ç ”ç©¶è·¯å¾„ï¼š
   - **å¼•å…¥å¤šæ¨¡æ€�æŒ‡ä»¤è¯„ä¼°**ï¼šå½“å‰�çš„IFEvalä»…é™�äº�æ–‡æœ¬ç”Ÿæˆ�ä»»åŠ¡ï¼Œå�¯ä»¥æ‰©å±•åˆ°å¤šæ¨¡æ€�ä»»åŠ¡ï¼Œå¦‚å›¾åƒ�ç”Ÿæˆ�ã€�è§†é¢‘æ��è¿°ç­‰ã€‚æœªæ�¥çš„ç ”ç©¶å�¯ä»¥å¼€å�‘åŸºäº�å¤šæ¨¡æ€�æ¨¡å�‹çš„æŒ‡ä»¤è·Ÿéš�è¯„ä¼°æ–¹æ³•ã€‚
   - **é�¢å�‘å¤�æ�‚ä»»åŠ¡çš„åŠ¨æ€�è¯„ä¼°**ï¼šåŸºäº�å›ºå®šæŒ‡ä»¤çš„è¯„ä¼°æ–¹æ³•éš¾ä»¥è¦†ç›–è¯¸å¦‚é€»è¾‘æ�¨ç�†ã€�ç¼–ç¨‹ç­‰å¤�æ�‚ä»»åŠ¡ã€‚å�¯ä»¥å¼€å�‘ä¸€ç§�åŠ¨æ€�ç”Ÿæˆ�æŒ‡ä»¤çš„è¯„ä¼°æ¡†æ�¶ï¼Œé€‚åº”æ›´å¤�æ�‚å’Œå¼€æ”¾çš„é—®é¢˜åœºæ™¯ã€‚
   - **åŸºäº�äººç±»äº’åŠ¨çš„å®�æ—¶è¯„ä¼°**ï¼šå°†IFEvalæ‰©å±•è‡³ä¸�äººç±»ç”¨æˆ·çš„äº’åŠ¨è¯„ä¼°ï¼Œè®¾è®¡å�¯æ¨¡æ‹ŸçœŸå®�åº”ç”¨åœºæ™¯çš„äº¤äº’æŒ‡ä»¤ï¼ˆå¦‚å®�æ—¶é—®ç­”ç³»ç»Ÿã€�å¯¹è¯�ç³»ç»Ÿçš„æŒ‡ä»¤è·Ÿéš�èƒ½åŠ›è¯„ä¼°ï¼‰ã€‚

4. ä¸ºæ–°çš„ç ”ç©¶è·¯å¾„åˆ¶å®šçš„ç ”ç©¶æ–¹æ¡ˆï¼š
   - **å¤šæ¨¡æ€�æŒ‡ä»¤è¯„ä¼°ç ”ç©¶æ–¹æ¡ˆ**ï¼š
     - ç ”ç©¶æ–¹æ³•ï¼šæ‰©å±•å½“å‰�çš„IFEvalï¼Œè®¾è®¡å¤šæ¨¡æ€�å�¯éªŒè¯�æŒ‡ä»¤ï¼Œä¾‹å¦‚â€œç”Ÿæˆ�ä¸€å¼ åŒ…å�«ç‰¹å®šå…ƒç´ çš„å›¾ç‰‡â€�æˆ–â€œæ��è¿°è§†é¢‘ä¸­çš„å…·ä½“åœºæ™¯â€�ç­‰ã€‚
     - æ­¥éª¤ï¼š
       1. æ‰©å±•æŒ‡ä»¤åº“ï¼ŒåŒ…å�«è§†è§‰ã€�éŸ³é¢‘ã€�æ–‡æœ¬ç­‰å¤šç§�æ¨¡æ€�çš„æŒ‡ä»¤ã€‚
       2. é‡‡é›†å¤šæ¨¡æ€�ä»»åŠ¡çš„æ•°æ�®é›†ï¼ŒåŒ…å�«è§†é¢‘ã€�å›¾åƒ�ã€�éŸ³é¢‘ç­‰å¤šæ ·åŒ–è¾“å…¥ã€‚
       3. è®¾è®¡è‡ªåŠ¨åŒ–éªŒè¯�ç¨‹åº�ï¼Œé€šè¿‡å›¾åƒ�è¯†åˆ«ã€�è§†é¢‘åˆ†æ��ç­‰æŠ€æœ¯éªŒè¯�æ¨¡å�‹æ˜¯å�¦é�µå¾ªäº†å¤šæ¨¡æ€�æŒ‡ä»¤ã€‚
       4. åœ¨å¤šä¸ªå¤šæ¨¡æ€�LLMï¼ˆå¦‚GPT-4ã€�CLIPã€�DALLÂ·Eç­‰ï¼‰ä¸Šè¿›è¡Œæµ‹è¯•ã€‚
     - æœŸæœ›æˆ�æ�œï¼šå¼€å�‘å‡ºé€‚ç”¨äº�å¤šæ¨¡æ€�ä»»åŠ¡çš„æŒ‡ä»¤è·Ÿéš�è¯„ä¼°æ ‡å‡†ï¼Œå¹¶ä¸”èƒ½å¤Ÿä¸ºæœªæ�¥çš„å¤šæ¨¡æ€�ç”Ÿæˆ�ä»»åŠ¡æ��ä¾›æ›´å…¨é�¢çš„è¯„ä¼°æ¡†æ�¶ã€‚

   - **åŠ¨æ€�æŒ‡ä»¤ç”Ÿæˆ�çš„å¤�æ�‚ä»»åŠ¡è¯„ä¼°æ–¹æ¡ˆ**ï¼š
     - ç ”ç©¶æ–¹æ³•ï¼šè®¾è®¡ä¸€ç§�èƒ½å¤ŸåŠ¨æ€�ç”Ÿæˆ�æŒ‡ä»¤çš„è¯„ä¼°æ¡†æ�¶ï¼Œé€‚åº”å¤�æ�‚ä»»åŠ¡åœºæ™¯ï¼Œå¦‚ç¼–ç¨‹ã€�é€»è¾‘æ�¨ç�†ç­‰ã€‚æ¯�æ¬¡ç”Ÿæˆ�çš„æŒ‡ä»¤ä¸�ä»…èƒ½éªŒè¯�ç»“æ�œï¼Œè¿˜èƒ½è¯„ä¼°æ¨¡å�‹çš„è¿‡ç¨‹å’Œæ�¨ç�†é€»è¾‘ã€‚
     - æ­¥éª¤ï¼š
       1. åˆ›å»ºåŠ¨æ€�æŒ‡ä»¤ç”Ÿæˆ�å™¨ï¼ŒåŸºäº�ä»»åŠ¡å¤�æ�‚åº¦å’Œåœºæ™¯è‡ªåŠ¨ç”Ÿæˆ�å¤šæ­¥éª¤æŒ‡ä»¤ã€‚
       2. è®¾è®¡ä»»åŠ¡åº“ï¼Œæ¶µç›–é€»è¾‘æ�¨ç�†ã€�ç¼–ç¨‹ã€�å¤�æ�‚å†³ç­–ç­‰å¤šç§�ä»»åŠ¡ç±»å�‹ã€‚
       3. é€šè¿‡å¤šå±‚æ¬¡çš„éªŒè¯�æ ‡å‡†ï¼Œè¯„ä¼°æ¨¡å�‹æ˜¯å�¦æ­£ç¡®é�µå¾ªäº†å¤�æ�‚æŒ‡ä»¤ï¼ŒåŒ…æ‹¬è¿‡ç¨‹éªŒè¯�ï¼ˆå¦‚æ­¥éª¤é¡ºåº�ï¼‰å’Œç»“æ�œéªŒè¯�ã€‚
       4. ä½¿ç”¨ä¸�å�Œå¤�æ�‚åº¦çš„LLMè¿›è¡Œæµ‹è¯•ï¼Œåˆ†æ��æ¨¡å�‹åœ¨å¤�æ�‚ä»»åŠ¡ä¸­çš„è¡¨ç�°å·®å¼‚ã€‚
     - æœŸæœ›æˆ�æ�œï¼šæ�„å»ºä¸€ä¸ªèƒ½å¤Ÿç”Ÿæˆ�å¤šæ­¥å¤�æ�‚æŒ‡ä»¤çš„è¯„ä¼°æ¡†æ�¶ï¼Œæ��å�‡å¯¹å¤�æ�‚ä»»åŠ¡åœºæ™¯ä¸­LLMæ€§èƒ½çš„è¯„ä¼°èƒ½åŠ›ã€‚

   - **åŸºäº�äººæœºäº¤äº’çš„å®�æ—¶è¯„ä¼°æ–¹æ¡ˆ**ï¼š
     - ç ”ç©¶æ–¹æ³•ï¼šè®¾è®¡ä¸€ä¸ªå®�æ—¶äº¤äº’ç³»ç»Ÿï¼Œè¯„ä¼°LLMåœ¨ä¸�äººç±»äº’åŠ¨è¿‡ç¨‹ä¸­é�µå¾ªæŒ‡ä»¤çš„èƒ½åŠ›ã€‚é€šè¿‡æ¨¡æ‹Ÿç”¨æˆ·çš„çœŸå®�éœ€æ±‚åœºæ™¯ï¼Œè¯„ä¼°LLMå¯¹æŒ‡ä»¤çš„å��åº”å’Œæ‰§è¡Œæƒ…å†µã€‚
     - æ­¥éª¤ï¼š
       1. è®¾è®¡ä¸€ä¸ªæ¨¡æ‹ŸçœŸå®�äº¤äº’åœºæ™¯çš„æ•°æ�®é›†ï¼ŒåŒ…æ‹¬å®�æ—¶é—®ç­”ã€�å¯¹è¯�ç³»ç»Ÿç­‰åº”ç”¨åœºæ™¯ã€‚
       2. å¼€å�‘å®�æ—¶å��é¦ˆå’Œè¯„åˆ†ç³»ç»Ÿï¼Œç”¨æˆ·åœ¨äº¤äº’ä¸­å�¯ä»¥ç»™å‡ºå�³æ—¶å��é¦ˆï¼Œè¯„ä¼°æ¨¡å�‹çš„æŒ‡ä»¤æ‰§è¡Œæƒ…å†µã€‚
       3. é€šè¿‡å®šé‡�å’Œå®šæ€§åˆ†æ��è¯„ä¼°LLMåœ¨ä¸�å�Œäº¤äº’åœºæ™¯ä¸­çš„è¡¨ç�°ã€‚
       4. å¯¹æ¯”ä¸�å�Œæ¨¡å�‹åœ¨å®�æ—¶äº¤äº’ä¸­çš„è¡¨ç�°ï¼Œç¡®å®šå…¶æŒ‡ä»¤è·Ÿéš�èƒ½åŠ›çš„å·®å¼‚ã€‚
     - æœŸæœ›æˆ�æ�œï¼šå¼€å�‘å‡ºä¸€ç§�èƒ½å¤Ÿåœ¨å®�æ—¶äº¤äº’ä¸­è¯„ä¼°æ¨¡å�‹æŒ‡ä»¤è·Ÿéš�èƒ½åŠ›çš„ç³»ç»Ÿï¼Œé€‚ç”¨äº�å¯¹è¯�æœºå™¨äººã€�è™šæ‹ŸåŠ©æ‰‹ç­‰åº”ç”¨åœºæ™¯çš„æ¨¡å�‹è¯„ä¼°ã€‚

:::

# Others

[arXiv-2023] Instruction-Following Evaluation for Large Language Models

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2404.11018 |
> | Code & Data | / |
> | Public      | arXiv |
> | Date        | 2024.09.11 |


# Summary Overview

æœ€æ–°æ‰©å±•çš„ä¸Šä¸‹æ–‡çª—å�£ä½¿æˆ‘ä»¬èƒ½å¤Ÿä½¿ç”¨æ•°ç™¾æˆ–æ•°å�ƒä¸ªç¤ºä¾‹æ�¥ç ”ç©¶ ICL ï¼Œè¿™å°±æ˜¯ many-shot æœºåˆ¶ã€‚ä»� few-shot åˆ° many-shotï¼Œè§‚å¯Ÿåˆ°äº† LLM åœ¨å�„ç§�ç”Ÿæˆ�ä»»åŠ¡ä¸Šçš„æ˜¾è‘—å¢�åŠ ã€‚ä½†æ˜¯å�¯ç”¨çš„æ‰‹å·¥ç”Ÿæˆ�çš„ç¤ºä¾‹çš„ç¼ºä¹�é™�åˆ¶äº† many-shotï¼Œä¸ºäº†å‡�å°‘è¿™ç§�é™�åˆ¶ï¼Œä½œè€…è®¾è®¡äº†ä¸¤ç±»æ–¹æ³•ï¼š1. Reinforced ICL ä½¿ç”¨æ¨¡å�‹ç”Ÿæˆ�çš„ç¤ºä¾‹è¿›è¡Œæ›¿ä»£ã€‚2. Unsupervised ICL å®Œå…¨åˆ é™¤äº†åˆ©ç”¨ä»…ä¿�ç•™äº†é—®é¢˜ã€‚ä¸¤ç±»æ–¹æ³•éƒ½å¯¹äº� many-shot æœ‰ç§¯æ��çš„å½±å“�ã€‚

# Main Content

Many-shot learning - ä½¿ç”¨å¤§é‡�çš„ç¤ºä¾‹è¿›è¡Œ ICLï¼Œå�¯ä»¥æ›´å¥½åœ°æŒ‡å®šä»»åŠ¡ï¼Œå‡�å°‘å¯¹å¾®è°ƒçš„éœ€æ±‚ï¼Œå¹¶å�¯èƒ½ä½¿ LLM æ›´å…·é€šç”¨æ€§å’Œé€‚åº”æ€§ã€‚

ä½¿ç”¨æˆ�ç™¾ä¸Šå�ƒçš„ shot å­¦ä¹ æ•ˆæ�œæ˜�æ˜¾ä¼˜äº�ä½¿ç”¨å°‘é‡� shot çš„ICL (Figure 1)ã€‚

![image.png](/static/img/a391290bb2465d82920a80aed8c4b024.image.png)

é€šå¸¸å�ªæœ‰å½“ shot æ•°é‡�è¾¾åˆ°æ•°å��ä¸‡ä¸ª token æ—¶ï¼Œæ‰�èƒ½å®�ç�°æœ€é«˜æ€§èƒ½ (Figure 2)ã€‚æˆ‘ä»¬å�‘ç�°ï¼Œåœ¨ MATHã€�GPQA å’Œ Big-Bench Hard ç­‰é—®é¢˜è§£å†³ä»»åŠ¡ä¸­ï¼Œreinforced ICL å’Œ unsupervised ICL æ¯”ä½¿ç”¨æ‰‹å·¥ç”Ÿæˆ�çš„ few-shot æ•ˆæ�œæ›´æœ‰æ•ˆã€‚

![image.png](/static/img/332c6105ea2d2d6ec6bf81cee7c2416e.image.png)

**Contributions**:
- **Scaling ICL**: ä½¿ç”¨äº† Gemini 1.5 Pro ç³»ç»Ÿåœ°è¯„ä¼°äº† ICL åœ¨å�„ç§�ä»»åŠ¡ä¸�å�Œè§„æ¨¡çš„ä¸Šä¸‹æ–‡ç¤ºä¾‹ä¸‹çš„æ€§èƒ½ã€‚ç»“æ�œè¡¨æ˜�ï¼Œä»� few-shot åˆ° many-shot æ—¶ï¼Œæ€§èƒ½ä¼šå‡ºç�°å¤§å¹…æ��å�‡ã€‚
- **Reforced and Unsupervised ICL**: å�‘ç�°ä½¿ç”¨æ¨¡å�‹ç”Ÿæˆ�çš„ç�†ç”±æˆ–æ˜¯å�ªä½¿ç”¨é—®é¢˜èƒ½å¤Ÿå‡�å°‘å¯¹äº�æ‰‹å·¥ç”Ÿæˆ�æ•°æ�®çš„ä¾�èµ–ã€‚
- **Analysing ICL**: many-shot ICL èƒ½å¤Ÿå…‹æœ�é¢„è®­ç»ƒå��å·®ï¼Œæ€§èƒ½ä¸�å¾®è°ƒç›¸å½“ï¼Œå¹¶ä¸”å�¯ä»¥å­¦ä¹ é�� NLP ä»»åŠ¡ï¼Œè€Œ few-shot ICL åœ¨è¿™äº›ä»»åŠ¡ä¸­åˆ™æ˜¯å��åˆ†çš„å�ƒåŠ›ã€‚è¿˜æ�­ç¤ºäº† next-token prediction loss å�¯èƒ½å¹¶ä¸�èƒ½å¾ˆå¥½åœ°é¢„æµ‹ ICL çš„æ€§èƒ½ã€‚

ä¸ºäº†è�·å¾—å�¯é� çš„ç»“æ�œï¼Œä½œè€…ä½¿ç”¨ä¸�å�Œçš„éš�æœºæ•°ç§�å­�å¯¹æ¯�ä¸ª K-shot promptè¿›è¡Œå¤šæ¬¡éš�æœºæŠ½æ ·ï¼Œå¹¶æŠ¥å‘Šå¹³å�‡æ€§èƒ½ä»¥å�Šå�•ä¸ª seed çš„æ€§èƒ½ã€‚ä¸ºäº†ç¡®ä¿�ä½¿ç”¨ K-shot prompt é€�æ­¥æ��ä¾›æ›´å¤šä¿¡æ�¯ï¼Œè®¾ç½®ä¸­çš„ ä»»ä½• K-shot éƒ½åŒ…æ‹¬å°‘äº� K-shot æ��ç¤ºä¸­çš„æ‰€æœ‰ä¸Šä¸‹æ–‡ç¤ºä¾‹ã€‚

**Scaling In-Context Learning**

- Machine Translation**

ä½¿ç”¨ chrF2++ å¯¹æµ‹è¯•é›†ä¸­çš„å‰� 150 ä¸ªå�¥å­�è¿›è¡Œäº†æ€§èƒ½è¯„ä¼°ï¼ŒchrF2++ æ˜¯åŸºäº�ç”Ÿæˆ�è¯‘æ–‡å’Œå�‚è€ƒè¯‘æ–‡ä¹‹é—´å­—ç¬¦å’Œå�•è¯� n-gram é‡�å�ˆåº¦çš„æ ‡å‡†æŒ‡æ ‡ã€‚

![image.png](/static/img/714c6ea26d3b37ac104cf6f3ec04cdef.image.png)

- Planning: Logistics Domain

![image.png](/static/img/7e67eae3a5ff20a4abadd7b05baea7a6.image.png)

- Reward Modelling with Many-Shot ICL: Learning Code Verifiers

![image.png](/static/img/e3b97f57e7180c65e15086a6a55228ec.image.png)

**Many-shot learning without Human-Written Rationales**

- Reinforced ICL
- Unsupervised ICL: ç”±ä»¥ä¸‹éƒ¨åˆ†ç»„æˆ� 1.å‰�è¨€ï¼Œå¦‚"You will be provided questions similar to the ones below." 2.æœªè§£å†³çš„è¾“å…¥æˆ–é—®é¢˜åˆ—è¡¨ 3. zero-shot instruction æˆ–å¸¦æœ‰æ‰€éœ€è¾“å‡ºæ ¼å¼�è¾“å‡ºçš„ few-shot prompt

å½“ LLM å·²ç»�æ�Œæ�¡äº†è§£å†³ä»»åŠ¡æ‰€éœ€çš„çŸ¥è¯†æ—¶ï¼Œåœ¨æ��ç¤ºä¸­æ�’å…¥ä»»ä½•å�¯ä»¥ç¼©å°�ä»»åŠ¡æ‰€éœ€çŸ¥è¯†èŒƒå›´çš„ä¿¡æ�¯éƒ½ä¼šæœ‰æ‰€å¸®åŠ©ã€‚

![image.png](/static/img/619c405fecd92ebfbef14abb686bd20e.image.png)

Singh ç­‰äººå�‘ç�°ï¼Œæ ¹æ�® MATH æ•°æ�®ç”Ÿæˆ�çš„æ¨¡å�‹æ•°æ�®é›†å¾®è°ƒçš„æ¨¡å�‹ï¼Œå�¯æ��é«˜ GSM8K çš„æµ‹è¯•æ€§èƒ½ï¼Œè€Œ GSM8K çš„é—®é¢˜åˆ†å¸ƒä¸� MATH ä¸�å�Œã€‚

![image.png](/static/img/ccbda1298a7330187394dcd93c0a6e78.image.png)

![image.png](/static/img/cf9e1151fcef1d42fb6e25410eb936f4.image.png)

å¦‚ Figure 9 æ‰€ç¤ºï¼Œåœ¨å‡ ä¹�æ‰€æœ‰çš„ä»»åŠ¡ä¸­ï¼ŒReinforced ICL éƒ½å¤§å¤§ä¼˜äº� Unsupervised ICLï¼Œè€Œ Unsupervised ICL å�ˆä¼˜äº�æ ‡å‡†çš„ 3-shot CoT promptã€‚

**Analyzing Many-Shot ICL**

![image.png](/static/img/8731b575b9bf3de034900a583ca12030.image.png)

è¿™è¡¨æ˜�ï¼Œåœ¨ shot è¾ƒå°‘çš„ ICL ä¸­ï¼Œæ¨¡å�‹éš¾ä»¥å…‹æœ�é¢„å…ˆè®­ç»ƒä¸­å­˜åœ¨çš„å��å·®ã€‚ç„¶è€Œï¼Œéš�ç�€ shot çš„å¢�åŠ ï¼Œç¿»è½¬æ ‡ç­¾å’ŒæŠ½è±¡æ ‡ç­¾çš„æ€§èƒ½æ˜¾è‘—æ��é«˜ï¼Œæ�¥è¿‘é»˜è®¤æ ‡ç­¾çš„æ€§èƒ½ã€‚

![image.png](/static/img/8187bf086caa19695b2dc6af8fbbca69.image.png)

![image.png](/static/img/db90ff91ba92ffa6acf28f7a54eeda3c.image.png)

å…·ä½“æ�¥è¯´ï¼Œæˆ‘ä»¬å¯¹æ�¥è‡ª MATH è®­ç»ƒæ‹†åˆ†çš„å›ºå®š 50 ä¸ªä¸Šä¸‹æ–‡ç¤ºä¾‹çš„å��ç§�ä¸�å�Œéš�æœºæ�’åº�è¿›è¡Œäº†è¯„ä¼°ï¼Œå¹¶å¯¹ä¿�ç•™çš„ MATH500 æµ‹è¯•é›†è¿›è¡Œäº†æ€§èƒ½è¯„ä¼°ã€‚

![image.png](/static/img/9d2221b86fb68571bc88cf163a566127.image.png)

ç›¸æ¯”ä¹‹ä¸‹ï¼Œå¤šé•œå¤´ ICL ä¸�éœ€è¦�ä»»ä½•è®­ç»ƒï¼Œä½†æ�¨ç�†æˆ�æœ¬è¾ƒå¤§ï¼Œè€Œ KV ç¼“å­˜å�¯ä»¥å¤§å¹…é™�ä½�æ�¨ç�†æˆ�æœ¬ã€‚

![image.png](/static/img/0960f01e7006bd8cc03aaf16797b6fdd.image.png)

æ­¤å¤–ï¼Œå¤šé•œå¤´æ€§èƒ½å¾ˆå�¯èƒ½æˆ�ä¸ºè¯„ä¼°é•¿è¯­å¢ƒæ¨¡å�‹è´¨é‡�çš„ä¸€ä¸ªæœ‰ä»·å€¼çš„æŒ‡æ ‡ï¼Œè¶…è¶Šäº†æµ�è¡Œçš„å¤§æµ·æ��é’ˆæµ‹è¯•ã€‚

# ğŸ¤–ChatGPT

:::info{title="ChapGPT"}
1. è®ºæ–‡çš„åˆ›æ–°ä¹‹å¤„ä¸�ç‹¬ç‰¹æ€§ï¼š
   - **æ‰©å±•çš„ä¸Šä¸‹æ–‡çª—å�£ä¸�å¤šç¤ºä¾‹å­¦ä¹ **ï¼šè¯¥è®ºæ–‡é€šè¿‡æ‰©å±•LLMçš„ä¸Šä¸‹æ–‡çª—å�£ï¼Œé¦–æ¬¡åœ¨è‡ªç„¶è¯­è¨€å¤„ç�†ä¸­æ·±å…¥ç ”ç©¶äº†å¤šç¤ºä¾‹æƒ…å¢ƒå­¦ä¹ ï¼ˆMany-Shot In-Context Learningï¼ŒICLï¼‰çš„æ€§èƒ½ã€‚ä¼ ç»Ÿçš„å°‘ç¤ºä¾‹ICLä»…å¤„ç�†å°‘é‡�è¾“å…¥-è¾“å‡ºç¤ºä¾‹ï¼Œè€Œå¤šç¤ºä¾‹ICLåˆ™èƒ½å¤Ÿå¤„ç�†æˆ�ç™¾ä¸Šå�ƒä¸ªç¤ºä¾‹ï¼Œå¤§å¹…æ��é«˜äº†æ¨¡å�‹åœ¨ç”Ÿæˆ�ä»»åŠ¡å’Œåˆ¤åˆ«ä»»åŠ¡ä¸­çš„è¡¨ç�°ã€‚ 
   - **å…‹æœ�é¢„è®­ç»ƒå��å·®**ï¼šè®ºæ–‡å�‘ç�°ï¼Œå¤šç¤ºä¾‹ICLåœ¨ä»»åŠ¡å¤�æ�‚åº¦å¢�åŠ æ—¶ï¼Œèƒ½å¤Ÿæ˜¾è‘—å…‹æœ�é¢„è®­ç»ƒæ¨¡å�‹çš„å��å·®ï¼Œè¿™æ˜¯åœ¨å°‘ç¤ºä¾‹å­¦ä¹ ä¸­éš¾ä»¥å�šåˆ°çš„ã€‚è®ºæ–‡æŒ‡å‡ºï¼ŒICLåœ¨æ•°å­¦é—®é¢˜è§£å†³å’Œæ�¨ç�†ä»»åŠ¡ä¸­çš„è¡¨ç�°ä¸�æ¨¡å�‹å¾®è°ƒï¼ˆfine-tuningï¼‰ç›¸å½“ã€‚
   - **æ¨¡å�‹ç”Ÿæˆ�æ�¨ç�†é“¾ä¸�æ— ç›‘ç�£å­¦ä¹ **ï¼šå¼•å…¥äº†å¼ºåŒ–ICLå’Œæ— ç›‘ç�£ICLä¸¤ç§�æ–°æ–¹å¼�ï¼Œä»¥åº”å¯¹å¤šç¤ºä¾‹å­¦ä¹ å¯¹äººç±»ç”Ÿæˆ�æ•°æ�®çš„é«˜éœ€æ±‚ã€‚é€šè¿‡ä½¿ç”¨æ¨¡å�‹ç”Ÿæˆ�çš„æ�¨ç�†é“¾æ›¿ä»£äººç±»æ�¨ç�†é“¾ï¼Œè®ºæ–‡æˆ�åŠŸåœ¨ä¸€äº›å¤�æ�‚æ�¨ç�†ä»»åŠ¡ä¸­å�–å¾—äº†æ˜¾è‘—æ•ˆæ�œã€‚
   
2. è®ºæ–‡ä¸­å­˜åœ¨çš„é—®é¢˜å�Šæ”¹è¿›å»ºè®®ï¼š
   - **ä¸Šä¸‹æ–‡çª—å�£å¯¹æ€§èƒ½çš„å½±å“�**ï¼šå°½ç®¡è®ºæ–‡æŒ‡å‡ºäº†å¤šç¤ºä¾‹ICLçš„æ˜¾è‘—ä¼˜åŠ¿ï¼Œä½†éƒ¨åˆ†ä»»åŠ¡åœ¨ç¤ºä¾‹æ•°ç›®å¢�åŠ å��ï¼Œæ€§èƒ½å��è€Œæœ‰æ‰€ä¸‹é™�ï¼ˆä¾‹å¦‚MATHä»»åŠ¡ï¼‰ã€‚è¿™è¡¨æ˜�ä»…å¢�åŠ ç¤ºä¾‹æ•°æœªå¿…æ€»æ˜¯å¸¦æ�¥æ€§èƒ½æ��å�‡ã€‚æ”¹è¿›å»ºè®®æ˜¯è¿›ä¸€æ­¥ç ”ç©¶ICLä¸­ç¤ºä¾‹é¡ºåº�å’Œä¸�å�Œç¤ºä¾‹ç»„å�ˆå¯¹æ€§èƒ½çš„å½±å“�ï¼Œä¼˜åŒ–ç¤ºä¾‹çš„é€‰æ‹©å’Œæ�’åº�ç­–ç•¥ã€‚
   - **è´Ÿå¯¹æ•°ä¼¼ç„¶ä¸�å®�é™…æ€§èƒ½ä¸�ä¸€è‡´**ï¼šè®ºæ–‡æŒ‡å‡ºï¼Œè´Ÿå¯¹æ•°ä¼¼ç„¶ï¼ˆNLLï¼‰å¹¶ä¸�èƒ½å¾ˆå¥½åœ°é¢„æµ‹ICLåœ¨å¤�æ�‚é—®é¢˜ä¸Šçš„è¡¨ç�°ã€‚è¿™æ�­ç¤ºäº†å½“å‰�è¯„ä¼°æŒ‡æ ‡çš„å±€é™�æ€§ã€‚å»ºè®®è¿›ä¸€æ­¥æ�¢ç´¢æ›´é€‚å�ˆè¯„ä¼°ICLçš„æŒ‡æ ‡ï¼Œä¾‹å¦‚å¤šæ ·æ€§ç”Ÿæˆ�ã€�æ�¨ç�†æ·±åº¦ç­‰ã€‚
   - **å¯¹æ¨¡å�‹ç”Ÿæˆ�æ�¨ç�†é“¾çš„ä¿¡ä»»é—®é¢˜**ï¼šè™½ç„¶æ¨¡å�‹ç”Ÿæˆ�æ�¨ç�†é“¾åœ¨å¼ºåŒ–ICLä¸­å�–å¾—äº†è¾ƒå¥½çš„è¡¨ç�°ï¼Œä½†å�¯èƒ½å‡ºç�°å�‡é˜³æ€§é—®é¢˜ï¼Œå�³é”™è¯¯çš„æ�¨ç�†é“¾å¯¼è‡´æ­£ç¡®çš„æœ€ç»ˆç­”æ¡ˆã€‚å»ºè®®ç»“å�ˆæ›´å¤šè´¨é‡�æ�§åˆ¶æœºåˆ¶ï¼Œä¾‹å¦‚åœ¨æ�¨ç�†é“¾ç”Ÿæˆ�è¿‡ç¨‹ä¸­å¼•å…¥å¤šæ ·æ€§é‡‡æ ·æˆ–é€šè¿‡å…¶ä»–éªŒè¯�æ¨¡å�‹è¿›è¡Œäº¤å�‰éªŒè¯�ã€‚

3. åŸºäº�è®ºæ–‡çš„å†…å®¹å’Œç ”ç©¶ç»“æ�œï¼Œæ��å‡ºçš„åˆ›æ–°ç‚¹æˆ–ç ”ç©¶è·¯å¾„ï¼š
   - **ç ”ç©¶ç‚¹1ï¼šå¤šä»»åŠ¡å¤šç¤ºä¾‹ICLçš„ç¤ºä¾‹æ�’åº�ç­–ç•¥ä¼˜åŒ–**ï¼šç ”ç©¶ç¤ºä¾‹åœ¨å¤šä»»åŠ¡ç�¯å¢ƒä¸‹çš„æ�’åº�ä¸�ç»„å�ˆå¦‚ä½•å½±å“�ICLæ€§èƒ½ã€‚
   - **ç ”ç©¶ç‚¹2ï¼šè·¨é¢†åŸŸçŸ¥è¯†è¿�ç§»ä¸‹çš„å¤šç¤ºä¾‹ICL**ï¼šæ�¢ç´¢å¤šç¤ºä¾‹ICLåœ¨è·¨é¢†åŸŸä»»åŠ¡ä¸­çš„è¿�ç§»èƒ½åŠ›ï¼Œå°¤å…¶æ˜¯åœ¨å®Œå…¨æœªçŸ¥é¢†åŸŸä¸­çš„è¡¨ç�°ã€‚
   - **ç ”ç©¶ç‚¹3ï¼šè‡ªé€‚åº”ç¤ºä¾‹é€‰æ‹©ä¸�è‡ªæˆ‘ç›‘ç�£çš„ç»“å�ˆ**ï¼šå¼€å�‘è‡ªé€‚åº”çš„ç¤ºä¾‹é€‰æ‹©ç­–ç•¥ï¼Œå¹¶ç»“å�ˆè‡ªæˆ‘ç›‘ç�£æœºåˆ¶ï¼Œå‡�å°‘å¯¹é«˜è´¨é‡�äººç±»ç”Ÿæˆ�æ•°æ�®çš„ä¾�èµ–ã€‚

4. ä¸ºæ–°çš„ç ”ç©¶è·¯å¾„åˆ¶å®šçš„ç ”ç©¶æ–¹æ¡ˆï¼š
   - **ç ”ç©¶ç‚¹1ï¼šå¤šä»»åŠ¡å¤šç¤ºä¾‹ICLçš„ç¤ºä¾‹æ�’åº�ç­–ç•¥ä¼˜åŒ–**
     - **ç ”ç©¶æ–¹æ³•**ï¼šåœ¨å¤šä»»åŠ¡ï¼ˆå¦‚ç¿»è¯‘ã€�æ�¨ç�†ã€�åˆ†ç±»ï¼‰ç�¯å¢ƒä¸‹ï¼Œä½¿ç”¨å¼ºåŒ–å­¦ä¹ æ–¹æ³•è®­ç»ƒç¤ºä¾‹é€‰æ‹©å™¨ï¼Œä»¥ç¡®å®šæœ€ä¼˜ç¤ºä¾‹æ�’åº�ç­–ç•¥ã€‚é€šè¿‡ç¤ºä¾‹çš„æ�’åˆ—ç»„å�ˆæ�¥è¯„ä¼°ä¸�å�Œä»»åŠ¡ä¸‹çš„ICLæ€§èƒ½ã€‚
     - **æ­¥éª¤**ï¼š
       1. æ”¶é›†å¤šä»»åŠ¡æ•°æ�®é›†ï¼Œç¡®ä¿�æ¯�ä¸ªä»»åŠ¡æœ‰è¶³å¤Ÿçš„ç¤ºä¾‹ã€‚
       2. è®­ç»ƒä¸€ä¸ªå¼ºåŒ–å­¦ä¹ ä»£ç�†ï¼Œé’ˆå¯¹æ¯�ä¸ªä»»åŠ¡å­¦ä¹ æœ€ä¼˜çš„ç¤ºä¾‹æ�’åº�ã€‚
       3. æ¯”è¾ƒä¸�å�Œæ�’åº�ç­–ç•¥å¯¹æ¨¡å�‹åœ¨æ¯�ä¸ªä»»åŠ¡ä¸Šçš„è¡¨ç�°ã€‚
       4. ä¼˜åŒ–ç¤ºä¾‹é€‰æ‹©æœºåˆ¶ï¼Œä½¿å…¶é€‚åº”ä¸�å�Œä»»åŠ¡éœ€æ±‚ã€‚
     - **æœŸæœ›æˆ�æ�œ**ï¼šæ��é«˜å¤šä»»åŠ¡æƒ…å¢ƒä¸‹çš„ICLè¡¨ç�°ï¼Œå¹¶ä¸”å‡�å°‘ç¤ºä¾‹æ�’åº�å¯¹æ¨¡å�‹æ€§èƒ½çš„è´Ÿé�¢å½±å“�ã€‚

   - **ç ”ç©¶ç‚¹2ï¼šè·¨é¢†åŸŸçŸ¥è¯†è¿�ç§»ä¸‹çš„å¤šç¤ºä¾‹ICL**
     - **ç ”ç©¶æ–¹æ³•**ï¼šè®¾è®¡å®�éªŒï¼ŒéªŒè¯�ICLåœ¨è·¨é¢†åŸŸä»»åŠ¡ä¸­çš„æ³›åŒ–èƒ½åŠ›ï¼Œå°¤å…¶æ˜¯åœ¨é¢„è®­ç»ƒæ•°æ�®ä¸­æ²¡æœ‰è¦†ç›–çš„ä»»åŠ¡ä¸Šã€‚ä½¿ç”¨ä¸�å�Œé¢†åŸŸçš„æ•°æ�®è¿›è¡Œè®­ç»ƒï¼Œç ”ç©¶æ¨¡å�‹å¦‚ä½•è¿�ç§»å­¦ä¹ ã€‚
     - **æ­¥éª¤**ï¼š
       1. ä»�ä¸�å�Œé¢†åŸŸï¼ˆå¦‚åŒ»å­¦ã€�æ³•å¾‹ã€�å·¥ç¨‹ç­‰ï¼‰æ”¶é›†ä»»åŠ¡æ•°æ�®ã€‚
       2. è®¾è®¡å®�éªŒï¼Œåˆ†åˆ«åœ¨å�Œé¢†åŸŸå’Œè·¨é¢†åŸŸè®¾ç½®ä¸‹è¿›è¡ŒICLè®­ç»ƒã€‚
       3. è¯„ä¼°æ¨¡å�‹åœ¨è·¨é¢†åŸŸä»»åŠ¡ä¸­çš„è¿�ç§»æ€§èƒ½ï¼Œå¹¶åˆ†æ��å½±å“�è¿�ç§»æ•ˆæ�œçš„å› ç´ ã€‚
     - **æœŸæœ›æˆ�æ�œ**ï¼šéªŒè¯�å¤šç¤ºä¾‹ICLåœ¨è·¨é¢†åŸŸä»»åŠ¡ä¸Šçš„è¿�ç§»èƒ½åŠ›ï¼Œæ��å‡ºæ”¹è¿›çš„å¤šç¤ºä¾‹ICLæ¨¡å�‹ï¼Œæ��å�‡å…¶è·¨é¢†åŸŸæ³›åŒ–èƒ½åŠ›ã€‚

   - **ç ”ç©¶ç‚¹3ï¼šè‡ªé€‚åº”ç¤ºä¾‹é€‰æ‹©ä¸�è‡ªæˆ‘ç›‘ç�£çš„ç»“å�ˆ**
     - **ç ”ç©¶æ–¹æ³•**ï¼šå¼€å�‘ä¸€ä¸ªåŸºäº�è‡ªæˆ‘ç›‘ç�£çš„ç¤ºä¾‹é€‰æ‹©æœºåˆ¶ï¼Œä½¿å¾—æ¨¡å�‹èƒ½å¤Ÿè‡ªåŠ¨ç­›é€‰å‡ºå¯¹ä»»åŠ¡æœ€æœ‰å¸®åŠ©çš„ç¤ºä¾‹ï¼Œå¹¶ä¸�æ–­è‡ªæˆ‘æ”¹è¿›ã€‚ç»“å�ˆå¼ºåŒ–ICLï¼Œå‡�å°‘å¯¹äººç±»ç”Ÿæˆ�æ•°æ�®çš„ä¾�èµ–ã€‚
     - **æ­¥éª¤**ï¼š
       1. è®¾è®¡è‡ªæˆ‘ç›‘ç�£æœºåˆ¶ï¼Œå…�è®¸æ¨¡å�‹ç”Ÿæˆ�å¹¶éªŒè¯�è‡ªèº«æ�¨ç�†é“¾ã€‚
       2. è®­ç»ƒæ¨¡å�‹å­¦ä¹ å“ªäº›ç¤ºä¾‹å¯¹ä»»åŠ¡å®Œæˆ�æœ€ä¸ºå…³é”®ï¼Œå¹¶åŠ¨æ€�è°ƒæ•´ç¤ºä¾‹é€‰æ‹©ã€‚
       3. åœ¨å¤šç§�ä»»åŠ¡ä¸Šæµ‹è¯•è‡ªé€‚åº”ç¤ºä¾‹é€‰æ‹©çš„æ•ˆæ�œï¼Œå¹¶å¯¹æ¯”ä¼ ç»Ÿçš„å›ºå®šç¤ºä¾‹é€‰æ‹©æ–¹æ³•ã€‚
     - **æœŸæœ›æˆ�æ�œ**ï¼šå‡�å°‘å¤šç¤ºä¾‹ICLå¯¹äººå·¥ç”Ÿæˆ�æ•°æ�®çš„ä¾�èµ–ï¼Œæ��é«˜ä»»åŠ¡æ�¨ç�†çš„è´¨é‡�ä¸�æ•ˆç�‡ã€‚

:::

# Others

GATEIO UTBYTE VILKET LANDSÄTTA IN PENGAR PÅ GATE.IO

GATE IO NIVÅ 1