Swallow (Mistral)

Changelog

• 2024-04-26: Released Swallow-MS-7b-instruct-v0.1.
• 2024-03-11: Released Swallow-MS-7b-v0.1 and Swallow-MX-8x7b-NVE-v0.1.

Overview

The large language models Swallow-MS 7B and Swallow-MX 8x7B were developed by research teams from the Okazaki and Yokota Laboratories at the School of Computing, Tokyo Institute of Technology, together with the National Institute of Advanced Industrial Science and Technology (AIST). To enhance the Japanese capabilities of the large language models Mistral 7B and Mixtral 8x7B, which exhibit strong performance in English language understanding and dialogue, the research team conducted continued pre-training using large-scale Japanese language data. In performance evaluations conducted by the team, Swallow-MS 7B achieved the highest performance among open 7B large language models on benchmarks related to Japanese knowledge, reasoning, and language generation (as of March 2024, in comparisons among base models). In addition, Swallow-MX 8x7B adopts a Mixture of Experts (MoE) architecture and is the first open model with enhanced Japanese capabilities based on this architecture. The released models can be downloaded from Hugging Face.

The licenses for Swallow-MS 7B and Swallow-MX 8x7B inherit the Apache 2.0 licenses of Mistral 7B and Mixtral 8x7B. As long as users comply with this license, the models may be used for both research and commercial purposes.

Performance

Swallow-MS 7B

Let us compare the Japanese performance of models obtained through continued pre-training of Mistral 7B and Llama 2, as well as other 7B models. Swallow-MS 7B steadily improves the Japanese capabilities of Mistral 7B through continued pre-training. On the Japanese benchmarks used for evaluation, Swallow-MS 7B shows higher average performance than other open 7B models available as of March 2024. In particular, it achieves substantial improvements on knowledge-oriented tasks such as JCommonsenseQA (JComQA) and NIILC, a trend similar to the performance gains observed for Swallow over Llama 2. In addition, Swallow-MS 7B extends Mistral 7B with Japanese vocabulary, enabling more few-shot examples to be packed into prompts, reducing garbled text generation, and accelerating generation speed.

Next, let us examine the performance in English.
As is commonly observed in continued pre-training with cross-lingual transfer, performance on English tasks declines in the post–continued pre-training model (Swallow-MS 7B) compared to the original model (Mistral 7B).
In line with the improvements in Japanese performance, the decline is particularly noticeable on knowledge-oriented tasks such as TriviaQA.
We attempted to mitigate the degradation in English performance while improving Japanese performance by mixing Japanese and English data at a 9:1 ratio and adjusting the learning rate. However, possibly due to the small model size, it was not possible to completely prevent the performance drop.
Nevertheless, the average English performance of Swallow-MS 7B slightly surpasses that of Llama 2 7B and remains higher than that of Swallow 7B, and we therefore expect it to be used as a model that is strong in both Japanese and English.

We present here radar charts visualizing selected portions of these evaluation results.

Swallow-MX 8x7B

Next, we examine Swallow-MX 8x7B.
Because this model is a Mixture of Experts (MoE) model that combines eight 7B models, we compare it with 70B-class models that have a similar total number of parameters (Mixtral shares attention and layer normalization parameters across experts, resulting in a total of 47B parameters).
Evaluation on Japanese benchmarks shows that continued pre-training steadily improves the Japanese capabilities of Mixtral 8x7B in Swallow-MX 8x7B.
Significant performance gains are again observed on knowledge-oriented tasks such as JCommonsenseQA (JComQA) and NIILC.
Although it does not surpass Swallow 70B, which has a larger total number of parameters, it demonstrates performance comparable to 70B-class models, highlighting the strong potential of MoE architectures.

Finally, we examine the English performance of Swallow-MX 8x7B.
Unlike in the case of Swallow-MS 7B, Swallow-MX 8x7B shows little degradation compared to the original model, Mixtral 8x7B Instruct.
A similar mitigation of English performance degradation has also been observed in Swallow 70B, suggesting that increasing the number of model parameters may help reduce performance loss, regardless of whether the architecture is MoE or not.
However, in the continued pre-training of Swallow-MX 8x7B, the training data used a Japanese-to-English ratio of 72:28 (due to historical circumstances related to debugging issues in the training framework). Therefore, we would like to further investigate the possibility that differences in the Japanese–English data mixing ratio also contributed to this outcome.

Note that benchmark scores cannot be compared across different datasets. For example, in a model’s Japanese evaluation results, even if the score for mathematics is higher than that for machine translation, this does not imply that the model is better at mathematics than at translation (it would be like comparing the results of entirely different exams with different difficulty levels and grading criteria). For the same reason, even if the average score on English tasks is higher than the average score on Japanese tasks for a given model, one cannot conclude that the model is stronger in English. Because evaluation scales and difficulty levels differ across benchmark datasets, it is inappropriate to discuss task strengths and weaknesses based solely on the shape of this radar chart.

Evaluation Benchmarks

For Japanese evaluation benchmarks, we used llm-jp-eval (v1.0.0) and the JP Language Model Evaluation Harness (commit #9b42d41). The breakdown is as follows:

• Multiple-choice question answering (JCommonsenseQA [Kurihara+, 2022])
• Free-form question answering (JEMHopQA [Ishii+, 2023])
• Free-form question answering (NIILC [Sekine, 2003])
• Machine reading comprehension (JSQuAD [Kurihara+, 2022])
• Automatic summarization (XL-Sum [Hasan+, 2021])
• Machine translation (WMT2020 ja–en [Barrault+, 2020])
• Machine translation (WMT2020 en–ja [Barrault+, 2020])
• Mathematics (MGSM [Shi+, 2023])

Note that natural language inference (NLI), which is commonly used as an evaluation benchmark for large language models, was excluded in this study. Language models tend to exhibit biased label predictions in NLI tasks, and when this bias happens to coincide with the correct labels, the resulting scores become artificially high. Consequently, the evaluation results—especially for 7B models—were unstable.

For English evaluation benchmarks, we used the Language Model Evaluation Harness (v0.3.0). The breakdown is as follows:

• Multiple-choice question answering (OpenBookQA [Mihaylov+, 2018])
• Free-form question answering (TriviaQA [Joshi+, 2017])
• Machine reading comprehension (SQuAD 2.0 [Rajpurkar+, 2018])
• Commonsense reasoning (XWINO [Tikhonov & Ryabinin, 2021])
• Natural language inference (HellaSwag [Zellers+, 2019])
• Mathematics (GSM8K [Cobbe+, 2021])

Method

Swallow-MS 7B and Swallow-MX 8x7B were constructed by applying continued pre-training to Mistral 7B and Mixtral 8x7B Instruct, respectively.
To develop Japanese large language models strong in arithmetic reasoning and code generation, source code corpora were mixed with text corpora during training.
Specifically, Swallow-MS 7B was trained on AlgebraicStack [Azerbayev+, 2024], a corpus of mathematics-related source code, while Swallow-MX 8x7B was trained on both AlgebraicStack and The Vault [Nguyen+, 2023], a corpus pairing natural language with source code.
The effects of incorporating source code corpora will be further investigated through comparative experiments in future work.

The text corpora followed the same configuration as Swallow, using a Japanese-to-English mixture ratio of 9:1 (except for Swallow-MX 8x7B, which used 72:28), and consisted of the Swallow corpus, Japanese Wikipedia, and for English, RefinedWeb and the arXiv subset of The Stack.
In this release, Japanese vocabulary expansion was applied only to Swallow-MS 7B and not to Swallow-MX 8x7B.
With the vocabulary expansion, the number of Hiragana characters included in the vocabulary increased from 58 to 83, Katakana from 76 to 87, and Kanji from 1,456 to 3,208.

Additional pre-training of Mistral and Mixtral was conducted using software developed in-house.

Acknowledgements

The research and development of Swallow-MS and Swallow-MX were supported by several initiatives, including the “Large-Scale Language Model Development Support Program” of the AI Bridging Cloud Infrastructure (ABCI), which is built and operated by AIST; the project “Development of AI Application Technologies to Support Decision-Making in Design Risk Assessment Based on Expert Perspectives” under the NEDO program “Development of Core Integrated Technologies for Next-Generation Artificial Intelligence and Robots” (JPNP18002); and other supporting programs. Part of these results was also achieved through the “Large-Scale Foundation Model Development Support Program” of ABCI. This program was jointly proposed in September 2023 by the LLM-jp study group—organized by the National Institute of Informatics (NII), AIST, and Tokyo Institute of Technology, and involving research teams from institutions such as NII, Tohoku University, the University of Tokyo, and Waseda University—and was subsequently selected. It provided an opportunity to exclusively use a portion of ABCI’s high-performance computational resources (referred to as A-nodes) for up to 60 days. In addition, evaluation experiments of the trained large language models utilized datasets and insights developed within the LLM-jp study group.