Alright, let's dive into the hot goss from the tech world that's got everyone talking. Picture this: an open-source AI model, dubbed "miqu-1-70b," just waltzed into the scene, and guess what? It's nipping at the heels of the big kahuna, GPT-4, in terms of smarts. This isn't your everyday tech leak; it's like finding a secret recipe that could rival the best chefs out there.
Want to use Mistral-medium API now?
Get onboard with Anakin AI to harness the power of Open Source!
Key Points: What's the Big Deal?
- "Miqu-1-70b" popped up out of the blue, making waves for being eerily similar to the top dog, GPT-4.
- This model was sneakily posted on HuggingFace, a hangout spot for AI enthusiasts, by someone going by "Miqu Dev."
- Chat rooms and socials, especially 4chan and X (yup, the one Musk revamped), are buzzing with chatter about this new kid on the block.
- Turns out, Mistral, the brains behind this operation, had a little internal slip-up, leading to the leak.
- This could be a game-changer, shaking up the cozy world where GPT-4's been lounging on the throne.
Leaked Version of Mistral-Medium: "Miqu-1-70b"
So, here's the lowdown: On a fine day around January 28, "Miqu Dev" decides to drop a bombshell on HuggingFace, throwing in a set of files that make up "miqu-1-70b." Now, HuggingFace isn't just any platform; it's like the go-to joint for folks who love tinkering with AI models. The buzz is that this "miqu-1-70b" model is sort of a doppelganger for Mistral's stuff, which is top-notch in the open-source AI world. They're the cool cats who fine-tuned something called Llama 2, and let me tell you, it's pretty slick.
Posted On 4Chan and X
But wait, there's more! The same day "Miqu Dev" did their thing on HuggingFace, some shadowy figure drops a link to the "miqu-1-70b" files on 4chan.
Yeah, that 4chan – home to all things wild and wacky on the internet. And just like that, folks start yakking about it, with some even taking to X to show off what this model can do. They're putting it through the wringer, stacking it up against GPT-4, and guess what? It's holding its own, like a champ.
The Debate on Miqu-1-70B, Is It Really A Leak?
Now, here's where it gets juicy. The tech heads start scratching their noggins, wondering if "Miqu" is Mistral's secret sauce they let slip. There's chatter about something called quantization – it's a tech trick to make AI models run on less beefy computers by trimming down the fat in the data. So, the word on the street is that "Miqu" might just be Mistral's latest brainchild, maybe even a lean, mean version of their usual fare.
Whatever Miqu is, it has some sort of special sauce. It gets an 83.5 on EQ-Bench (evaluated locally), surpassing *every other LLM in the world except GPT-4*. EQ-Bench has a 0.97 correlation w/ MMLU, and a 0.94 correlation w/ Arena Elo. It *beats* Mistral Medium - at Q4_K_M. I… pic.twitter.com/0gOOPjxjPD
— N8 Programs (@N8Programs) January 30, 2024
Might be late but I am now 100% convinced that Miqu is the same model that's accessible as Mistral-Medium on Perplexity Labs. It was plausible that it knows standard puzzles, but there ain't no way in Hell a pranker has tuned it to identically phrase the responses in Russian too. pic.twitter.com/zZMcpspXch
— Teortaxes▶️ (@teortaxesTex) January 30, 2024
Mistral AI Confirms the Leak
And then, boom! Arthur Mensch, the big boss over at Mistral, steps into the spotlight. He's like, "Yup, our bad. An eager beaver from our early access crowd got a bit carried away and leaked an old model." But here's the kicker: they've been cooking up something even better since then. Mensch's dropping hints left and right that they're on the brink of unveiling a model that could go toe-to-toe with GPT-4, or dare we say, outshine it.
An over-enthusiastic employee of one of our early access customers leaked a quantised (and watermarked) version of an old model we trained and distributed quite openly.
— Arthur Mensch (@arthurmensch) January 31, 2024
To quickly start working with a few selected customers, we retrained this model from Llama 2 the minute we got…
What This Means for the AI World
Imagine the possibilities here. If Mistral rolls out an open-source model that's on par with GPT-4, and it's free for all to use, that's going to send shockwaves through the AI scene. OpenAI might have to watch its back because there's a new contender in town, ready to challenge the status quo. This leak could very well be the spark that lights up a whole new era in AI, where the big names have to share the spotlight with the open-source underdogs.
MIQU-1-70B Technical Review
The enigmatic MIQU-1-70B model, suspected to be a leaked artifact from MistralAI, has generated considerable intrigue in the AI realm. This model, presumed to be either a variant of Mistral Medium or a relic from an older MoE experiment, underwent rigorous testing to ascertain its translation capabilities, instruction adherence, and proficiency in handling multilingual content.
Testing Methodology
The MIQU-1-70B was evaluated through four professional German data protection training exams, reflecting the actual certification tests required for employees. These tests were aimed at assessing the model's prowess in German-English translation and its ability to comply with specific commands, like acknowledging received information with a simple "OK."
Performance Analysis of Miqu-1-70b
Miqu-1-70B's performance was commendable, with the model correctly answering 17 out of 18 multiple-choice questions, showcasing its robust understanding of the content. However, it failed to adhere to the instruction of responding with "OK" to acknowledge information, which marks a shortfall in instruction compliance.
Benchmark of miqu-1-70b
| Rank | Model | Size | Format | Context | Prompt | 1st Score | 2nd Score | OK | +/- |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-4 | GPT-4 | API | 18/18 ✓ | 18/18 ✓ | ✓ | ✓ | ||
| 1 | goliath-120b-GGUF | 120B | GGUF | Q2_K | 4K Vicuna 1.1 | 18/18 ✓ | 18/18 ✓ | ✓ | ✓ |
| 1 | Tess-XL-v1.0-GGUF | 120B | GGUF | Q2_K | 4K Synthia | 18/18 ✓ | 18/18 ✓ | ✓ | ✓ |
| 1 | Nous-Capybara-34B-GGUF | 34B | GGUF | Q4_0 | 16K Vicuna 1.1 | 18/18 ✓ | 18/18 ✓ | ✓ | ✓ |
| 2 | Venus-120b-v1.0 | 120B | EXL2 | 3.0bpw | 4K Alpaca | 18/18 ✓ | 18/18 ✓ | ✓ | ✗ |
| 3 | lzlv_70B-GGUF | 70B | GGUF | Q4_0 | 4K Vicuna 1.1 | 18/18 ✓ | 17/18 | ✓ | ✓ |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 19 🆕 | miqudev/miqu-1-70b | 70B | GGUF | Q5_K_M | 32K Mistral | 17/18 | 13/18 | ✗ | ✗ |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
In a comparative analysis against other models, Miqu-1-70B showed proficient language skills and bilingual abilities. Despite these strengths, it did not outperform the Mixtral-8x7B-Instruct-v0.1 model or other high-ranking models like GPT-4, Goliath-120B-GGUF, and Tess-XL-v1.0-GGUF, all of which achieved perfect scores in both testing rounds and adhered to the "OK" instruction.
- MIQU-1-70B: Scored 17/18 with a failure to consistently respond with "OK".
- Top Models: GPT-4, Goliath-120B-GGUF, Tess-XL-v1.0-GGUF all scored 18/18, showcasing perfect understanding and instruction adherence.
Is Miqu-1-70B Really Leaked Version of Mistral-Medium?
Speculation about MIQU-1-70B's origin includes theories of it being a leaked Mistral Medium model or an older experimental version. The model demonstrated notable Mixtral-like features, such as excellent bilingual capabilities and additional commentary in responses, yet did not surpass the performance of leading Mixtral models.
MIQU-1-70B emerges as a capable model with significant potential in language comprehension and translation. Nevertheless, its performance, when benchmarked against other leading models, reveals gaps, particularly in following precise instructions. The origins and exact nature of MIQU-1-70B remain speculative, with its performance igniting discussions and debates in the tech community.
Want to use Mistral-medium API now?
Get onboard with Anakin AI to harness the power of Open Source!
What is Mistral-Medium?
Mistral Medium emerges as a formidable language model within the Mistral AI suite, showcasing remarkable capabilities that extend beyond its predecessors, Mistral-tiny and Mistral-small. Characterized by its extensive context window of 32k tokens, approximately translating to 24,000 words, Mistral Medium is designed to facilitate higher reasoning, setting new benchmarks in AI language models.
How Good is Mistral-Medium?
Mistral Medium has demonstrated superior performance, particularly notable in the MMLU (Multiple-Choice Questions in 57 subjects) benchmark, where it achieved a score of 75.3%.
This score signifies its advanced understanding and analytical capabilities, positioning it ahead of Mistral-8x7b and Mistral-7b in comparative assessments.
Mistral-Medium Pricing
The pricing model for Mistral Medium is meticulously structured, with costs delineated as:
- 2.5€ per 1M tokens for input
- 7.5€ per 1M tokens for output
The model adheres to a rate limit of 2 requests per second, enabling users to process up to 2 million tokens per minute and up to 200 million tokens monthly, ensuring both scalability and accessibility for diverse applications.
Is Mistral-Medium Better than GPT-4?
The comparison between Mistral Medium and GPT-4 is a topic of debate, with varying opinions on their respective capabilities. Some discussions and evaluations have provided insights into their performance on different tasks. Here are some key points from the search results:
- Mistral Medium has been reported to outperform GPT-4.5 on certain benchmarks, such as the censorship benchmark.
- A quick comparison involving general knowledge questions, logic/common sense questions, and questions designed to test for hallucinations, showed that Mistral Medium performed well in reasoning questions but was surprisingly less effective in handling hallucination questions compared to GPT-4.
- There are mixed opinions about the performance of Mistral Medium compared to GPT-4, with some expressing concerns about its effectiveness in certain areas.
- The benchmarks for Mistral Medium have been described as underwhelming by some, citing a score of 75.3 on the MMLU (MCQ in 57 subjects) benchmarks.
- Some users have expressed the need for more comprehensive evaluations and comparisons between Mistral Medium and GPT-4 to determine their suitability for specific applications.
In summary, the comparison between Mistral Medium and GPT-4 is complex, with differing opinions on their performance across various tasks. Further in-depth evaluations and comparisons may be necessary to make a conclusive determination of which model is better suited for specific use cases.
Want to use Mistral-medium API now?
Get onboard with Anakin AI to harness the power of Open Source!
Conclusion and Reflection on Open Source AI
Open-source AI models, such as Mistral Medium, embody the spirit of collaboration, innovation, and accessibility that is central to the open-source movement. They offer an alternative to the proprietary models developed by tech giants, democratizing access to cutting-edge AI technologies and enabling a broader range of individuals and organizations to contribute to, and benefit from, the advancements in AI.
The significance of this shift cannot be overstated. Open-source models have the potential to foster a more inclusive and diverse AI ecosystem, where researchers, developers, and enthusiasts from around the world can collaborate, share knowledge, and drive innovation. This collaborative environment can accelerate the pace of AI advancements, leading to more robust, versatile, and ethical AI solutions.
from Anakin Blog http://anakin.ai/blog/miqu-1-70b/
via IFTTT
No comments:
Post a Comment