AI inside CEO Taku Toguchi on the road to LLM development
Meeting generative AI
In 2014, a paper titled “Generative Adversarial Nets” (GAN) was published. That was the beginning of “generative AI” in my own experience. As I recall, around 2017 we had already brought GANs into AI inside’s image-recognition work — specifically to augment training data. In an era when we were still synthesizing images by hand, systematically generating and blending noise to amplify the few training samples we had, GANs were strikingly advanced.
In 2018 came a paper on pre-training deep bidirectional transformers for language understanding: “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” This is what would come to be widely known as the “Transformer model.” Honestly, the first time I touched BERT, what I felt was no more than “a high-performance natural-language model has arrived.” It turned out to be the revolutionary step that would later lead to ChatGPT.
How I felt about ChatGPT
The jolt of touching GPT was nothing like BERT’s. It was a shock strong enough to make me feel, intuitively, that “the era has changed.” By 2023 it had advanced to version 4 and won broad public acceptance. Thanks to the new interface called “ChatGPT,” even people far removed from IT were talking about the technology and taking an interest in it.
In the U.S., the conversation was increasingly about innovation, bias, and privacy. In Japan, the discussion was dominated — almost entirely — by “what should Japan do?” and “using ChatGPT is Japan’s path to success.” Against that, I will be honest: I felt a miserable, underdog kind of emotion — frustrated that Japan had still produced nothing like GPT, and that no one was setting out to beat the American companies.
The conviction that led to PolySphere-1
But I couldn’t let it end at frustration. I kept developing and investigating on my own, trial after trial. In the process I arrived at a hypothesis: “OpenAI’s real bottleneck is the exhaustion of GPUs.” That is — even GPT, which looks as if its capability is rising without limit, in fact has an invisible ceiling, and compute will run short. On top of that, as users surge, the resource per user is divided down, so functions that ought to be available cannot be offered. In practice, the token length you can feed in is short. It can certainly be made longer. Fine-tuning isn’t offered either — these look like capabilities that could have been delivered long ago but are being held back and deferred.
If that assumption was right, then from a technology-and-business standpoint what’s needed is “an efficient architecture” and “a scale fit for the purpose,” and on features, “the things ChatGPT lacks — long tokens and fine-tuning.” Of that I became convinced.
Amid the ChatGPT flood, announcing PolySphere-1
While I was turning this over and experimenting, many companies were issuing press releases about ChatGPT. Even though the API had not yet been released, more than a few listed companies announced things like “we’ve integrated with the API” — a real mess. Few people were in a position to point this out anywhere. During this period I, too, was asked again and again: “How does AI inside think about, and respond to, ChatGPT and LLMs?” My answer was this: “I have no intention of issuing a press release strung together from buzzwords. Nor of announcing a mere integration that users can’t actually use. Our policy is to speak only once we have given value a form.”
And so, after founding a team called XResearch inside the company, on June 8, 2023 we announced AI inside’s LLM, PolySphere-1. I have consistently worked for the sake of humanity, but this time, honestly, the drive to “aim for a strong Japan and go to win” was especially strong. I also carried a quiet resolve: we cannot let the “lost thirty years” stretch on any further.
What PolySphere-1 is
How is PolySphere-1 different from ChatGPT? I get this question a lot. So let me first explain a little about PolySphere-1’s basic characteristics.
On parameter scale
PolySphere-1 is, at present, a general-purpose design. That is, it isn’t specialized for a particular purpose; like GPT, it can handle a range of tasks. As for the number of parameters, it stands at 14 billion — about ten times GPT-2, and roughly a tenth of GPT-3. Internally, expanding from 40 billion parameters, and to still larger counts, does come up for discussion. It is genuinely feasible.
But parameter count is not a simple “more means smarter.” Keeping an appropriate scale matters from the standpoint of training speed and applicability. In other words, to deliver the best performance “as a business,” what’s required is not merely raising the parameter count but the balance and optimization of it.
On security
The questions I get most from investors are about security, response accuracy, and response speed.
First, security. AI inside is a Japanese listed company, and PolySphere-1’s processing runs on servers inside Japan. For Japanese companies, that is an easy point to accept. This is already achieved.
To add to that, on June 12, 2023 AI inside announced a new service called “AnyData”. It is a public-cloud service in which each customer can build their own environment and run data and processing in a private setting. Auto-scaling comes built in as standard. If, on this kind of infrastructure, one could easily launch and manage, say, a PolySphere-1 image container, the barrier to using AI would drop dramatically. This is in development and will be ready soon.
On response speed
Speed, too, is shaped by this environment. In a shared environment you’re affected by other users’ usage, so you also have to think about stability; but if you can build and run a private environment, stable operation is possible, and depending on the resources you provision, speed rises as well. If you care about latency, make the resources stronger; if slow is fine, run it slow and cheap. That, I think, is the right way.
On response accuracy
On response accuracy, we’re still at an early stage, evaluating at the level of felt experience. To put it grandly, you might think of it as a kind of “Turing test.” An earlier in-house test at 3 billion parameters could hold a conversation but was not at a level applicable to real work. As it stands now, it has evolved to a level genuinely usable in real work. We’re aiming for further, concrete improvement.
As a next goal, I’m also considering measuring the accuracy gap against other large language models quantitatively, in scenarios that assume specific tasks, and using those results to drive improvement. This is a point of differentiation, because our AI, “PolySphere-1,” aims not merely to converse but to do “work.” That said, this is for evaluation; the users and scenarios who actually use it — that is, the UX — are, without a doubt, everything.
On data and architecture
Secret.
Of course, our architecture differs from GPT’s. That is part of the strategy for the differentiation I described. But this development keeps evolving without pause, so it will surely change.
Judging from experience so far, a year from now today’s effort will probably look like the moment a caveman first lit a fire — a small, important step. (That’s what I hope for.)
Where XResearch goes next
When it comes to LLMs, too much attention falls on the model itself and on the chat feature. In truth, the focus should be on user experience (UX). We need to probe deeply into how users engage with AI day to day — its usefulness, and the “meaning of using it daily.”
Turning cutting-edge technology into a form users can operate with extreme ease — this is exactly the territory AI inside is good at, and at the same time where the big opportunity lies.
The XResearch team won’t only explore generative-AI technology; it will look at “how it gets accepted,” and take on challenges beyond that scope as well. Already, together with the Yoshida Laboratory at Tohoku University, we are advancing joint research on AI for space-resource exploration. What we aim for is always, in the end, to contribute to the progress of humanity.