Field Testing Google’s New AI Assistant

Dear Reader,

I’m a firm believer that one of the best ways to understand the potential of a new product or service is to try it yourself.

In that spirit, I recently got my hands on a new AI tool from Google called NotebookLM.

I first heard about this during Google’s AI event in May. I took that chance to join the waitlist for it.

NotebookLM allows users to upload documents and then leverage Google’s AI to pull facts, answer questions, and interact with the data.

The example Google provided during the demo was compelling: a student takes notes over the course of a semester and then uses NotebookLM to interact with that data.

The project is still labeled as “experimental” by Google, and it comes with some limitations. Each account can only have five projects going at a single time. Additionally, each project can only have five sources. Each source is limited to 10,000 words or less.

The data is limited to Google Documents – or just plain text. I was hoping spreadsheets and structured data could be used, but I assume that’s next.

I decided to put NotebookLM through its paces. Today, I’ll share the good, the bad, and my conclusions about the product.

Pro: Excellent at Search and Summarize

I decided to put the system through a test using data it should find familiar: Alphabet’s quarterly earnings reports over the past five quarters.

After uploading the documents, NotebookLM immediately offered a summary. It also suggested some “questions to try” and other topics I could click on to get a response.

The system auto-summarized 2022’s first-quarter results despite having more recent financial documents available in the data set.

Next, I wanted to see if the system was capable of performing a fairly basic task of determining Google’s revenue growth.

It returned the correct answer of 3%.

That’s useful. But it’s also something that could have easily been found with a quick web search. In order for this product to really stand out, it needs to go beyond everyday web search capabilities.

And that’s where NotebookLM fell short…

Con: Lacking Imagination

It’s one thing for NotebookLM to pull data from a document. I wanted to see if it could also provide insights.

I asked NotebookLM, “How can Google improve its business?” The response wasn’t overly impressive.

The responses included:

• Investing in new technologies

• Expanding into new markets

• Improving its customer experience

• Partnering with other companies

That’s not exactly world-shattering insight. The responses were bland, generic, and lacked any specificity. Any human thinking about how to improve a business would already know about these vague ideas for growth.

While I wasn’t expecting the system to generate overly creative responses, the basic nature of the results leaves something to be desired.

What’s strange is that other generative AI are capable of much more creative “thinking.” There are famous examples of ChatGPT creating an entire business plan – including a business name and logo – from scratch.

So, why is Google’s NotebookLM so bland?

I’m speculating a bit here, but it’s possible Google is putting “guardrails” on the AI’s ability to be creative. Perhaps the company is worried the AI will say something shocking… Or perhaps it’s worried it will reveal something important to Google’s competitors.

I don’t know the reason. But let’s hope the AI improves from here. Otherwise, this product won’t stand out as a game-changing AI assistant.

Conclusions and Final Grade

If I were to “grade” NotebookLM, I might give it a six out of 10.

The AI was more than capable of providing basic summaries and finding answers to simple questions. That’s definitely useful. But it’s only moderately more useful than a traditional web search.

What I was looking for was creative insight. With all the experience and resources Google has, its AI system could only pull basic facts and figures. That means it’s a tool that can augment workers – not replace them.

NotebookLM isn’t going to replace the hard work of going through financial documents and coming up with deep insights. But it could save time digging for specific figures and summarizing key takeaways.

NotebookLM is a step towards a more personal AI assistant. It can’t do your work for you, but it can help you get work done faster.

This highlights a broader trend that I see happening where AI becomes more personal. As we plug tools like NotebookLM into our daily lives and workflow, it’ll save us time digging for answers. And the ability to feed it data that is current or relevant will become more common.

Regards,

Colin Tedards
Editor, The Bleeding Edge