But when it comes to confidentiality, having fine-grained authorization securing your RAG layer is the only valid solution that I've seen in used in industry. Injecting data into the context window and relying on prompting will never be secure.

In these situations a real person would just ignore them. But most LLMs will cheerfully continue the conversation, and potentially make false promises or give away information they shouldn't.

Prompting with clarity seems to help alleviate any accumulated response pressure where it's having to reach beyond what it has readily available.

When it comes up short, it seems to dig deeper and come up with more than intended, or over respond.

Jumping to solutions remains one of the biggest challenges.

Why does a single-step task imply a coinflip to you?

There are more than two possible choices for an instruction like: "Lookup the status of order X".

The same pattern continues for a couple of iterations until I get the correct solution.

The problem is, the llm responses are so slow that I could just work out the problem myself in the time (I typically ask questions that I know I can solve, it just takes too much time at the moment, e.g. Just yesterday I asked a question about some interlocked indeces, which I was to lazy to work out myself at the time).

Instead of the llms with increasing benchmark scores I want an llm that is of similar level to the current ones, but answers instantaneously so I can iterate quickly.

Not sure there is much of a real world takeaway from this.

One could read this paper as Salesforce publicly weighing their own reputation for wielding existing tools with competence against the challenges they met getting those tools to work. Seemingly they would not want to sully that reputation by publishing a half-baked experiment, easily refuted by a competitor to their shame? It’s not conclusive, but it is relevant evidence about the state of LLMs today.

Although it would still be interesting to know if they could hold on to secrets, even if they never should need to.

They published their code. If you have an agent you think will do better, run it with their setup.

The choice of test is interesting as well. Instead of doing CRM and confidentiality tests they could have done a “quickly generate a listicle of plausible-sounding ant facts” test, which an LLM would surely be more likely to pass.

If going against a datasource (like with Retrieval Augmented Generation), yes.

If the information is just part of the context window, no.

A team led by Kung-Hsiang Huang, a Salesforce AI researcher, showed that using a new benchmark relying on synthetic data, LLM agents achieve around a 58 percent success rate on tasks that can be completed in a single step without needing follow-up actions or more information.

and

The Salesforce AI Research team argued that existing benchmarks failed to rigorously measure the capabilities or limitations of AI agents, and largely ignored an assessment of their ability to recognize sensitive information and adhere to appropriate data handling protocols.

Edit: Unless "Salesforce AI Research" is not a part of Salesforce, I think Salesforce did do the research.