How Salesforce Built a Scalable AI Puzzle App in Six Weeks

Today on the Salesforce Admins Podcast, we talk to Jagan Nathan, Senior AI Architect at Salesforce.

Join us as we chat about how he built a Slack app for Salesforce’s “Million Dollar Puzzle” ad, where millions of concurrent users raced to solve riddles by chatting with Slackbots.

You should subscribe for the full episode, but here are a few takeaways from our conversation with Jagan Nathan.

The tech behind the “Million Dollar Puzzle” contest

If you were watching the big game, you might have seen Salesforce’s “Million Dollar Puzzle.” As soon as the ad aired, the race was on for contestants to solve puzzles by chatting with Slackbots to discover the location of a secret vault.

My guest this week, Jagan Nathan, built the Agentforce-powered Slack app that made everything possible. They needed Slackbots that could act as a conversational gateway for millions of concurrent users. And with a million dollars on the line, they needed to be sure it couldn’t be tricked into giving away the answer.

Even more incredibly, Jagan pulled all of this off in only six weeks. I sat down with him to find out how he did it and what he learned along the way.

Using AI to build quickly at scale

Just like with any project, the first step was to gather requirements—though it’s more fun when your stakeholder is the “Puzzle Master.” Jagan and his team needed a thorough understanding of what they were dealing with and how they could translate those gaming mechanics into the application.

The clock was ticking, and AI was pivotal for accelerating the development timeline. Jagan and his team used Salesforce Vibes for quick prototypes and wireframes to help them decide what to build. But the development team still needed to take these ideas the rest of the way. “AI cannot help us solve all of the architecture problems,” Jagan explains, “we have to have a human in the loop.”

With millions of concurrent users, they needed to do thorough testing—and quickly. Luckily, they could use Salesforce Scale Center to stress-test for performance issues and identify bottlenecks so they could be sure that when the ad ran, the app would work flawlessly.

Designing for security with a million dollars on the line

I know this might be shocking, but when you’re holding a contest with a million-dollar prize, some people will try to cheat. So Jagan and his team needed to make sure there were plenty of guardrails in place to make sure someone could win it fair and square.

They needed to start with a security mindset and think through any security vulnerabilities as they designed the application. The Einstein Trust Layer was crucial for toxicity detection and monitoring for anything malicious, like prompt injections.

Most importantly, they were very careful with what information the Slackbot knew and what it didn’t. It didn’t have the solutions to the puzzles, so even if someone managed to crack it, they wouldn’t be able to get very far. Instead, puzzle answers were always validated by humans.

There’s a lot more from Jagan about building for scale quickly, so make sure to take a listen. And don’t forget to subscribe to the Salesforce Admins Podcast so you never miss an episode.

Podcast swag

Learn more

Admin Trailblazers Group

Social

Full show transcript

Mike Gerholdt:

When you think about building on Salesforce, it’s easy to focus on features. But today’s admins aren’t just implementing features. They’re designing systems that have to perform, scale, and hold up under real world pressure.

In today’s episode, I’m going to sit down with Jagan, an architect who helped deliver a high stakes, high scale experience serving millions of users through a custom Slack-based AI system. We’re going to unpack what it actually takes to compress months of work into weeks and how to design for trust and guardrails from day one, and where human judgment still matters in an AI-driven architecture.

So if you’ve ever wondered how your role evolves from building flows to orchestrating full systems, this is the conversation because this just isn’t about speed. It’s about responsibility at scale. Now if you enjoy this episode, hit Subscribe, share it with a fellow admin and let’s get Jagan on the podcast.

So Jagan, welcome to the podcast.

Jagan Nathan:

Hi, Mike. Thank you for having me.

Mike Gerholdt:

Yeah. Well, I was excited when you sent me the Slack DM about some of the projects that you’ve worked on here at Salesforce. But for people that don’t know you or maybe didn’t listen to the episode of the Developer Podcast you were on, could you tell us a little bit about what you do at Salesforce and the fun project you got to work on recently?

Jagan Nathan:

Sure. Hey everyone. I’m Jaganir. I’m architect part of our AA practice team here at Salesforce, building agents for our customers. This is my third podcast on our channel. My previous episodes were focused on event monitoring and threat detection.

Mike Gerholdt:

Wow.

Let’s talk about some of the stuff that you got to work on around the big football game that happens in February that Salesforce had an ad on.

Jagan Nathan:

Yeah, sure. Very excited. This is our first ever online puzzle app, which was built entirely on our Salesforce platform.

Mike Gerholdt:

So you got to be a part of that. I’d love to know from a Salesforce admin listening to this, we used the platform for the puzzle. Can you give me a little insight into that?

Jagan Nathan:

Yeah, so let’s unpack this. So what we did is we didn’t just build a chatbot. We deployed our custom Slackbot, of course, powered by our Agentforce, which acted as a conversation gate for millions of participants grounded in real-time data. So we just collapsed a nine-month roadmap into just six weeks, which is 42 days. Within that, we designed iterator and we went live on day of February 8th.

Mike Gerholdt:

Wow. Nine-month roadmap. Holy cow. That’s insane. I mean, can you talk a little bit about the architecture problem solving and how you moved so fast so quickly?

Jagan Nathan:

Yeah, sure. Of course, we are a customer zero company. So we leverage our internal tools and the external tools, what we have developed. So for this entire application, we use our Agentforce Vibes to accelerate the development. We quickly do a lot of wireframes and the prototypes, and then we figure out what sort of user experience and what sort of applications we need to build using the power of Agentforce Vibes that accelerated our development timeframe to start with.

Mike Gerholdt:

Oh, that’s nice. I mean, I’ve definitely seen Vibes do a lot of things. I’m assuming we really took it out for a test spin on this one.

Jagan Nathan:

Yeah. So we use Agentforce Vibes to do this. And then we also leverage our scale testing product because this particular application, we anticipated there’s going to be at least millions of users who is going to access this application concurrently. So performance is super important for this application. So we leverage scale testing product to identify quickly and resolve all the problem bottlenecks as well.

Mike Gerholdt:

Yeah. Wow.

I’ve had some talks with PM for, was it Scale Center? Is that what you used?

Jagan Nathan:

Yep. Yes.

Mike Gerholdt:

Yeah. And we’ve put out a couple of blog posts about those. So for admins that aren’t familiar with what Scale Center is and what some scale testing helps them do, what did you use it for?

Jagan Nathan:

Yeah, Scale Test is the performance evaluation tool. So that gives a lot of metrics. Before customers used to run some automated performance testing to figure out what sort of performance bottlenecks they have in the ACRM instance. But with the power of Scale Test, we can write a test plan creation. We can do a test environment set up. And we can also simulate all those real-time performance challenges, and then we can schedule those test trend to see where exactly fails and at what threshold it fails so that we can understand the spike and then we can fine tune our application.

Mike Gerholdt:

Okay. That’s super helpful. I mean, was there any surprises that happened in your testing?

Jagan Nathan:

Oh, yeah. Since we anticipated there’s going to be at least millions of concurrent users, there’s going to be a lot of DB rights back and forth. And then we ran into a lot of issues. We were able to figure out with the help of scale testing, we figured out all those in the sandbox, and then we fine-tuned our code and the logic and everything to make sure it is scalable for concurrent users as well.

Mike Gerholdt:

That sounds like a lot. Holy cow. I don’t think I’ve ever built an application where I had to plan on a million people using it concurrently.

So you talked about Agentforce Vibes. I’m sure we leaned into a lot of AI to build this. How did you decide really what AI should do versus what humans should still control during the project?

Jagan Nathan:

Yeah, that’s a great question. So of course we can use AI, but AI cannot help us solve all the architecture problems. One of the architecture problems, we have to use human in the loop. For an example, when we started designing this application, considering the scale of users, we thought just by writing a prompt, it’s not going to solve the problems. Prompt is going to give, based on the request prompt is giving the response back, but that alone is not sufficient enough. So that’s where we use humans power to understand and architect well for this [inaudible 00:06:50] game.

Mike Gerholdt:

Yeah. Wow.

On top of that, you got to think of guardrails because I’m sure somebody was thinking, “Well, since Slackbot was in use, what if somebody just asked it, what’s the answer?” That seems like the most obvious guardrail to put in place, but how did you sit down and think about some of the guardrails to put in place for an experience like that?

Jagan Nathan:

Yeah. So there’s an aha moment here. So when we built this custom Slackbot, we intentionally built in a such a way that this custom Slackbot doesn’t know the puzzle answers. So even if someone tries to crack the puzzle Slackbot, they would not be able to get the answers out of it. So that is the first so that no one walks away with the puzzle answer, right?

Mike Gerholdt:

Right.

Jagan Nathan:

So that is the first thing what we did. And then the next thing what we did is automated toxicity detection at the hedge. So what we tried to do is we leveraged, of course, the power of Einstein Trust Layer and then it’s not like a traditional AI chatbot. So we just built intentionally a custom Slackbot with all those automated toxicity detection so that if the Slackbot finds some harmful response or some sort of response which is not supposed to do it, if the user on the other side, if the user is a bad actor trying to do some prompt injections or whatnot, our platform is able to monitor and mitigate the risk immediately. It’s being very proactive stage of monitoring.

Mike Gerholdt:

Yeah. No, I guess I just thought the Slackbot would have access to the answer and you’d just have to guard it really well. But if you set it up and never give it access the answer, then it can’t accidentally hallucinate and give the answer. So wow, that was probably a little bit of planning on your end.

So looking at what you built and deployed, if a Salesforce admin wanted to recreate some of the best learnings that you had from this, what were some of the key takeaways that you had?

Jagan Nathan:

Yeah. So I had that security mindset to start with. Let’s say you are a bad actor or else you are trying to act the system, what sort of loopholes you might find in the system and then find all those security vulnerabilities, why you designed the application itself. It’s like having a security mindset is super important. And that too for the scale of the cash prize is one million. So definitely there’s going to be a lot of participants going to crack the Slackbot or else they try to trick the Slackbot and try to get the puzzle’s data out of it. So I would say security mindset is super important. And then start with the understanding of, basic understanding of how prompts work and then try to build a Hello World app in Slack to start with.

Mike Gerholdt:

Yeah.

Did you have a whole team of people that were just, I guess it’s called negatively testing because there’s always positive testing, which is test it to make sure it does what it’s supposed to, but then there’s negative testing, which is to make sure it doesn’t do what it’s not supposed to, right?

Jagan Nathan:

Yes. So we did advisory testing as well. So of course, we used the power of AI. We built our own agents to try to craft this agent to see where it bakes. And then we try to enhance our prompts and all the safety guardrails to make sure. We did a ton of testing during our design and implementation phase just to make sure Slackbot is not supposed to provide any information which is not needed or which is kind of out of privacy zone.

Mike Gerholdt:

Yeah. Wow.

So given the scale of everything that you did, what is something that you maybe hit a limit with AI that you didn’t know would exist?

Jagan Nathan:

What we felt down the road is like, AI cannot solve all the problems for sure. That too for the scale of millions of concurrent users trying to access the system. We thought there might be places in which there might be some timeouts happening, but knock on the wood, nothing happened. We were able to build a scalable application.

Now, we don’t rely 100% on AI. We also added a human in the loop, for example, some sort of puzzle answer validation, mainly the final puzzle answer validation. We don’t just rely only on the AI. We use a human in the loop. So that we brought in experts on the puzzles who validated the answers and lot of stuff at the backend.

Mike Gerholdt:

Yeah, that’s interesting.

What was it like to have to translate a puzzle into a technical requirement? Because I feel like a lot of Salesforce admins often get business processes that can sound like a puzzle and they have to translate that into a technical requirement.

Jagan Nathan:

Yeah. Great question. That was very interesting and challenging because not all of our teammates worked on the puzzles or solved the puzzles before. If you take me as an example, right? I haven’t solved any puzzle in the past. I went through the basics of understanding what sort of puzzles exist in the real world. Some are logical puzzles, a lot more Sudoku puzzles, some are mathematical puzzles, some are geolocation puzzles. So for us, it took some time for us to understand the basics of what sort of puzzles we have. And then we had conversations with the puzzle masters and then who built this puzzle. We tried to understand from the puzzle master what sort of application they wanted to build because they know very much on the gaming side and we know very much from the technical side. It goes hand in hand.

That was very interesting and challenging conversations we had to understand. It took some time for us to understand and translate the gaming mechanics into our application.

Mike Gerholdt:

Wow. Okay. So be honest, how cool wouldn’t it be to have a title named Puzzle Master? Right?

Jagan Nathan:

Yes.

Mike Gerholdt:

So one of the last questions, because I bet you use Slack a lot as the project team to get things done. I’ve noticed as I work at Salesforce and we use Slack in different channels, I’ve noticed changes in behavior, ways that we work faster with Slack. Was there something you left this project with where you’re like, “Oh, I’m going to start posting this way in Slack or I’m going to start sharing information this way in Slack,” because it was something that actually was a way that you worked in that team.

For example, I’ve noticed that when we have really long threads rather than posting one big post that could take up a lot of screen, somebody will post a comment thread about a topic and then put all of that into the comments because then it keeps their feed kind of cleaner.

Was there anything like that that you kind of learned as like, “Ooh, I’m going to start doing this more often now in Slack?”

Jagan Nathan:

Well, yeah. So what we did is when we started working on this project, we had a 14 member team, few members were focusing on the technical side, few members were having conversations with the puzzle master. So we used Slack as a project management tool.

So what we did is we created a Slack Canvas with a list of team members and then what sort of priority task we need to work on. And then we automated to the next level by adding a Slack list as well. We created a list, like what is the roles and responsibilities for each team member? And we started tracking the list. And then every week we were running the Slack workflows.

We extensively used the Slack workflows. We created a ton of workflows, one workflows for summarizing the Slack channel so that what happened in this one week’s … Because see, we had at least six weeks to develop and build and deploy, right?

Mike Gerholdt:

Yeah.

Jagan Nathan:

So each week we were running the Slack summary workflows to understand what is the latest going on around them. And then we were extensively using Slack, mainly the Slack Canvas and then the Slack list to start with.

Mike Gerholdt:

Yeah. Wow. I love building workflows. I tell you, they’re really cool. I can almost build too many sometimes.

Well, I appreciate you coming on and sharing what you could about this. I think the amount of work that was completed for this project and in the amount of time is super incredible and I’m sure it was quite the learning for you. So it also teaches us that even the biggest projects we can still accomplish very quickly. So thanks for coming on the podcast.

Jagan Nathan:

Thank you for having me, Mike.

Mike Gerholdt:

So it was great of Jagan to come on and share his experience on that project. I think it’s a good reminder that building on Salesforce today means thinking beyond automation. It’s about really designing systems that are resilient, trustworthy, and ready for real world scale. From AI guardrails to perform testing to human in the loop decision-making, this is the work that the modern admin is balancing speed with responsibility.

Now, if you found this episode helpful, share it with another Salesforce admin who’s thinking bigger about the systems they run and making sure that you’re subscribed so you don’t miss what’s next. Until next time, we’ll see you in the cloud.

The post How Salesforce Built a Scalable AI Puzzle App in Six Weeks appeared first on Salesforce Admins.

Related Posts