Implementing G-Brain for a company — Part 2

In part one we discussed setup, however that’s not the entire story. Gbrain comes with out of the box recipies only for a few things. For example, hubspot, slack dont exist. These are all custom built for one our partners at yellowintelligence.ai. Further, in this part we discuss how to setup permissions, separate gbrains for different functions and how to keep it always refreshed. Then we discuss Gbrain’s out of the box evals that help make gbrain better.

Permissions

During ingestion you’ve already filtered out personal or non relevant messages. However, even authoritative company information may require restricted access within the organisation.

Gbrain breaks this up using concepts: 1. Source 2. Scope

Sources

Source can be anything such as sales, product, engineering, finance etc. Its the same Gbrain database, however split into teams. The granularity of the team can be determined at the time of setup. Do you want one sales source or should EU and MENA be separate. There are some nuances in this, that some may find limiting, but honestly I think its well opinonated to make companies more open and transparent.

1. Users can be configured to be able to read information by source. i.e. Nayna can read only from “Executive” source while Patrick can read from both Executive and Sales.
2. Access controls do not extend to per-user isolation within a source. Once Nayna has access to the Executive source, she can read everything in that source. Makes sense, since it is a knowledge graph and there may be backlinking across different topics and users. Introducing such a concept would break the brain.
3. “Agent writing” in Gbrain is the concept where through normal chat you can write or update a note in gbrain, “Update Gbrain that I had a call with the client and they asked me to send the quotation”. Gbrain will identify which note and what the update should be for such an instruction. This is calle a write. While users can read multiple Sources, an agent can be configured to write only to one source at a time. This is a limitation of Gbrain. So if we we have Sales, Finance, Executive as sources, and the agent is configured for Executive, then only those users that have write access to Executive can write into it. You can not give the Sales team write access to the Sales source.

Scope

Scope has basically 3 permission categories for users. Read, write and admin. Read and write are straightforward, Admin typically adds permission-management and source-level control, such as configuring access, users, or source settings. This is typically mapped to the company’s oauth, or in my case since our agent is Slack/Teams enabled it was mapped to their Slack IDs.

Custom Recipies

This is a tricky topic. Because recipies is not a one size fits all. Even the out of the box recipies are more for personal use and hence assume some room for messiness. I would strictly advise USE THE GBRAIN RECIPIES AS REFERENCE TO MAKE YOUR OWN. Organisations may share the same tool stack, but each has its own nuances on how it uses it.

I went through this iteratively i.e. I started off with a basic recipe for hubspot, one I derived by using Gbrain’s standard recipie as reference. But I did short dry runs, of one month’s worth of data. I visually looked at the notes and then decided the changes, and ran the recipie again on the same data. Finally I arrived at what I liked. Below are Hubspot and Slack recipies I built that you can use to get a head start in your process.

When evaluating the recipies, keep a sharp eye for whether the note is backlinking correctly. eg. If the deal has correct backlinks to a company, or if contacts have correct backlinks to companies and deals. This is important for the knowledege graph to be accurate.

Evals

When users ask the agent questions, and the agent looks into Gbrain for answers, then is it looking in the right notes? Thats what evals do. Gbrain comes with an out of the box eval framework that evaluates this exact thing.

The biggest challenge in evals is creating a test data set. The test set does not primarily contain expected written answers. It contains:

1. The question a user might ask.
2. The GBrain page or pages containing the correct information.
3. Optionally, the page that should rank first.

You first start with realistic question users may ask, and then for every question, manually identify the page or pages that should answer it.

This is easier said than done. Obviously you can get an LLM run to do this, however, how do we determine if the LLM has done a correct job. I made a few tries, however I’ve not been successful at getting an LLM driven test set that gave me an output that Im happy with.

This is somethign still pending from my end, and once Im done Ill update it here as well.