Better Prompting: The Skill That Unlocks Everything AI Can Do for You

Webinar introduction (0:07)

Irma Roman: Good morning and good afternoon, everyone. My name is Irma Roman. I’m the implementations manager at Streamlyne and I am very happy to host this webinar. This is the third webinar in the series and today it’s a very special webinar. We have our presenter, Tricia Callahan from Emory University. Presenting the theme Better Prompting, we also have our CTO, Janior Valle, who will be assisting Tricia and also summarizing our previous webinars for you.

Janior Valle: So I can go ahead and continue from here. Thank you so much for attending today. As Irma said, this is the third webinar in our series. We started doing these webinars back in June to help the research community learn better prompting skills, create reusable workflows, and in general come up to speed with AI. AI is a rapidly evolving industry. As someone who works with AI on a daily basis and helps develop new features with AI, every month there are new models; they get smarter. You have to use them to figure out which ones work best for you. We’re going to talk a little bit about that today, some of the basics, and some of the things that we covered in past sessions.

Janior Valle: We’ll go ahead and pass it off to Tricia, who will proceed with the better prompting session. If you have any questions, please put those in the chat. We do have a dedicated Q&A section at the end of the session. If there’s a question that comes up and we’re on that same topic where there’s a lull or pause, we’ll pose those questions; otherwise we’ll respond in the chat or leave it to the Q&A session at the end. This session is being recorded. After the call, we’ll share the recording with all the participants, so if you have to leave early or something comes up, or for those who weren’t able to attend and are listening after the fact, we’ll share it after the call.

Janior Valle: Okay. So let’s talk a little bit about some of the basics that we covered in prior sessions. As I mentioned, this is the third webinar in a series that we started in June, but in reality we’ve been doing this for over a year and a half. We started doing these at the beginning of last year at institutions around the country where we would give workshops to provide hands-on training with AI: how to work with AI, what the nuances are, how to prompt, and things to watch out for.

Janior Valle: Legal compliance, that sort of thing. We thought it would be a good idea to do this for the broader research community and not just at certain institutions, which is why we promoted this on the res admin and different areas as well. When it comes to AI, there are different types: vision, text, image, and audio. Every AI model has strengths and weaknesses, which is why I always recommend, if you have access to models, to try different types. There are ChatGPT, Copilot, Claude, Gemini, and many others on the market. Not all of them are the same.

Janior Valle: Some models are audio only. Some are image only. Some are multimodal, meaning they have multiple capabilities. I mentioned vision, image, audio, and text. Some models can do text and vision and image. Some can only do text and image. It depends on the model. For example, ChatGPT is a multimodal model: it can see images, it has access to text, it can create images, and it can transcribe audio. It uses speech-to-text and text-to-speech.

Janior Valle: Claude is also a multimodal model; however, it can only do text, image, and vision. It can’t do audio. Keep that in mind. We all have different workflows and processes that we use AI for. If you have access to various models, try the same task on multiple models and see which one works best for you. I find that experimentation helps you learn how to work with models more efficiently and helps determine which model suits which task. For example, maybe budget justification is better on one model than another. Experiment a little bit.

Janior Valle: I also want to give a privacy and compliance disclosure. Definitely double check with your legal and IT before you put any information into these systems. How these systems work—ChatGPT, Claude, Copilot—depends on whether you have an institutional license. If you have an institutional license, chances are you have zero data retention enabled, which means the AI provider can respond to your requests but is not allowed to train on the data or the prompts you input. Internally at Streamlyne, we use ChatGPT and Claude, and with our accounts we have zero data retention enabled. When we’re doing internal processes or development work, that data isn’t being shared with those companies.

Janior Valle: I highly recommend that before you put any information into these systems, you double check with your legal and compliance teams. A common question I get is, if I have an account and it’s not a free account—if I pay for that account—do I still have to ask IT or legal? The answer is yes. Often those free accounts and paid accounts don’t have zero data retention enabled. It’s important to double check because that’s usually tied to your institutional licenses and contracts.

Janior Valle: What kind of work does AI help with? Where does it fit in? AI is really good at drafting and editing, formatting, data translation, and similar tasks. Some of the ways I use AI include images—for example, this PowerPoint contains an AI-generated image. Many models are multimodal and can create images. When we do presentations, we use AI to help spruce up templates. Other things include development, formatting policies and procedures, and tedious drafting work. Be careful, because AI does hallucinate. AI is not a replacement; it augments. It won’t replace us, but it can help us do things faster. I always caution folks to double check AI-generated outputs because AI will sometimes make up information and go off the rails. A common topic that comes up is web searching versus prompting. That’s a good analogy to continue on, and after this we’ll turn it over to Tricia.

Web search versus AI prompting (10:58)

Janior Valle: With web searching, think about your typical Google search. You’re putting in keywords into the search box and then you’re getting a list of results. Those results are static. Google indexes a bunch of web pages and then, depending on the keywords and other thousands of factors, you get the results that are displayed on the screen. AI is similar to that in the sense that with prompting you’re not necessarily going to put keywords. You’re going to enter something in natural language and then you’re going to get a set of results. With AI, it’s more of an iterative approach. Typically that first prompt may not give you everything that you need, but then you can have a conversation with the AI to fine-tune what you are looking for. For example, if you’re looking through an opportunity announcement like an RFP or an RFA, maybe you start asking questions to the AI about what’s in that announcement and it gives you a nice summary but misses completely what the eligibility is. You can prompt AI again to tell you more about the eligibility and what’s involved there, what the sponsor is looking for. When it comes to prompting, keep in mind that it is an iterative approach. In my experience, you’re rarely going to get the exact output that you were looking for at the beginning; you have to iterate on it and, over time, the number of iterations will decrease. I remember when I first started using AI, I had to do a lot of back and forth with AI, almost exhausting. Nowadays, as I’ve improved my prompting skills, I find that usually within about two to three turns back and forth I can get what I need. I also save my prompts so that I can reuse them in the future. So again, I don’t want to bore you with the slides. We kind of covered this: web searches are static, prompts are iterative. You definitely want to refine, critique, and build upon those previous outputs. With that, I’m going to turn it over to Tricia Callahan.

Tricia Callahan: Great. Thanks so much, Janior. I really appreciate that. I smiled when you said it was exhausting, and that’s the whole point of today: to make it less exhausting, to talk about how to craft an effective prompt so that you’re not going through so many iterations. You going through the difference between web searching and prompting leads nicely into this conversation. When I think about web prompting, I think about it like an index search: I’m just putting in a keyword. You mentioned searching for a funding opportunity. Usually I’m going out to the web and looking for a specific funding opportunity. I’m typing in NIH parent opportunity or what have you just to find it, and then I’m going through and reading it myself. That’s completely different than what I would use AI for. With AI I’m thinking about generative AI prompting as a task that I am delegating to a highly capable, highly efficient research assistant. I would take what I found from the web prompt, take that funding opportunity announcement, and say, hey, could you go through this for me and highlight any deadlines, any budget caps, anything we need to be aware of to bring to consideration for the PI? Could you make a checklist for me? I would hand that task off to an assistant, but instead I’m handing it to AI. I think of AI as my highly capable, highly efficient research assistant. The information on this slide provides a framework that will help you move from that quick keyword search you would normally do when searching a web browser to a very intentional, goal-driven conversation. It really is a conversation. Hopefully when you leave here today, you can use this craft framework as that mental checklist to make sure you’re giving AI, whichever tool you’re using, the exact parameters to generate high-quality output so you’re not so exhausted trying to use these tools. We’re going to break it down. We have this crafty acronym. We’re not necessarily going to start left to right; we really start with the task when building, but we want to think about all of these things. C is for context: the background information that you provide your AI. I’m going to call it the researcher or service assistant. You’re establishing the environment, the situation, any constraints, and you’re telling AI why you’re doing what you’re doing and what constraints are applicable. For example, in my role at Emory University I help organize a conference for professional development to bring the Emory research administration community together. My context that I could provide my AI tool would be: I’m hosting a two-day or three-day professional development conference for busy research administrative staff. These staff are experiencing fatigue and screen fatigue because we’re all virtual and we don’t get to interact in person. I’m telling it some of those constraints because if I want to use this tool to help me build an agenda and program, I need it to understand what I want to get out of this program. The next is the role. This is the persona: assigning the specific identity or expertise level to AI, which will help shape the tone and perspective. For example, act as an instructional designer or act as an event organizer. “AI, act as an event organizer and provide me with the agenda for a two-day conference for research administration professionals who are working remotely and don’t get to come together often.” It sounds like a lot, but putting that in there sets the stage and directs AI. Then the audience: who is the end user who will use the final output. Sometimes the role and the audience are the same; sometimes they’re different. This will help set the language, the tone, and the complexity. For example, tell AI that this conference is for adult learners in higher education who are research administrators. Define the format: this is the format for the output you will get. It is the structure; you are telling AI what you want. Do I want an agenda, a bulleted list, an executive summary, a step-by-step guide, or a 30-second presentation? Be very specific in your format. If I was using AI to help put a conference template together, I would ask for a table with columns for an activity name, the duration, what the setup looks like, associated costs, and benefits to the participants. Tell AI the format of the output rather than just getting some bullet points. Sometimes you’re not looking for bullet points; sometimes you’re looking for a checklist or something you can use in an email. Sometimes I’ll say without using bullet points or in paragraph form. Once you get an output, you could look at it and say, oh, this was useful, but I could really use some bullet points or a checklist. You could tell AI, that’s a great job; however, can you turn this into a checklist? Even better is when AI gives you suggestions: “Would you like me to create a template?” and I’m like, yes, please do that. Or “Would you like me to create a PowerPoint?” Yes, please do that. Then you can build on that output. The task, even though it’s at the end of the acronym, is where you start with the very clear specific action you want AI to perform.

Prompt framework example (21:10)

Tricia Callahan: And it’s usually in the form of a command or it could be an open-ended question. So I might say, create a template for me and then with all of these other parameters that I just went through, the context, the role, the audience, the format, and that includes I might say I want it to include two general sessions in space for four concurrent sessions. I might want it to include four interactive breakout sessions that encourage active participation because we’re there in person. I can really help define that task and then with that you get that final prompt that’s really going to set you up for success and hopefully make you not so exhausted. What I’m going to put in the chat here is using this framework, here’s what my prompt might look like. Right, I might add that, and I’m not going to read that to you. You can read that in the chat. But what we’re going to do is actually go to the next slide, where the next slide is where we have some research administration specific examples, and we’ve gone through some good, better, and best examples using this craft framework. So, in the example here, we are analyzing funding opportunities. Remember, a web search is just to find it, but now we want to take that funding opportunity, we want to do something with it. We want to analyze it, we want some kind of output that somebody could use. So we start with this framework and we start thinking about the context that we need to give it and we start thinking about the task we want it to do. So our task could be create a checklist. Based on what is the content. The content could be that NIH funding opportunity that you found through your web search. So, you see that a good prompt has your task and it has the context and I think the context is so important so that’s why I made sure I included that in the good. We can make it better by going back to this framework and then the better example here now includes a role. Act as a departmental research administrator, which typically has different roles and responsibilities than, say, a central level research administrator. You could even be more specific and say a departmental pre-award administrator, and then you’re letting AI start to understand who that persona is. This is a person who understands sponsor requirements. This is a person who understands internal deadlines. It’s somebody who understands budget applicability and cost principles. You’re feeding it that information because it’s out there in the background learning what a typical pre-award departmental administrator’s typical roles, responsibilities, and knowledge are. So we’ve made our prompt even better by adding on to the context, adding on to our task, by providing it with how it’s going to act, what the role is. And then we can make it best if we include all of the elements that are shown in this framework. So our final, our best now includes the audience, who is the output going to be for. In this case we’re defining the audience. It’s a principal investigator. This lets AI know that any response that it gives, the output that it gives, should be tailored to an investigator who’s preparing to submit a proposal to this particular funding opportunity that we have provided it in the contextual information and then we’ve made it get even better again by including the format we want: a checklist and in that checklist we want to see included specific subsections that we want because why get that information and then start adding to it when we can just ask AI to do all that work because it can do it so much more quickly and efficiently than we can. With that I want to pause because I did see a question or comment. I was wondering if a great job is helpful. I can say that I have just heard from some experts and read at least one article on how it can be helpful to use the please and thank yous and the good job. Good job tells whichever AI that you’re using it’s on the right track. Sometimes AI will have the thumbs up or thumbs down. I actually was using AI earlier and it didn’t give me what I wanted. I gave it a thumbs down and it was like why did you give me a thumbs down you didn’t give me a good job and it gave me a list of alternatives. It was like the output wasn’t what I intended; it gave me some different responses to choose from. Good job—it’s on the right track, right? Norris, you’re nodding your head. Is there anything you would like to add to that?

Reinforcing AI responses (26:31)

Janior Valle: No, you’re absolutely right. It is a good indicator for the AI to latch on to the positive outcomes. When you’re telling it that it’s doing a good job, it provides reinforcement to the model to know, okay, I’m on the right path. I also want to mention, I really loved what you mentioned about the constraints and the context. I myself am not a research administrator. I’m a developer and I build a lot of research administration applications, but I find myself even using a lot of the same framework. Context is key. You have to tell it in what environment it’s operating in. I especially love the constraints because it’s super important to provide what is the expectation for what you’re expecting to be output and what are the requirements around that output. Thank you, Tricia.

Tricia Callahan: Oh yeah, you’re welcome. Context is everything in research administration and everything in training. I wish we had more time to even train context in that contextual information. That takes a while, but it doesn’t take too long with AI. You can feed it that context. You can give it that history. You can give it that backstory so that way it’s oriented and it understands the ask a little bit better. Have you as a research administrator ever had somebody just ask you a question without context? What’s our standard answer? I want to see all of these come in. What’s the standard answer in research administration to just about every question that we get? Yep, right. It depends. And what does it depend on? It depends on the contextual situation. Just like we need to provide that context when we’re asking questions of each other, we need to provide that context when we are interacting with our AI tools. Before we move to the next slide, on this one I showed you an example of a good, better, best. On the next slide, we’re going to walk through another good, better, best example. Now I’m going to need your help in identifying the craft elements in each of these prompts and what makes them good, better, and best. So if you would please move to the next slide, I would appreciate it.

Tricia Callahan: In here, we’re going to analyze a notice of award for key project and compliance information. Under good, we have context, role, and task. So I’m going to start with the task. What is the task? What is our ask here under that good example where it says act as a central research administrator? Over there on the left in the orange, if you could type into the chat what you think the task is. Yep, Beth: create a list. And I see some others coming in after. Let’s create a list. Exactly. Create a list of key project and compliance information. That’s the task. It needs a little bit more information. In our good prompt, we provided a role. What was the role that the AI is supposed to take on as it creates this list? Who is it thinking like? A central administrator. Again, all of the knowledge of the central administrator would bring to reviewing or analyzing a notice of award, which could be different than if an investigator or a unit-level or department-level administrator was looking at it. You could even put in, if you have role and responsibility matrices at your institution like we just developed at Emory, which I’m so excited about, you could even put in here’s my role, here’s the hat I play, here’s what I’m responsible for. You could provide it that information and really beef up and not let AI guess what the central administrator does at your institution. It’s looking across a huge body of knowledge; you can provide it that information. And then what is the context here? What piece of information have we given this in our good prompt? We provided a sample NOA, whether you’re providing it a link to an NOA or you’re actually attaching that NOA. We provided MOA notice of award. We provided that sponsor’s award document. This goes back to the beginning where you were talking about careful what you’re putting in and understanding your institution’s rules and regulations.

Prompting best practices (31:20)

Tricia Callahan: Do I want to be out there in something that’s not under MOA? Emory’s domain and protected under Emory and putting out a notice of award just say in Gemini or Claude or some kind of open source probably not; there could be some confidential information, so make sure you really understand what your institution’s parameters are as you start interacting with these tools. Okay. Then in our better response we have added in the audience. Who is the audience in the prompt that’s in the blue under better? Who is the output intended for? Okay, great. So our audience is the PI. We want to make sure that we are analyzing the funding opportunity but we’re pulling out those key project details: it could be project dates, it could be reporting dates and compliance information, to make sure that we are highlighting those appropriately for our principal investigator. Great job. And then finally we have best there in green. What is the specified format that we’re looking for our output in? In that longer little bit better-best, what are our formats? What are we looking for it to generate? A clear, structured summary, right? A clear, structured summary of key project and compliance information. We even said some elements that we want to see in there: deadline dates, compliance information. We gave it a little bit. Now looking at this best prompt, I think we can always get better. So how might the format of this prompt be improved? Just that format piece. Because we told it clear, structured summary — how could we make it even better? What additional specifications? Bullet points? Exactly. Bullet points or checklists? Ask for a specified size. Right. We could be very specific. Do we want a table? Do we want an executive summary? Do we want a checklist? The really great thing about prompting is organized sections. The great thing about prompting for me is the fact that it’s iterative; we’ve used that word before — iterative, meaning we can refine, we can build on that output. We can tell AI, great job, you’re on the right track, or sometimes I actually will open up a whole new kind of interactive window. I’ll just call it a chat window if it’s really going down the path and maybe I let it down the wrong path. It’s going down the wrong path and I need it to stop looking back at what you’ve produced because that’s not the direction I want to go. I will either ask it to forget or oftentimes I just open up a whole new window and start from scratch. The point is you can improve on the output each time. You don’t have to get it right on the single try. When I put this together originally and I was thinking, oh, I did a good job, me with my craft here, my prompt — I was like, you know what, my format was a little vague; that could be better. Once I got information back from that prompt I probably would have looked at it and said, oh, okay, this is good information; however, now can you build a checklist or could you draft an email to the PI relating all of these elements to it, maybe add deadline dates or what have you to it. So much we can do once we build that initial prompt. While I think CRAFT is a really good framework, we need to make sure we have these elements. One thing I have noticed from users is getting a little bit too long and giving it too much up front. Remember, you can iterate. Remember, you can add, you can refine later. Sometimes, and I’ve definitely been guilty of doing this, I write this really long, drawn-out prompt; it has all of my elements and then some and then I’m like, really? From all of that that’s all I got out of it because I realized I gave it too much. AI will forget if we start making our prompts too long; it may just pay attention to some of the first information you put in or last information, just like we do as humans. We remember information presented to us early on or later; there’s that primacy and recency effect. I think we experience AI falling under those same effects and then it forgets some of the other stuff maybe in the middle and we don’t get what we want out of it. It’s definitely an art to crafting these prompts, and once you find that sweet spot of your prompt, save them. Save those templates. We have a QR code that we showed at the beginning; we’ll show it again at the end where we have actually created a couple of templates for you which you can even take and improve on. A good rule of thumb for length — that’s a good question. I don’t know if there’s a good rule of thumb for the length as much as making sure you get in the elements that we’ve introduced here: provide some context, start out with the basic context that it needs, the role, audience, and the format. You might even leave off the format if it’s starting to get long with contextual information and then later go back and use that as your iteration and prompt it to give you a different format because it already has that output. Make sure the output’s what you want and then just have it reformat. So Janior, would you answer that differently or how would you answer that question too as far as prompting?

Janior Valle: That was perfect. One thing that at least I’ve found over time: start off with the CRAFT framework. Try not to overload the AI with too much information. As someone who builds a lot of these systems and works with a lot of these models, AI can be thought of as really powerful auto-complete. If you give it too much information, it may go off the rails and give you outputs that you didn’t necessarily want. When you’re following the CRAFT framework, be specific, exactly like what we have on the screen here. You can include additional information, but that would not be my go-to on the first round. You’re going to hear us say the word iteration a lot on this call, but it is really important to iterate on these prompts. As I’ve been working with a lot of these systems over time, my prompts have gotten smaller in some cases and in some cases have gotten larger, so it very much is trial and error and just seeing what works. As a starting point, start off with specifics like what the CRAFT framework is recommending, and then if you find that you’re iterating too much, start adding more context or more requirements or what the format should be, instead of just starting off with a huge prompt up front.

Tricia Callahan: Yeah, I agree. You can always get that format at the end, but it has to get the output right because it has to know that content, context, role, audience. Jennifer, I think you’re totally right. It’s important to keep the AI tool focused. It can drift. It can hallucinate. Sometimes it’s a big left-hand turn; that’s not what I expected. Sometimes it’s because I look back and see why it did that because of the words I used. Words matter a lot. Really stop and think before you start interacting with the tool. One of the earliest times I ever worked with an AI tool I asked it to draft a script; I don’t even remember what it was. I remember saying draft a script and all of a sudden I got this acting script out of it and I just started laughing. I was like, oh, you gave me exactly what I asked for — I asked for a script, you assumed I meant an acting script, and I didn’t want a script. Words matter. Oftentimes when I find it’s drifting, it’s because I’ve put something in there. Remember that it remembers. If it does start drifting and you’re trying to get it to stop that drift, you may just need to start from scratch because sometimes it does forget and other times it will hold on tight. Even when I ask it to forget, it’s kind of like that person the judge tells, “strike it from the record.” They can’t forget; they heard it. The same thing with AI. I love that somebody in here asked, can I use AI to help me write a good prompt? Absolutely. Give it a whirl. You could ask it to do that. You’re going to have to provide the elements of a good prompt to get it to give you a good prompt. So what is it you’re wanting a prompt for?

Building production-ready prompts (41:24)

Tricia Callahan: You have to provide that contextual information. You might provide some of the elements here in this craft model, too, to make sure it builds in some of those elements. So you can definitely help it refine your prompt as well. So I like that. Can we have an AI prompt our prompts? I think we’ve answered questions and identified what’s in the chat.

Janior Valle: Mm-hmm.

Tricia Callahan: With that, on this next slide, now that we have covered the craft framework, we’re going to put it into practice. I invite you to think about a task that you do daily. We are going to build a production-ready prompt together and then we’re going to pull up an AI tool and test our prompt out. In the chat, think of one routine or time-consuming task in research administration that you would love AI to assist with, whether you’re in pre-award, post-award, compliance, department, central. Just think of a task because we have to start with a task and pop that task in there. We’re not going to be able to do all of them, but we’ll pick one. If you can’t think of one, we can always go reviewing contracts and agreements: justification, compliance, post-award reports. I wish we had time to go through all of these. You can even work on your own if you don’t want to work on the one that we pick. Thank you for putting some of those in there. Let’s start with reviewing a contract or an agreement. That’s the one that was put in there first, so we’re going to go with that one: reviewing contract agreement. I’m going to highlight it so I don’t forget it there. Now give me a role. So we’re reviewing this contract or an agreement, but are we a PI? Are we a central administrator? Are we a pre-award administrator? Are we a department-level administrator? Who are we? We are a central administrator, so we have so far that we wanted to review an agreement. I’m building this in the chat as we go, so bear with me. I see central pre-award research administrators in the chat. Here’s what we have so far. That should be review. We’re giving it a little bit of a command. We have to provide it some context. Is it just the agreement? I understand you all don’t have an agreement to put in. So let’s say review. How about we say we’re going to attach a prior approved agreement. This is prior approved, so we’re not at the review approval stage. That’s providing context. Prior approved and attached agreement—you’re giving that contextual information. I see other things coming in on compare and contrast. Who is the audience? Some suggestions: a billing agreement; summarize the terms and conditions and highlight the most significant ones for a public university. Be very specific on the ask. The public university starts getting into that audience piece. Even more specifically, who is the audience? Is the audience the same as the role? Are we developing something for that pre-award central administrator to use? Are we developing something for a department administrator to use, our investigator to use? We can be even more specific on that audience. The only thing is we’re not going to have an agreement that we can put into the system to test this. I don’t have an agreement stuck in my back pocket to do this. So review an agreement: here’s what I have so far—review an agreement as a central pre-award research administrator. Summarize the terms and conditions and highlight the most significant ones for a public university. You can have additional audiences, but we don’t want to put too much in here. Think up front about who will be using the output. You could start broad with public institutions and then, if you want a version for your investigator versus a department administrator, use iteration and you might come out with slightly different tools and foci depending on those differences in iteration.

Janior Valle: If I can just share that.

Tricia Callahan: I like that. Please do.

Janior Valle: What’s worked well for me with two or more audiences is to stick with one audience to start. Start with the broader audience, the more general audience, and once I have a good output that works and I’ve locked it down to exactly what I need, I ask the AI to make another pass with a different audience. If I had to think about this internally—I’m a developer of a lot of these research admin systems—maybe I have a memo that needs to go out to my development team; that’s a broader audience. I would draft that and finalize it. But maybe I want something for my executive team and that’s a totally different audience. Once I finalize that output, I’ll ask AI to do another pass at it with the perspective that the audience is going to change, and that will change some of the outputs and the formatting. That’s what I do and what tends to work for me, so I wanted to share.

Tricia Callahan: Great. Thank you for that. I appreciate that. I’m going to put what we have in here so far: review an agreement as a Central Pre-Award Research Administrator. Summarize the terms and conditions and highlight the most significant ones. A contract reviewer at a public institution would be concerned with that. We probably want to make some refinements and clean it up a little bit. What we don’t have yet includes the format. What do we want to get out? We’ve just asked it to review it. It might highlight some things, but do we want a checklist? I saw someone suggest compare and contrast; I really like that. If at your institution you have a list of typical clauses agreeable to your institution, maybe alternative language, put in your agreeable information and ask the AI to compare and contrast with the agreement. Show where there’s a discrepancy. Once discrepancies are identified, provide alternative language using our institution’s boilerplate information and iterate to improve the output. The one thing I hadn’t asked for yet was format.

AI best practices for research administration (51:32)

Tricia Callahan: But again, that’s where I might stop and not put in the format. Maybe I would ask it to create some kind of list that highlighted or to go through and highlight a document that I’ve put in there. Great job. I can’t really pop this into an AI tool to see how good our prompt is only because I don’t have an agreement at the ready, but I think you all are understanding how this works, what’s working. Take this, copy it, use it on your own, tweak it, use it on your own. If you have an agreement that you can put into your institution’s AI tool, see what it comes out with. See if your tool gives you additional prompts like, would you like me to create a compare and contrast document? Again, that’s where I would absolutely love for you to do that because sometimes it thinks of things that I didn’t think of. I appreciate you all working through this framework with me for one of the tasks. I apologize that we couldn’t do all of the tasks that were there. If we’re ever together in a conference setting we can do some group work on this. With that, I think we will then move on. Let me make sure—oh, great. Yes, Heather, you are welcome.

Tricia Callahan: Let’s go ahead and go in here to the last few minutes of the presentation part before we move more formally into Q&A and just talk about some of the best practices. We invite you to share your best practices. I’m really glad that you’ve shared yours as well. Some best practices that I have found just in working with different AI tools: really deciding on that output, stopping to think about if you were handing this task off to a research assistant, everything that you would need to tell it. Tell your research assistant; same thing here. Words matter. You want to be directive.

Tricia Callahan: You want your researchers to consent to having that plan, what it is that you’re expecting from them, what it is you want them to do. You don’t necessarily have to start over if it’s going down a path that you don’t want; maybe you led it down the wrong path or it just took a wrong turn. You don’t necessarily have to restart. Provide critiques and start doing that iteration. Take the follow-ups. I’ve mentioned that where it makes some suggestions and I’m like, I love it. I didn’t even think of that. Please do that. Save your prompts. Sometimes when I write a really good prompt I’m like, I have to save that one because it took me a while to draft it and to get it exactly the way I wanted it, so I will save the prompt in the prompt template. The most important thing for me on this slide, and we’re going to talk even more about it on the next slide, is really to fact check it: take everything that is produced and look through it. Sometimes I’ll take parts of it and leave parts of it depending upon the situation that I’m using it for. Before we move into that more human element and critical review, is there anything that you would like to add or any comments in the chat on any habits or things that you have found helpful when you’re working with these tools?

Janior Valle: One thing I can mention: we talked a little bit about hallucinations earlier. It’s definitely something you should watch out for where AI is just making up information. The models have gotten a lot better over time. Hallucinations nowadays tend to be because of a lack of context or, again, going back to what Tricia indicated, words matter. If you find yourself in a situation where the AI is making up information or generating things that you didn’t ask for, chances are you need to add a little bit of context and iterate on the prompt.

Tricia Callahan: Great. Thank you. We can move to our next slide before we move into the Q&A. That human review is so important. Make sure you’re owning that final product. Make sure that you have reviewed it for accuracy, for compliance with any of your institutional rules and regulations, sponsor rules and regulations. If you’re using AI to draft a component of a proposal, let’s say a budget justification, are you even allowed to do that under the sponsor guidance? Are they checking to see that components were written with AI? You don’t want a proposal thrown out because you’re not in compliance with your sponsor’s requirements.

Tricia Callahan: Make sure you do that human review. Address hallucinations, that incorrect, fabricated, or misleading information that can be presented as fact. You read it and think it’s right, but it’s not. I use AI a lot in my training development. We’ve created a tool here at Emory University led by my boss, Lisa Wilson, where we have a bot that we have fed and educated on Emory University policy and procedure. Every time I ask it to generate content for training it provides references: where did it pull this information from? I look to see are the references accurate, are they up to date, was this a policy that was in place 25 years ago and there’s a newer policy but for some reason your bot didn’t pick that up. Always make sure you’re looking at that information. I love that at Emory we use Copilot which is associated with our Outlook tools. When I draft an email using Copilot it reminds me before I send it that it was drafted by AI and it says, have you looked things over before sending?

Tricia Callahan: It’s a good reminder. I don’t want to just hit send. I need to double check to make sure everything in it is the way I need it to be. We talked about long conversations: if you put too much in, it can forget steps. Use iteration to restate and refine. With that, I want to turn it over to see if you have anything else to add to that human element, making sure we stay in the loop before we move to Q&A.

Janior Valle: No, I think we covered everything. This is a good time to mention the session resources on the screen. It’s a QR code you can scan with your phone. It has a lot of example prompts that you can take back with you and share internally with your colleagues, departments, and teams. With that said, we can start taking some questions.

Janior Valle: You’re welcome. I believe we have the ability to come off mute. If you want to come off mute and you feel better about that than typing, by all means, please.

Janior Valle: So our first question: is there a prompt directory for the RA community?

Janior Valle: Tricia, do you know of any directories out there that share prompts for the community?

Tricia Callahan: I do not. I know there is a collaborative community under the National Council of University Research Administrators. I don’t know if they have a prompt directory. It’s something I can try to look into. I know our institution tried to build a prompt directory, but that would be institution specific. That might be something to put out there if you’re part of that research admin listserv: ask if anybody has seen a prompt directory. If not, what a wide open opportunity.

Janior Valle: That’s a great idea. I see people ask questions on the ResAdmin community listserv and everybody jumps in to help. I’m sure folks are more than willing to share what they are using and what works for them.

Tricia Callahan: I see we’re getting a lot of thanks and some people having to go, which is completely understandable.

Tricia Callahan: Are there any other questions from the community? You’re welcome. I think you probably have our contact information if you need to follow up directly. Happy to try to answer questions. If I don’t know the answer, I’ll ask AI.

Tricia Callahan: Yes, you can tell it output. You can say whether you want a spreadsheet, document, or presentation format. Sometimes I’ll ask it to create training slides, and because I use Gemini it will create them as a Google Slide presentation. You can definitely specify spreadsheet, document, or presentation format. Good question, Kurt.

AI prompting and thanks (1:02:04)

Tricia Callahan: AI prompting an RA AI prompting bank would be amazing, wouldn’t it? And we thank you all for being here with us today, too. Appreciate your time and your input and your participation.

Janior Valle: Definitely. And just a reminder, we did record the session. We’ll go ahead and distribute that after the call today. It’ll take a few hours to get the recording situated, but we’ll make sure we share that along with the slides so you’ll have that available to you.

Tricia Callahan: Well, I also want to thank Streamlyne for the partnership. I’m really excited to be able to partner with you and I’ve been excited to partner with you over the years on other initiatives. So thank you.

Janior Valle: Yeah, thank you, Tricia. We really appreciate it. This was great. I even took some notes myself. I learned a few things.

Irma Roman: All right. So I think with that, we can conclude for today. Once again, thank you to Tricia for being an excellent resource. Thank you, Jay. And thank you to everyone for attending. We will see you in the next webinar.

Janior Valle: Thank you, everyone.

Tricia Callahan: Thank you.

Irma Roman: Bye-bye, Gubbin.