As a principal investigator, a program officer is another research resource, but it is on you to use that resource widely. Jo Miller of JM Grants does an excellent job explaining the grant seekers role to that of a foundation’s program officer in Program Officer: Friend or Foe? Like their foundation counterparts, federal research agency program officers serve as gatekeepers to federal research funds. Most of my knowledge arises from my experience as a National Science Foundation (NSF) principal investigator, reviewer, and panel member.
Program officers at (NSF) are your scientific peers. Some of them are permanent staff and some rotate in on temporary appointments. Rotating program officers are really your colleagues on leave from their academic jobs. They understand the college and university environments of NSF grant applicants. They are all federal employees. Federal employees must follow very strict rules. Every time I read grant seeking advice about building relationships with program officers, I get scared. Federal program officers cannot get too close to a proposed project because of these rules. Appearances count. It was another agency, but I know of a program officer who was moved to a different area because of possible favoritism.
Program officers have discretionary power, but are not necessarily experts in specific areas. They must rely on technical reviews, panel member reviews, and the panel discussion for their funding decisions. Generally what happens in the panel room stays in the panel room, but this presentation from an NSF program officer summarizes the roles of the program officer and the different reviewers as well as how grants receive funding at the agency.
One of the best practices for a grant seeker is to always contact the program officer with questions about the guidelines, funding trends, specific problems, or even toss out ideas for feasibility. Just remember there are lines that neither party can cross.