Our First Preprint: What We Learned from 2,660 Conversations

Physicians vs Nurses: what are they really looking for?

2,660 conversations with EMSy reveal surprising differences among emergency professionals

When we decided to analyze the data from our first prototype, we didn't know what to expect. Nine months of use, 46,000 messages, 2,660 conversations. Numbers that, translated into insights, tell a story we didn't anticipate.

The result? Nurses and EMS physicians ask completely different questions. This isn't an intuition: it's what clearly emerges from the data.

Who Used EMSy

Between February and October 2025, the EMSy prototype was used by emergency professionals spontaneously, without marketing campaigns. Of these conversations, despite being anonymous and non-identifiable data, we were able to identify the professional role in approximately half of the cases:

• 698 physicians (53% of those identified)
• 532 nurses (41%)
• 77 EMTs/paramedics (6%)

The growth was organic: colleagues talking with colleagues, links shared in shift WhatsApp groups, word of mouth among crews.

The Discovery: Different Professions, Different Questions

By grouping questions by clinical category, a very clear pattern emerged.

Nurses Seek Procedures

For nurses, 41% of questions concern procedures and operational protocols. These are practical queries, often posed in real work contexts and under time pressure:

• "How do I dilute norepinephrine?"
• "Intraosseous access: what are the criteria?"
• "Nitroglycerin for chest pain: always indicated?"

For physicians, this category represents only 5% of conversations.

Physicians Seek Understanding and Training

Physicians tend to use EMSy differently. 38% of their conversations fall into the General / Training category: clinical reasoning, quizzes, case discussion, theoretical deep dives.

They also show relatively greater interest in certain areas:

• Cardiovascular (15% vs 11% of nurses)
• Obstetrics (4% vs 0.6%)
• Trauma (5% vs 2%)

EMTs: An Intermediate Position

The EMT/paramedic sample is smaller, but the behavior is consistent. Their use of EMSy falls midway between nurses and physicians, combining operational requests (21% procedures) and training use (34% quizzes and training).

The Numbers Behind the Story

We didn't stop at percentages; we wanted to involve a professional from outside the healthcare world who could analyze and validate the data collected from the EMSy prototype as objectively as possible. Ilia Khashei, a student at Politecnico di Torino who conducted the statistical analysis, performed a rigorous statistical evaluation of the data.

In simple terms, these differences are extremely unlikely to be attributed to chance. The association between professional role and type of question is statistically solid and consistent.

Complete Distribution by Profession

What This Means for Training

These results have concrete implications for those designing clinical support tools and training programs in emergency medicine.

Nurses need immediate operational support.
Their questions reflect a strongly hands-on role in the ambulance: drug preparation, access management, protocol application. For them, a useful AI must be fast, clear, and directly applicable in practice.

Physicians seek continuous learning.
They use the tool to study, test themselves, and explore less frequent clinical scenarios. In this context, AI becomes not just a reference, but a true training companion.

The Silent Feedback

Another interesting finding: only 7.5% of users left an explicit rating (thumbs up/down). But among those who did, 97% is positive (577 upvotes vs 17 downvotes).

What does this mean? Healthcare professionals, under time pressure, rarely stop to rate. But when they do, the judgment is clear.

Interview with Ilia Khashei

Ilia Khashei, a student at Politecnico di Torino, conducted the data analysis that led to the publication of the preprint on medRxiv. We asked him to tell us what emerged from the work.

How did you come to work on EMSy? Did you already have experience in emergency healthcare?

I came to EMSy with a background in data analysis and machine learning, without direct clinical experience. I was particularly interested in the opportunity to study how a clinical AI system is actually used outside of controlled contexts. Working on EMSy allowed me to learn a lot about emergency healthcare through the data and thanks to continuous dialogue with clinicians.

Analyzing the data, was there a moment when you thought "I didn't expect this"?

The biggest surprise was the clarity with which usage patterns differed based on professional role. I expected some variability, but not such clear and consistent differences across thousands of conversations. It was one of those cases where the data tells a very strong story on its own.

One of the main findings is that nurses ask about procedures in 41% of cases, while physicians only 5%. How do you interpret this difference?

In my view, it reflects the reality of prehospital work. Nurses are often responsible for the practical execution of procedures and preparation of treatments in real time, so their information needs are strongly operational. Physicians, on the other hand, seem to use the system more for clinical reasoning, training, and exploring less frequent scenarios.

You worked on a system with very limited logging. What was the main difficulty?

The biggest challenge was extracting meaningful information from incomplete and imperfect data. It was necessary to be very cautious in assumptions, validate every step of the analysis, and clearly document the system's limitations. These constraints influenced the entire methodology.

What do you take away from this experience?

This project confirmed how valuable real-world usage data is, even when it's "messy." It also highlighted the importance of interdisciplinary collaboration when working on AI in healthcare: both technical rigor and clinical expertise are needed to interpret results responsibly.

Read the Complete Preprint

The complete analysis is available as a preprint on medRxiv:

📄 Retrospective Quality Analysis of a Clinical RAG Chatbot for Prehospital Emergency Medicine: Observable Signals and Lessons Learned

The paper describes the methodology, study limitations, and lessons learned for those building AI systems in healthcare.

Conclusion

2,660 conversations have taught us that there is no generic "emergency professional". Physicians, nurses, and EMTs have different information needs, and the tools we build should reflect this.

This work represents only a first step. The data comes from a prototype, with evident limitations. We chose to document them openly because we believe transparency is a value, especially when building AI tools for clinical use.