The Innovation Horizons Project – An Organizational View on Fostering a Culture of Innovation and Entrepreneurship

As the Executive Director for Innovation at the U.S. Department of Health and Human Services (HHS), I have learned a great deal about large institutions, whether governmental, profit-driven businesses, or non-profit entities, that thrive on well-honed, routine management processes. I have also seen how bureaucracies serve these organizations well by bringing consistency, predictability, reliability, and stability into the important work that they do. These elements, for want of a better term, are what makes these institutions hum.

External and internal forces continuously challenge these routine management processes and our leaders are at times inadequately positioned to respond to them armed only with the outdated approaches of the past. Increasingly, leaders are looking for countervailing and transformational forces that their organization can harness and apply to overcome these new threats to their continued success. Often they ask, how can we test and adopt new business approaches that are needed to create new solutions, improve performance, and, sometimes even, retain our relevance?

This is at the heart of the issues examined in the Innovation Horizons Project – Fostering a Culture of Innovation and Entrepreneurship that the HHS IDEA Lab is publishing today.

The Innovation Horizons Project (IHP) was undertaken in the summer of 2016 to assess innovative management options for organizations to consider in promoting attitudes and mechanisms that enable new ideas to be tested and results to be embraced and enacted quickly.

The backdrop for this project was nearly eight years of activity within the Office of the Secretary of HHS. Our aim was to promote and advocate for innovation and our focus was to bring entrepreneurship practices to government.

In a massive government agency, I focused on creating a culture or system that encouraged and rewarded employees who responsibly embraced risk and experimentation. These pioneers were armed with new solutions aimed at enhancing the performance of our mission and protect the health and wellbeing of all Americans through HHS programs, projects, and policies.

Our thesis is designed as a look into the future for organizations undertaking an innovation agenda so they can understand critical success factors in achieving culture and process change to improve performance. Through an open, self-examining lens, this report shares experiences in management challenges that leaders should consider embracing to take innovation head-on:  top-down management driven versus employee-initiated approaches; innovation connected to strategic priorities or driven by opportunity; and ultimately, the question of how much value their innovation agenda brings to their organization in terms of results.

There are important new findings revealed from this report toward the question of how culture can be created and sustained. Successful programs rely on strong leadership presence and expressed values that are presented with clarity and specificity for all employees to embrace. The importance of metrics and linkage of projects to program outcomes and values is paramount to organizational acceptance.  Empathy – the ability to perceive change through the experiences of all stakeholders – must be valued and practiced at every step by those leading change. And, not only to acceptance of failure as one manifestation of risk-taking innovation practices, but in addition, setting an organization’s expectation of failure as a necessity for growth and impact.

My desire is that this thesis inspires more conversation about the culture of organizations and enables others to take their first step on their innovation journey.  I invite you to share your comments and suggestions in an open dialogue.

Finally, it is my duty to thank the hundreds of contributors to this work, and the leadership of Todd Park, Bryan Sivak, and Susannah Fox in their roles as HHS Chief Technology Officer who inspired it.

How Cross-Agency Collaboration Can Work for the Greater Good


photo of puzzle pieces being put together

by Jeffrey S. Reznick and Brett Bobley

Jeffrey S. Reznick is the chief of the History of Medicine Division of the National Library of Medicine.

Brett Bobley is the chief information officer for the National Endowment for the Humanities and the director of the NEH Office of Digital Humanities.

Note: The original, full-length version of this article appeared in the July 2016 issue of The Public Manager, a unique digital publication published monthly by the Association for Talent Development about federal government leadership that works.

Collaboration between like-minded federal agencies doesn’t make good sense, it makes great sense—strategically, financially, and managerially. A synergistic collaboration helps agencies:

  • achieve their respective missions and serve the public effectively
  • leverage their resources and use them efficiently
  • foster creativity and teamwork
  • expand learning and professional networks.

Yet, as many of us working in federal government realize, something so simple in concept and function, is often quite difficult to execute successfully.

We’ve been able to find collaboration success through our work at the National Endowment for the Humanities (NEH) and the National Library of Medicine (NLM), by combining professional networks, advancing interdisciplinary research agendas, and making more widely known and publicly accessible a variety of research resources. How did we do this? What elements make up a successful interagency collaboration? Read on.

The Opportunity

Interagency collaboration can begin in different ways, mean different things, and involve a variety of activities, a report from the Congressional Research Service explains. Ultimately, the character of any collaboration depends on how it develops, why it exists, and who is working to achieve it.

The NEH is an independent grant-making agency in the executive branch dedicated to supporting research, education, preservation, and public programs in the humanities and in those social sciences that use humanistic methods. The NLM is the world’s largest biomedical library, also an executive-branch agency, and a unit of the National Institutes of Health of the Department of Health and Human Services.

Through conversations started via mutual connections, we learned about the work being done in our respective organizations specifically within the departments we oversee: the History of Medicine of the NLM and the Office of Digital Humanities of the NEH. From there, we established a joint interest and willingness to build something together under the well-known premise that two (or more) minds are greater than one.

Our key next step involved delineating clearly the complementary missions and interests of our agencies. This effort helped us define the roles of our agencies in a collaboration, how each could support the other in mutually beneficial ways.

The Office of Digital Humanities of the NEH supports research, education, collaboration, and interdisciplinary learning related to the broad and lasting impact of computers and technology on our culture and society. Its annual Digging into Data challenge is a unique program that supports humanistic inquiry into how digital technology is changing our world. Why, then, collaborate with the NLM? Primarily because it was a means to reaching a community of researchers who, as they study the history of medicine, are increasingly interested in methods and tools of the digital humanities.The NLM is a leader in the application of computer and communications technology to the advancement of health and improvement of the human condition. Its History of Medicine Division embraces the future as a steward of the past by documenting the history of human health and disease through a collection of traditional and 21st-century materials—from books to blogs, manuscripts to videos. Why, then, would it collaborate with the NEH? The methods and tools of the digital humanities, hold great promise to unlock the richness of these artifacts for the benefit of research, teaching, and learning about the human condition.

We also thought about the complementary resources each agency could bring to the table. The NEH could offer funding mechanisms in the area of digital humanities, a field that is increasingly of interest to historians of medicine and medical librarians who regularly use NLM historical collections, databases, and data sets. The NLM could offer deep connections into these communities, particularly through its National Network of Libraries of Medicine and its popular blog, Circulating Now. Additionally, the NLM could offer the resources of its History of Medicine Reading Room and state-of-the-art meeting space at the National Institutes of Health.

The Solution

By looking critically at what each of our program areas could contribute to a collaboration, we defined the terms of the memorandum of understanding between our agencies.  Securing the support of our leadership was essential as we defined these terms, as was emphasizing the strategic value of the collaboration for both agencies and the relatively minimal financial cost of working together.

Another step in crafting this memorandum was close cooperation with our respective executive offices, specifically legal counsel, to ensure mutual agreement on the format, content, and intent of the document.

Integral to the development of our collaboration has been our rapport, our mutual respect and support, and our clear and consistent communication. We modeled this relationship with and for our respective teams so that they also would contribute their skills to partnering with each other and with the other organizations that have become a part of our collaboration. Admittedly, the top-level support we received was a linchpin to our success, but our strong rapport has been—and remains—a key factor in the success of the initial collaboration and its subsequent reaffirmation.

The Results

Between 2013 and 2015, we worked with a network of national and international partners to convene two well-received symposia and one hands-on workshop, all to the end of creating and supporting an international conversation about big-data approaches to research in the humanities and social sciences.

Our collaboration took root around Shared Horizons: Data, Biomedicine, and the Digital Humanities, an international symposium held in April 2013, funded by the NEH through a grant to the Maryland Institute for Technology in the Humanities of the University of Maryland, and co-sponsored by the NLM and Research Councils UK.

Shared Horizons, which has garnered widespread praise, brought together researchers from the digital humanities and bioinformatics communities to explore collaboration, research methodologies, and the interpretation of evidence arising from the burgeoning area of “big data” and biomedical-driven humanities scholarship. The success of Shared Horizons led us to collaborate again on a second symposium, An Epidemiology of Information: Data Mining the 1918 Influenza Pandemic.

Shortly thereafter, we discussed how we could leverage the momentum of our collaboration to develop a workshop uniquely for medical historians, medical librarians, and information technology professionals who are studying medical history and contemporary medicine. We facilitated a conversation between our colleagues at Virginia Tech and the Wellcome Trust, encouraging a collaboration around the workshop. Virginia Tech subsequently applied to the NEH for the necessary funding. The Wellcome Trust would lend support for selected scholars from the United Kingdom to participate; the NLM agreed to host the workshop and offer support in other ways.

In the early spring of 2015, NEH awarded a cooperative agreement to Virginia Tech to support the workshop Images and Texts in Medical History: An Introduction to Methods, Tools, and Data from the Digital Humanities. The organizations involved contributed to publicizing the event. Through its Twitter hashtag #medhistws, the event reached a wide audience, achieved broad and sustained engagement, and reinforced the foundation for future collaboration.

Bottom line: Strategically designed, mutually supportive, and impactful partnerships make great sense. In fact, they can have cascading effects that you may have never originally envisioned. They testify to powerful institutional synergies and the determination of colleagues to build and sustain a collaboration on multiple levels. The NEH/NLM collaboration is an example of one such partnership that has yielded effective public programming, sound financial decisions, and collaborative management. Our investment of time and effort was substantial, but it has yielded lasting dividends for everyone involved, especially the public we serve.

While our collaboration is unique in many ways, the approaches we took to build it, and the values we held to develop it, can operate in any agency culture to serve the greater good.

Further reading re: collaboration? Check out an earlier IDEA Lab post entitled – Innovation for a Safe and Healthier World: A Scientific Collaboration between CDC and the Georgia Institute for Technology

The Secret to the Perfect Project Pitch: Practice in an Elevator

Two elevators shown in the lobby of the Hubert H. Humphrey Building

In the Hubert H. Humphrey building in downtown Washington, D.C., it takes between twenty and thirty seconds to go up eight floors in an elevator—give or take a stop or two.

On any run of the mill day, those floors are the forgotten seconds it takes to get from the first floor to your office door. It’s the amount of time it takes to get in a few sips of the coffee you brought in from the Starbucks down the street. Or the time it takes to listen to a couple more verses of your favorite iTunes, Spotify, or Pandora song.

Or. It’s just the right amount of time to pitch your next big idea.

At the beginning of March, twenty-three teams were selected for the Spring 2016 Ignite Accelerator, the Department’s internal innovation startup program that provides coaching and support to help HHS Staff take their big ideas from conception to tested prototype. All twenty-three teams recently completed a three-day Boot Camp, an event that’s part training and part design-sprint to kickstart their 3 months’ effort.

A crucial part of their training has been learning how to communicate their project to a number of different audiences, including stakeholders, end-users, decision-makers, fellow entrepreneurs, venture capitalists, and senior-level government officials. Throughout their three-day Boot Camp, teams were reminded to have an “elevator pitch” in their back pocket at all times, because you never know who you might run into and when you might need it.

What is an elevator pitch? It’s a concise, clear, plain language explanation about a project you are working on and why it should be funded or supported by an influential person, company, or agency. The pitch is given in the amount of time it takes to ride a series of floors in an elevator.

Bright and early at 8:30 a.m. on the third and final day of each Boot Camp, IDEA Lab staff took the the concept of an elevator pitch literally, brewing up the ultimate communications test—Boot Camp teams ended up “accidently” riding the Humphrey building elevator with HHS Chief Technology Officer Susannah Fox (all her idea). After introducing herself, she asked each team: So what are you working on? The teams had the length of the elevator ride—eight flights, 60 seconds at best (it was the morning rush after all)—to answer the question.

I got to ride along as a fly on the wall as teams presented their pitches to Susannah. Here are some things I learned by watching them:

DO introduce yourself. In the excitement and adrenaline of the moment (oh my gosh, I can’t believe I’m talking to…), it can be easy to forget this crucial and important first step.

Even though you’re short on time, DO give your organization’s full name instead of the acronym—not everyone is familiar with the alphabet soup that makes up the names of many government agencies.

Now that five seconds have gone by, DO get straight to the point. What is your project about, why is it important, and what sets it apart? And…GO!

In the middle of all this information (15 seconds in!), DO mention the specific problem you are trying to address. This provides your listener with an understanding of how your project fits in a larger story and why it’s important that you find a solution.

DO speak clearly and use plain language. You might be freaking out on the inside (the elevator keeps going up, the seconds keep counting down), but stay calm on the outside. Chances are the person you are pitching has never heard about your project before. Explain complex terms, try not to use slang or jargon, and speak clearly and with confidence.

DON’T get caught up in the details of the technology you are using or want to use, but instead highlight the business need for your project and the benefits of that need.

Before it’s all over, before you hear the mechanical elevator voice announce your floor, DO ask for a follow-up opportunity. That could mean exchanging email addresses or getting the names of others who would be interested in your project.

After it’s all over, after you’ve stepped through the automatic sliding doors, DO take a deep breath. You did it!

In the video below we captured Carin Kosmoski from the National Institute for Occupational Safety and Health (NIOSH) at the Centers for Disease Control and Prevention (CDC) pitching her Ignite Accelerator project—NIOSH Nexus: Helping NIOSH employees make connections of who does what where.

The Invent Health Initiative: Hardware Innovations Hard at Work

vegetables and peppercorn seeds scattered on a cutting board

As a part of our invent health blog series, we’re featuring examples of hardware innovations—both new as well as those improving or finding new uses for physical objects that already exist—that have helped enable people to live more independently, in better health, and with greater dignity.

Today we’re featuring examples of assistive devices for cooking and eating.

For example, people with hand tremor are often unable to keep food on a fork or spoon. An inventor created—and tested with the support of a grant from the National Institutes of Health—an assistive device that counteracts the tremor, allowing people living with conditions such as Parkinsons to feed themselves.

Cooking is another challenge that can be overcome for people living with disability. The Amputee Coalition is a nonprofit organization that educates people about how to navigate the kitchen using adapted tools such as spring-loaded scissors, L-shaped knives, and single-handed cutting boards.


Share with us: When have you seen examples of hardware innovations improving someone’s ability to cook or eat? Tell us your story if you’ve ever created an assistive device or improved upon existing technology that made someone’s everyday tasks—once an impossibility—now accessible.  


Visit our blog post on Medium to leave us your thoughts. Not sure how to leave a response? Read this.

The Invent Health Initiative: The 30 Year Evolution of 3D Print Technology

Photo of a 3D printed protein

3D print of hemagglutinin (HA), one of the proteins found on the surface of influenza virus that enables the virus to infect human cells. Credit: National Institutes of Health, NIAID Flickr 3D Prints

Darrell Hurt, Ph.D., led development of the 3D Print Exchange at the National Institutes of Health (NIH), one of many initiatives sponsored by the IDEA Lab at the U.S. Department of Health and Human Services (HHS). This is the third post in our Invent Health blog series

Because I lived overseas in 1986, I was not among the millions of school children who witnessed the Challenger space shuttle disaster on live TV. But President Reagan’s use of John G. Magee’s poetry to describe how the seven NASA astronauts “slipped the surly bonds of earth to touch the face of God” stirs my emotions, even thirty years later. That year also saw a number of other noteworthy events, many of which would have profound effects on our lives. One particular event in 1986 received little attention because, at the time, we could not conceive the monumental impact it would make in the years that followed.

That year, the U.S. Patent and Trademark Office approved a patent filed by Charles Hull for his “Apparatus for production of three-dimensional objects by stereolithography.” It was the world’s first patent for a device that we now call a 3D printer. It would be another three years before 3D printers became commercially available through Hull’s company, 3D Systems. The technology was first embraced by the manufacturing industry, where the new method of “additive manufacturing” allowed designers to create prototypes rapidly and on site, which eliminated more costly and time-consuming traditional processes such as milling or tool and die making. Despite these advantages, the machines were large, expensive, and developments were driven by specific commercial interests. It would be 25 years after 3D Systems’ first sale before 3D printers began to reach the mainstream consumer market.

Hull’s original patent, and subsequent patents developed by 3D Systems and others, were revolutionary, but the truly exciting developments came decades later, when these patents expired, creating opportunities for others to use the technology and build upon it. When the patent on fused deposition modeling (one of the techniques used in 3D printing) ran out in 2009, efforts by Dr. Adrian Bowyer and his open-source RepRap (Rapid Replicating Platform) initiative led to the modern 3D printing movement.

The explosion of innovation stemming from that movement led to people using 3D printing in ways that Charles Hull never could have dreamed of back in 1986:

  • Thirty years after the Challenger tragedy, NASA is using 3D printing in all sorts of applications to improve the safety and well-being of its astronauts aboard the International Space Station.
  • In February 1986, Pixar Animation studios opened in California; now, they regularly use 3D printing in animation.
  • Microsoft put out its initial public offering on March 14, 1986; less than 30 years later, they entered the 3D printing software market and are spearheading a new file format for computer-assisted design.
  • In April 1986, the nuclear reactor at Chernobyl caused a deadly and toxic environmental disaster; “flying” 3D printers are now being used to clean up radioactive sites.
  • Greg LeMond became the first non-European professional to win the Tour de France in July 1986. This year, Columbian cyclist Nairo Quintana hopes to win the competition in clothing developed with the help of 3D printing.
  • The Statue of Liberty centennial was commemorated in 1986 with numerous replicas and souvenirs; now, 3D printing allows anyone, anywhere, to 3D print their very own replica.

I remember 1986, but like everyone else at the time, I had no idea of the impact that 3D printing and making in general would have on me personally and professionally. As a structural biologist, my research focuses on discovering, through experimental methods, the structures of proteins and using computer-based methods to comprehend those structures. Because structure determines function in molecular biology, understanding the complicated shape of proteins is critical to understanding the molecular mechanisms underlying all life.

Just as Watson and Crick built and benefitted from a physical model of DNA back in 1953, I knew that it would help me both understand and tell others about the much more complicated structures of proteins if only I could hold a model of a biomolecule in my hand. But manufacturing such complex structures is virtually impossible to do with traditional processes.

About eight years ago, I was fortunate enough to have access to a large, state-of-the-art commercial 3D printer, so I started playing around with 3D printing to make models of proteins. Finally, I had highly accurate, physical models of these beautiful and complex molecules. Using them, my research collaborators at NIH and I immediately began to see these structures in a new light. New insights led to new research discoveries and my enthusiasm for 3D printing grew.

Thanks to those expired patents and innovations by makers, hundreds of consumer-level, “desktop” 3D printers are now available, some for as little as $300. I saw how it might be possible for more people to print their own scientific and biomedical models, but there weren’t many designs available online, and they are difficult to createeven for someone with my experience. My team at the National Institute of Allergy and Infectious Diseases, along with collaborators at the National Library of Medicine and the Eunice Kennedy Shriver National Institute for Child Health and Human Development, created the NIH 3D Print Exchange. The Exchange is a website where anyone can download or share biomedical models for free, and we also built custom tools to help people with no experience create their own scientific models.

Since we launched the Exchange, we have done a lot of outreach in the Maker Movement, and the incredible creativity in the community never ceases to amaze me. From people like Dr. Matthew Bramlet, who is creating a library of 3D-printable hearts to help doctors and patients better understand congenital heart disease, to Dr. Jon Schull and the founders and members of the e-NABLE network, who are changing the lives of children and adults with limb differences, there are countless stories of everyday people creating (and sharing!) innovative tools to improve health and quality of life. From what I have witnessed just over the last few years, the potential for 3D printing to revolutionize healthcare, and health research, is immense, and I am eager to see where it takes us.

Finally, I want to recognize the HHS IDEA Lab, for teaching me and my team that approaching challenges with an entrepreneurial spirit can increase the impact of science. The IDEA Lab was instrumental in helping us to, as one of my team members put it, “break all the rules” in creating the NIH 3D Print Exchange. As Dr. Nicole Lurie, Assistant Secretary for Preparedness and Response, so aptly put it in the Invent Health Town Hall held on January 28, it is “a lot easier to get to ‘yes’ if you don’t have to go through ‘no’ first.” If you would like to learn more about our project and experience 3D-printed science firsthand, I invite you to visit us at the USA Science and Engineering Festival. The event takes place at the D.C. Convention Center April 16-17, 2016, and is free to the public. I hope to see you there!

The NIH 3D Print Exchange was developed with funding and support from the HHS Ignite and HHS Ventures Initiatives as well as significant internal support from Office of Cyber Infrastructure and Computational Biology at NIAID. The Project Team received an HHS Innovates Award in 2015, along with the HHS Innovates “Secretary’s Pick.” The Exchange has been featured at various Maker events, including the 2014 White House Maker Faire and the 2015 National Maker Faire.

This was post was co-written with Meghan Coakley, Ph.D., founding member and current project lead for the NIH 3D Print Exchange. If you have questions or comments about the NIH 3D Print Exchange, please direct them to Dr. Coakley at 3dprint [at] nih [dot] gov.

Making Employee Engagement a Management Priority

Next up in our blog series highlighting new efforts, and new ideas across the Department of Health and Human Services to improve the way we work, features the Administration for Children and Families (ACF). Mark Greenberg, ACF’s Acting Assistant Secretary, shares about their agency’s efforts to make employee engagement a management priority.

At a high level and for those who are unfamiliar, how does the Administration for Children and Families (ACF) promote the economic and social well-being of families, children, individuals and communities?

The Administration for Children and Families seeks to promote the economic and social well-being of families, children, individuals and communities. We have federal responsibility for more than 60 programs, with a budget of $53 billion, and about 1,300 employees in a central office and 10 regional offices. The programs we administer include Head Start, child care, child support, child welfare, Temporary Assistance for Needy Families, programming for runaway and homeless youth and victims of domestic violence, refugee assistance, the unaccompanied children program, the Community Services Block Grant, the Social Services Block Grant, Low Income Home Energy Assistance, the Assets for Independence Program, assistance to trafficking victims and more. We work with States, Tribes, Head Start agencies, Community Action agencies, and other governmental and non-governmental organizations, and we provide research, training, technical assistance and Federal leadership in support of ACF’s mission and responsibilities.

What triggered your current efforts to make employee engagement a management priority at ACF?

I became Acting Assistant Secretary in 2013. When we received and reviewed the Federal Employee Viewpoint Survey (FEVS) scores for ACF early in 2014, it was clear that there was much room for improvement. ACF staff are strongly committed to the goals and mission of ACF, but had a wide range of concerns, including ones about training, supervision, communication, participation and transparency in decision-making. After reviewing our FEVS scores, we initiated a set of focus groups, and took a number of immediate steps — establishing all-staff meetings, restoring training funds that had been cut under severe budget pressure and expanding staff and supervisory training, strengthening internal communications. Over the next year, we initiated a number of ways to encourage ACF staff to express their views about how we could improve ACF. We:

  • Established an anonymous electronic suggestion box;
  • Held a week in which everyone was invited to “tweet” their recommendations;
  • Expanded senior staff meetings to include additional staff from program and regional offices;
  • Created a monthly opportunity for staff to sign up to meet with me;
  • Focused extensively on employee engagement in the context of a move for all ACF DC staff to a new office location; and
  • Created an ACF Deputies group with a strong focus on improving ACF management

In addition, we had our workforce office meet with each ACF office to do a detailed debrief on office-level FEVS results and asked all offices to develop plans for how to be responsive to issues that arose in their FEVS scores. Finally, we formed an internal communications work group made up of representatives from a cross section of ACF offices to assess our current internal communications and make recommendations for improvements. And, we worked with the National Treasury Employees’ Union to establish an ACF Labor-Management Committee.

You have laid out 4 management priorities relating to employee engagement for 2015-2016.  How did you and your team determine these priorities?

Based largely on the feedback we got from staff, we articulated four management priorities for 2015-2016:

  1. Improve the hiring process
  2. Strengthen training and professional development
  3. Develop a system for feedback on supervisory performance and invest in supervisory performance
  4. Strengthen attention to work-life issues at ACF

ACF’s Acting Assistant Secretary Mark Greenberg participates in a panel discussion.These four areas arose repeatedly in talking with and hearing from ACF staff, and we thought it was important to make a clear commitment to address them. We think we’re making significant progress in all four areas, but are committed to doing more in 2016.

Given limited resources available for training and development in general, through your current efforts, how are you able to accommodate and meet the development needs of ACF’s workforce?

Although we were able to expand training funding, we know it’s only meeting part of the need. So we need to look more closely at the quality of current training, what we should do more or less of, and what we should do differently. Our Office of Workforce Planning and Development recently invited all staff to participate in a training assessment. We will be releasing the results to all staff and acting on what we learned. We’ve also tried to tailor specific trainings based on the needs of individual offices and will continue these efforts this year. One important point that emerged in conversations with ACF staff is that formal training is only one part of learning and professional development in an organization. In 2016, we’re planning to expand developmental details, i.e., opportunities for ACF staff to work in different parts of ACF for several months and learn more about another office and its programs. We are also creating “communities of practice” groups where staff can be connected with others doing similar work in different parts of ACF. And, we recently created a new position — Deputy Assistant Secretary for Planning, Research and Evaluation. We’ve asked the new Deputy to spend her first 90 days having conversations in and out of ACF and make recommendations for what we need to do to become a true “learning organization.”

Having launched efforts to improve ACF’s internal culture, thus far what would you consider your biggest win?  What hasn’t worked and what would you do differently?   Are there any other insights you would like to share?

This fall, we established a requirement that every supervisor in ACF, including all of Senior Leadership, participate in a 360 review process, followed by a session with a skilled coach to review results. We did this because we understand that supervision is a challenging responsibility and supervisors need more support. At the same time, I think one of the key needs in a hierarchical organization is to provide a safe way for staff to provide constructive feedback to supervisors about how to do better. Over the next six months, all ACF supervisors will go through the 360 process and we look forward to learning from it. After we complete this process, we plan to identify the most critical supervisor competencies that need attention, and develop formal and informal training opportunities to improve those competencies. Our greatest continuing challenge is the hiring process — it takes too long, we lose good candidates along the way, and we’re often not able to hire the most qualified people. Part of this is within our control, and part of it isn’t. Within ACF, we’ve improved data tracking to analyze bottlenecks, devolved authority to program offices so that they can make decisions more rapidly, made use of a committee of Office Deputy Directors to share insights and strategies and to raise concerns to the Department and the Office of Personnel Management (OPM) about aspects of the process within their control. Overall, we’ve cut the average time to hire in half, but it’s still much too long, and that has an impact on productivity and morale. We have to find ways to do better. My biggest insight may be an obvious one: ACF’s staff values and is proud of the work we do. However, in a large bureaucratic organization, too often, people believe that there’s no way for them to be heard, and that if they raise concerns, nothing will change. So, I think it’s crucial to institutionalize an organizational commitment to listening and then acting based on what we’ve heard, and that’s what we’re trying to do. Interested in other employee engagement examples within HHS?  Check out our recent post with strategic advice from the Office of the Assistant Secretary on Planning and Evaluation. [/vc_column_text][/vc_column][/vc_row]

Stay Close to Your Customer

I recently received an email from a young man who wanted my advice about how to create an information resource that would serve people like him, who share a certain health condition. I wrote back that in my role as the Chief Technology Officer at HHS, it is not appropriate for me to advise individuals, but I can recommend the principle that I have adhered to throughout my career: Stay close to your customer (and don’t assume that you are a typical customer of your own product).

Here’s a story to illustrate:

Way back in the 1990s, anthropologist Diana Forsythe conducted fieldwork in an artificial intelligence lab that was tasked with creating an information kiosk for newly diagnosed migraine patients. Yes, a kiosk. A big box with a screen that would sit in the waiting room of a doctor’s office.

The idea was that patients could walk up to the kiosk, punch in questions, and get some answers before or after they saw their doctor. It was a nice idea, ahead of its time in some ways. But when it launched, it was a failure. Patients didn’t use it after their first try. Why? Because the kiosk’s designers had not asked patients what they wanted to learn about migraines. They relied on an interview of a single doctor to tell them what he thought patients should want to know.

As Forsythe wrote:

“The research team simply assumed that what patients wanted to know about migraine was what neurologists want to explain.”

The mismatch was complete. The kiosk failed to answer the number one question among people newly diagnosed with migraine: Am I going to die from this pain? It’s an irrelevant, even silly question from the viewpoint of a neurologist, but it is a secret fear that people may have felt comfortable expressing to a kiosk.

If the designers had spent time with the target audience, they may have discovered this — even in just a few conversations or interviews. Even better, if they had created a prototype of their solution and invited recently diagnosed migraine patients to react to it, they would have found out how far off the mark they were.

These are the principles we teach in the HHS Ignite Accelerator program which is open to HHS employees who want to explore and test an idea.

For example, HHS Ignite Accelerator teams are encouraged to conduct 50 interviews of their potential customers over the 3 month period of developing a prototype. Teams typically pivot two or three times thanks to the feedback they receive. One of my favorite lessons learned by one of our teams from a food-inspection division of the FDA is that their coworkers enjoy getting out of the office to buy the meat to be tested. They wanted help, instead, with all the manual data entry associated with their work. The team switched their focus and delivered an innovation that was welcomed by their “customers.”

So take my advice and the advice of every successful entrepreneur and design thinker: Test your idea with potential customers and — this is key — listen to them, especially if their feedback is not what you expected.

To learn more about the HHS Ignite Accelerator program click here.

Featured image photo credit:  Melvin Gaal on Flickr

How to Optimize your Experience When Purchasing Services from a Crowdsourcing Platform

[vc_row][vc_column width=”1/1″][vc_column_text][vc_row][vc_column width=”1/1″][vc_column_text]Blair Corcoran de Castillo, Keyon Smith, and Stan Koutstaal are from the Administration for Children and Families’ Office of Family Assistance.  Their project, Building a Design-Minded and More Collaborative OFA, was selected as part of the second round of the HHS Entrepreneurs-in-Residence program at the U.S. Department of Health and Human Services (HHS).

Earlier this year, we set out to create a publication to share the lessons we learned from our HHS Entrepreneurs-in-Residence (EIR) project.   Unfortunately, we did not have graphic designers in-house and the options available through an existing contract were limited and too expensive.  While we considered competing a contract for design services, we realized that it would take more time, effort and money than we had.  So, much like our EIR project, we opted to test a new approach:  utilizing a crowdsourcing platform for design services.

How Crowdsourcing Services Work

Crowdsourcing is the process of soliciting ideas, products or services from the ‘crowd,’ or those external to your organization and its normal suppliers.  There are a number of crowdsourcing sites out there that can be found by just searching for ‘crowdsourcing services’ in your favorite search engine.  Marketplaces for crowdsourcing services normally work through contests.  While the exact steps may vary from one site to another, most crowdsourcing platforms work like this: a contest is formed when individuals or organizations interested in design services create a design contest on the crowdsourcing website.  

For a contest to commence, you will complete a design brief in which you describe the services required. You will also select a design package, which is how much you pay and includes the amount awarded to the winning designer.  Once the contest starts, dozens of designers submit design options over the course of the contest period.  Our contest period lasted one week.

During the contest period, you are encouraged to provide feedback to designers and to rate their designs.  As designers receive this feedback, they are able to submit different versions of their design for your consideration.  Towards the end of the contest period, the platform requires you to select finalists.  At this point, depending on the type and quality of the designs submitted, you have the option to extend the contest (in order to give additional feedback or reach new designers) or end the contest.  Once you complete the contest, you will select a winner who will make some final tweaks to the design and then turn it over to you, along with the rights to use it as you please.

Can a Crowdsourcing Platform Work For You?

Our experience utilizing a crowdsourcing platform for design services delivered to us a unique and creative publication that was fast and affordable.  We are proud to share “Creating Solutions Together: Design Thinking, The Office of Family Assistance & 3 Grantees” with others.   Using a crowdsourcing platform for services might be of interest to you if you are:

  •      Looking to access a broader range of service providers.

Normal contracting procedures encourage the selection of vendors from vetted lists, primarily including companies with prior experience in working with the government.  Utilizing a crowdsourcing platform allowed our office to interact with individuals that have the technical expertise to provide quality design services, but may have never had the chance to work with the government.

  •      Interested in only paying for exactly what you want.

Unlike a binding contract, if we did not like any of the designs that were submitted, we did not have to go with any of them.  In fact, we had the opportunity to cancel the contest and not lose any money.  Having the ability to provide timely feedback on the designs over the course of the contest was helpful in making sure that the designers knew what we wanted and that we got what we envisioned.   And the best part was that we were not charged anything additional for these rounds of feedback.  Engaging in this design contest ultimately reduced the risk of our office paying for services that did not meet our needs or standards of quality

  •       Focused on receiving affordable, timely and quality services.

This approach reduced our team’s level of effort to achieve a quality deliverable.  Since the design contest was time bound, our office knew exactly how long we needed to be available to provide feedback to designers.  In addition, the cost of receiving design services was much lower than what it could have been using traditional procurement practices as well.  In fact, we are able to receive cover design, layout design and illustration services for a highly visual 30-page publication for less than $3,000.  Using a traditional contracting route, it would have cost us upwards of $5,000.

 Tips to Optimize the Experience

As easy as working with a crowdsourcing platform can be, it is much different than the government contracting process typically used. In addition, its services do not fully align with the normal work processes found in government institutions.  For this reason, we found that there are a few ways to optimize your experience:

  1. Become familiar with the platform.

Make sure to read up on the platform and how to use it prior to creating a contest.  This will help you better prepare your contest for success.  If you have questions that are not answered through the platform’s website, do not hesitate to contact customer service.  Crowdsourcing platforms are often run by small companies.  So, not only can you get real people with intimate knowledge of the platform on the phone, but they can provide you with best practices for running a successful competition. If they don’t volunteer them, just ask.  We found that customer service representatives were happy to spend time with us.

  1. Pick a point person.

If you are working on a team, be sure to select a project lead for the contest.  This is helpful because some platforms only support one user account per contest.  For platforms with this constraint, the point person will be responsible for creating a user profile, inputting the design brief and managing the interactions with designers on the platform.  We found that having a lead placed the primary responsibility of platform-related activities on one person to manage.  This helped us to avoid missing milestones in the contest or providing confusing feedback had we all had access to the platform.

  1. Get organized.

Planning is an important part of working with crowdsourcing platforms because they are not confined by business hours or the work week.  To be prepared for these challenges, the point person should lay out the platform’s contest timeline, along with all milestones.  In addition, the lead should propose a process for how the team’s feedback will be incorporated at each important step of the contest.  We found that having set meeting times to review designs and discuss feedback as a group ensured that all of our opinions were taken into consideration.  It also enabled us to provide timely feedback to designers, allowing the feedback loops to work as the platform intended.

  1. Delegate.

As you get organized, think about how members of the team and the platform itself can be helpful to you in reaching your goals. For example, there are hundreds of design contests being posted daily on these types of platforms.  In order to have designers interested in ours, we learned that we needed to reach out to them individually. Assign members of your team to help create a list of designers that the point person can contact individually through the platform.  (As it turns out, our winning designer was actually someone we contacted!)  Platforms also know how best to reach service providers, and they might provide opportunities for your contest to be featured for a small fee.  Consider employing the platform’s services when it makes sense, since they have more experience running contests and have direct access to a huge number of service providers.

  1. Remain flexible.

No matter how much planning you do, you never know how a contest will turn out.  For this reason, you should expect the unexpected.  For us, we had to be flexible with the timeline because very few designers originally submitted designs.  Thus, we ended up extending our contest to give those designers that we reached out to time to submit designs.  (As you might have imagined, that’s when we learned the value of #1 and #3!). Ultimately, we received a design that we were excited about, but don’t worry if you experience challenges or questions along the way.

Helpful Reminders from the Designer’s Viewpoint

Our office developed a great relationship with our winning designer; however, there is still a fairly steep learning curve for any designer of a government-developed publication.  To learn how we could improve experiences for future designers, we sought feedback from our winning designer.  Here are a few of his helpful reminders for how government offices can create a shared language for and understanding of a design contest through a crowdsourcing site.

  •      Cut out the acronyms.

Our designer shared that he “initially didn’t understand all the acronyms and terminology used in the briefing and found it was initially difficult to work from.”  If our designer found it confusing, surely the general public would find our publication confusing when disseminated. What’s really important is the content, so don’t make it overly confusing by including the government alphabet soup.

  •       Provide visuals when possible.

Design contests are time-limited, so designers do not have a lot of time to research your topic.  Additionally, designers are generally not experts in government work either.  Include visuals in your design brief that can quickly get the designer up to speed and provide context for the illustrations that you want to receive.

  •       Have a 508/Accessibility expert on speed dial.

Our designer noted 508 compliance as the most difficult aspect of the project.  In fact, he shared that his primary issue was that he uses a Mac and all the tutorial resources were optimized for a PC.  He had to do a lot of experimenting to solve some of the formatting issues. To avoid a similar situation, get in touch with your agency’s 508 team and engage them in your design contest from the beginning.  

This project was also supported by the IDEA Lab’s HHS Buyers Club which is focused on modernizing the acquisition lifecycle from beginning to end, and ensuring that all stakeholders adopt the newer, more effective model of collaborating with the common goal of ensuring that each IT service acquisition leads to success. From a leadership perspective, the HHS Buyers Club is a strategic consulting service that offers guidance and tools. We don’t process the procurement, budget, IT, or other approvals, we simply bring everyone together in order to maximize success.

For more information about the HHS Buyers Club, click here.[/vc_column_text][/vc_column][/vc_row]

The HHS Ignite Accelerator: How We Chose the Spring 2016 Finalists

The results are in! We have selected the finalists for the Spring 2016 HHS Ignite Accelerator, the Department’s internal innovation startup program.

In the spirit of transparency and knowledge sharing, we are publishing our methodology for how we selected these projects. This blog post mirrors previous posts for previous Rounds of Ignite. Here are the links to our methodologies for Winter 2015 and Summer 2015.

For this upcoming 5th Round (AKA “Spring 2016”), of this cyclical accelerator program, we accepted proposals between October 1 and November 6, 2015. From that pool of proposals, we have narrowed things down to a group of finalists. This post is meant to document that methodology.

We received 82 proposals

Each submitted project idea identified a project lead, and we asked for that person’s Agency (or as we call it here in HHS, that person’s Operational Division [OpDiv]). Here’s how the 82 submitted proposals break down, by the project lead’s OpDiv:

ACF = 7
ACL = 0 *
AHRQ = 1
CDC = 17
CMS = 12
FDA = 12
HRSA = 6
IHS = 6
NIH = 16
OS = 4

A couple other data points:

  • Previous Rounds saw submission numbers of 65, 72, 74, and 42, respectively. So this is the most yet.
  • [*] One staff member from ACL was on one of the proposals although they weren’t the project lead on any proposal. That means that every Agency of HHS was represented.
  • There were 5 proposals that included individuals from outside of HHS.
  • The Total number of individuals on all proposals was 252. Teams could have up to 3 individuals (plus a project sponsor) on their project team. This is a lower limit than in past Rounds, where the cap on team members was set at 5.
  • HRSA is currently and simultaneously running their own internal accelerator pilot which they’re calling the HRSA Idea Spring. (FYI – It’s awesome.) If you add the submissions to the HRSA Idea Spring to those of HHS Ignite Accelerator, then there were actually over 100 project ideas submitted. WOW.

These 82 proposals were scored by 25 Reviewers

The Reviewers were comprised of previous Ignite team members…

  • Lewissa Swanson, OS/OASH
  • Michael Johansson, CDC
  • Dan Stowell, CDC
  • Bethany Applebaum, HRSA
  • Amy Wiatr-Rodriguez, ACL
  • Jon Langmead, CMS
  • Shana Freno, SAMHSA
  • Dan Duplantier, OS/ASPE
  • Katerina Horska, OS/IOS
  • Melody Stevens, CDC
  • Leigh Willis, CDC
  • Dan Baden, CDC
  • Cynthia Vinson, NIH
  • Vinay Pai, NIH
  • Sabrina Matoff-Stepp, HRSA
  • Michelle Davis, OS/OASH

As well as folks from the IDEA Lab family…

  • Andy Burton, HHS IDEA Lab (part time detail from NIH)
  • Betsy Hsu, HHS IDEA Lab
  • Corinne Fukayama, HHS IDEA Lab
  • Damon Davis, HHS IDEA Lab
  • Malini Sekhar, HHS IDEA Lab
  • Mark Naggar, HHS IDEA Lab
  • Serena Parks, ACF (formerly of the IDEA Lab. Serena, we miss you 🙂 )
  • Katrina Jones, HRSA Idea Spring
  • Candace Webb, HRSA Idea Spring

Each proposal was scored 5 times

We broke our 15 Reviewers into 3 panels and then randomly distributed the 42 proposals across the panels. This means that each proposal was scored 5 times. We used the average of the 5 scores to make a finals score that we used in our analysis.

Each Reviewer received the same instructions and guidance for how to score. Now even with that said, certainly some reviewers were “nicer” and others were “meaner.” This means that some panels were “nicer” and others were “meaner”. We, of course, don’t want that to influence who was selected as finalists, so we used Z-scores to help normalize the scores. See more about that below in the section called: “There Were Four Ways A Project Idea Could Advance“

We asked each individual to self-identify if they should recuse themselves. There was one recusal for one proposal.

Based upon defined criteria

Each proposal was scored on a 0-100 range based upon our communicated criteria:

  • The project’s alignment to the Office, Agency mission [20 points]
  • The proposal’s explanation of the process, product, or system to be addressed. [60 points]
  • How well the proposed solution aligns with the communicated problem. [20 points]

After scoring, reviewers were asked on a binary scale: do you think this proposal should be considered to become a finalist? Each Reviewer was also asked to provide brief comments on the proposal that help justify their score.

There Were Four Ways A Project Idea Could Advance

The following were ways in which a proposal was able to advance:

  • The top 20 z-scores overall
  • Top 25% z-score from each OpDiv (minimum of 2 per OpDiv)
  • Review Panel unanimously votes to advance it (regardless of score)
  • Wildcards picked by IDEA Lab staff

As a refresher on statistical methods: Z-scores are ways to normalize across groups. You can imagine how one of our Panels was “nicer” than another panel. We’d hate for a project to get a lower score simply because of the makeup of their panel. Z-scores, also called “standard scores” help prevent that from happening. Here’s the wikipedia article on z-scores if you’d like to dive deeper.

Some proposals fell into more than one category. For example, a proposal may have been both a top scoring proposal and also top in it’s OpDiv. Using the first three rules, 31 proposals emerged.

We believe in rules and algorithms to ensure fairness, but we also hesitate to over-rely on them at the risk of losing the touch of human judgement. So we always have a Wildcard category. Wildcards were picked by two staff from the IDEA Lab (specifically, Read Holman and Will Yang).  We went through every proposal and picked proposals that we felt eligible for a wild card slot. Reviewers were able to lobby for the ones they liked. From the pool of eligible project ideas, 16 were selected as wildcards.

We Selected 47 Project Ideas as Ignite Finalists

Through the above criteria, a little more than half of the project ideas submitted were identified as Finalists. For the sake of space and to keep the methodology and process information separate from the announcement, we posted information on those finalists on a separate blog post.

Continue Reading: From The HHS Ignite Accelerator: The 47 Finalists for Spring 2016!