Lucidworks: Search and Big Data- Navigating the Clinical Data Labyrinth

CIO VendorGrant Ingersoll, CTO
The landscape of clinical trial management and the broader pharmaceutical arena is often characterized by varied and complex data types. As data is being continuously produced, finding the information at times still proves to be difficult. Lucidworks, an enterprise search technology organization, has built its success on not only gathering data from multiple sources, but also in making that information easily accessible for users. A key focus for Lucidworks, explains CTO Grant Ingersoll, “is not just the content side of the equation but also the way users interact with it. A lot of time, enterprise applications fail because they fail to understand how the user interacts with the content.”

The Lucidworks Fusion platform was created to address an organization’s search and discovery needs. “As with any system, you have to be able to get the data into it,” Ingersoll says. “We focus a lot of time and energy on connecting all the different data sources that a company might have,” which can include files in the cloud, a company’s internal files, and other pertinent information. “We leverage core technologies around search, machine learning, and natural language processing to bring together all those different ways of analyzing data and deriving in-depth insights.”

Also critical is keeping that information secure to address regulatory concerns with compliance and HIPAA, as well as other policies for dealing with sensitive data. With all the data they integrate, “We make sure that we overlay security on to all of that. This is both security of the underlying system as far as who can access it, but also the data within that system.”

In order to easily access this information, Lucidworks has gone past just the basic search box.

We leverage core technologies around search, machine learning, and natural language processing to analyze data and derive in-depth insights

Because the general public is attuned to searching for information, “it’s a very natural way for users to interact with the data.” However, it’s more than finding a simple answer to a basic question. “This is where we really leverage technologies like machine learning and natural language processing to analyze not only the core content that we brought into the system but also how users interact with it,” Ingersoll states. “It becomes really important to think about how users are interacting with that content because that’s often where you can find connections that aren’t necessarily obvious.”

For example, Lucidworks assisted in a client’s project that married both research and clinical trial findings with government-initiated data, such as, the FDA’s Adverse Events database, and the NIH for funding research. A user’s initial search will then yield a primary set results, he says, but then ties those to results to more data. “Everything else was driven through discovery by the user navigating to find chemical compounds related to that search,” Ingersoll says. “From there, we pivoted to find other related compounds, then added relationships to the main ones that the user selected.” So, if a user initiates a search for what medications can be used to treat a headache, the traditional answers for popular medications would then go much further, including compounds related to the molecular compounds in the initial search, latest research, reported adverse effects, funding cycles, and more. “We are enhancing the findability of data and enriching it to produce value throughout the pipeline,” he said.

Lucidworks is determined to keep focusing on what is important: the clients. “Providing connectors, security, and ease-of-use is key,” he says. “At the end of the day, it’s focusing on the users and helping them figure out what’s important, what’s the next best action, and what they need to know right now. That is really the linchpin of everything we do.”