Overcoming Data Limitations with Drupal Views

By Jason Safro

If you’ve developed with Drupal, you’ve probably worked with Views, the pre-eminent solution for displaying multiple pieces of content. It is a flexible tool and an excellent solution 90% of the time. But, what about that other 10%? Don’t worry! You can still use Views! We just ran into one such situation and wanted to share how we made it work.

_{In Drupal, “view” refers to a formatted list of content displayed on a website, and “Views” to the Drupal functionality that allows you to configure formatted lists of content. Views can be used to display nodes, users, taxonomy terms, groups, and just about any other entity Drupal knows about. They can interact with other common Drupal tools like Entityqueues, Flags, or Apache Solr search results. Results are output in a multitude of formats including lists, tables, grids, maps, RSS feeds, CSV files, and slideshows.}

Project background

On a recent project, we hit a limit on what Views can do. The client is a university system with multiple campuses and a particularly tall hierarchy of data for each campus, as displayed in the image below.

The client’s site includes a Campus Search tool that allows users to search for campuses based on a variety of criteria including geographic distance, Award Level, and Area of Study. We implemented Apache Solr and Search API for full text search and scalability and our team put a lot of effort into making the Campus Search slick, stylish, and functional. As can be the case, our solution was challenged by a last minute feature request.

The challenging new request

During the final round of UAT, a key stakeholder requested functionality that had never been discussed or documented before. They wanted users who were searching for campuses within a geographic radius to have the option to also see campuses with fully online programs. Due to the unexpected changes brought by the pandemic, we could all see how this new filter added significant value to the end user.

But there was one problem, this feature request presented a fundamental challenge, the new filter was incompatible with a Solr index-based view. The existing view could not return the “fully online” options. No single view could get the “fully online” options and leverage the benefits of Solr.

For example, in a common use case, a student might want to find out where he can attend classes for a Master’s Degree in Business. He might be interested in attending classes in person or fully online. Using the Campus Search tool, this student might select “Business” from the Area of Study filter and also check the checkbox to “Include Fully Online programs in the search results”. Based on this use case, Solr would be asked to return all campuses where Area of Studyl TID == 39 and Fully Online == ‘Yes’.

However, as detailed in the chart below, Solr would return only North City Campus. This is a valid result based on the flattened data in the Solr index, but it is not correct based on the hierarchical data in the Drupal database.

Because the Solr index does not know that the Master’s Degree in Business is not offered “fully online”, it returns the wrong results.

Why was the request incompatible?

Drupal Views are not designed to query multiple data sources, but to make this request work, we needed data from both Solr and data from the Drupal database. We implemented Apache Solr because it provided Solr’s more efficient search algorithm and reduced the load on the Drupal database. However, this came with a hidden cost. Solr indexes are optimized for search and do not maintain relationships between entities. The Solr index could tell us if there is a “fully online” Degree available at a Campus, but it could not tell us which of the Campus’s Degrees is “fully online”.

The figure below shows an example of a campus that offers two degrees: a Bachelor’s Degree in Art History and a Masters Degree in Business. Note that only the Bachelor’s Degree is offered fully online which means it would not return .

_{Figure 2: In this example, a simplified Drupal data structure highlights this situation. We have the following types of entities:
Node: Campus
Paragraph: Campus Degree
Taxonomy: Degree
Taxonomy: Award Level
Taxonomy: Area of Study}

We didn’t want to scrap Solr because the Campus Search view was built upon a Solr index for several good reasons:

• Solr provides a more efficient engine for search.
• Processing searches with Solr reduces the processing done by the Drupal database and redistributes some to the Solr server.
• Solr provides good functionality for geo-location proximity searches.

Our team wanted to meet this new request while also retaining the benefits of Solr, even though the new feature request was fundamentally incompatible with that Solr index. There are probably many ways to meet this challenge, but here’s how we did it…

Our solution

Views are typically used as an all-in-one solution that both fetches data and outputs the rendered HTML results. Views can also be called programmatically, granting access to the data directly. This is the crux of our solution.

The full solution required three views and a custom plugin for a Contextual Filter Default Value (CFDV). The CFDV calls two of the views programmatically and returns a filtered list of campus NIDs. This CFDV is used by the third view, which handles the remaining filtering, sorting, pagination, and rendering HTML output.

1. Geographic Proximity View

One of the new Views found all campuses based on geographic proximity. This View takes an address and a radius as inputs. It processes against the Solr index. It returns campuses within the radius of the address.

2. Fully Online Programs View

The second new View queries the Drupal database directly. It looks for any campuses offering programs that are fully online and match the Award Level (Bachelor’s, Master’s, etc) and Area of Study (Arts, Business, etc) criteria selected by the user.

3. Programmatic Access In A Contextual Filter Default Value Plugin

The two new Views are just part of the solution. They each provide a list of campuses filtered for some of the user criteria. We brought these two Views together by accessing them programmatically inside a custom plugin that provides a Contextual Filter Default Value to:

• Access the user’s selections for address, radius, “Include fully online…”, etc
• Load the two Views with their appropriate parameters.
• Return a collated string of Campus NID (separated with ‘+’)

The final View is also based on the Solr index. It uses the new custom CFDV in a Campus ID contextual filter. This View also handles the remaining filters plus the sorting, pagination, “no results” text, and generates results rendered in HTML.

_{Figure 3: The recipe.}

Our solution accesses two different Views programmatically inside of a Default Argument for Views Contextual Filters plugin. The plugin collates that data on the back-end and puts it to use. If you want to try accessing data from a View programmatically, please check the code sample below.

Developer questions

1. How do you programmatically retrieve data from Drupal Views (because the solution involved querying storage three times, refining the result one after the other)?
   The answer is the code itself in methods setCampusIdsForFullyOnline() and setCampusIdsByProximity(). We programmatically retrieved data from a Drupal View.
2. How do you index nested data into Solr (because Solr flattens the indexed data)?
   We didn’t figure that out. We took another approach.
3. How do you store referenced data in Solr?
   Solr does store referenced data. Check below “The North City Campus might appear in the Solr index as follows:” That shows where Solr is storing all of the data related to a campus together, associated with that campus.
4. How do you filter one View’s result with another and how do you pass results of one view to another view as arguments?
   The entire code sample is an answer to that, that’s exactly what we did. We programmatically loaded data from two views and passed it to a third via contextual filter. Also, views_field_view allows you to populate a View FIELD with another View. We wanted to populate a View FILTER ARGUMENTS (VFA) with another View, so VFV would not have worked. And, even if VFV did handle contextual filter arguments, it can’t collate data from two data sources (ie, two Views).

Learn more

• Jobs: Current openings at GovWebworks/Portland Webworks
• Issues we care about: GWW on Twitter
• Things we do: PWW on Facebook
• Who we are: PWW on LinkedIn

Postscript: Code sample

This code sample shows three things:

• A sample custom plugin for a Default Argument for Views Contextual Filters.
• Programmatic use of a standard view (see method setCampusIdsForFullyOnline()).
• Programmatic use of a view based on a Search API index (see method setCampusIdsByProximity()).

This code sample is task specific. It contains some hard-coded strings, references to project specific views, and logic written specifically for the task.

<?php

namespace Drupal\module_name\Plugin\views\argument_default;

use Drupal\civiserv\Entity\Location;
use Drupal\Core\Cache\CacheableDependencyInterface;
use Drupal\Core\Form\FormStateInterface;
use Drupal\search_api\Item\Field;
use Drupal\search_api\Item\Item;
use Drupal\views\Plugin\views\argument_default\ArgumentDefaultPluginBase;
use Drupal\views\Views;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
* Default argument plugin to extract a user from request.
*
* @ViewsArgumentDefault(
*   id = "college_search_proximity_helper",
*   title = @Translation("College Search Proximity Helper")
* )
*/
class CollegeSearchProximityHelper extends ArgumentDefaultPluginBase implements CacheableDependencyInterface {

 static $class_name = 'CollegeSearchProximityHelper';

 /**
  * A container for all Campus (Location entity) IDs that will be returned.
  *
  * @var array
  */
 protected $campus_ids = [];

 /**
  * Constructs a new User instance.
  *
  * @param array $configuration
  *   A configuration array containing information about the plugin instance.
  * @param string $plugin_id
  *   The plugin_id for the plugin instance.
  * @param mixed $plugin_definition
  *   The plugin implementation definition.
  */
 public function __construct(array $configuration, $plugin_id, $plugin_definition) {
   // Run the parent constructor.
   parent::__construct($configuration, $plugin_id, $plugin_definition);
 }

 /**
  * {@inheritdoc}
  */
 public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
   return new static(
     $configuration,
     $plugin_id,
     $plugin_definition
   );
 }

 /**
  * {@inheritdoc}
  */
 public function buildOptionsForm(&$form, FormStateInterface $form_state) {
   $form['header_educator'] = [
     '#type' => 'html_tag',
     '#tag' => 'header',
     '#value' => 'Caution! This is a task-specific plugin designed to be used on campus search. The plugin returns Location IDs.<br /><br />You MUST select More > Allow Multiple Values when configuring this contextual filter.',
     '#attributes' => [
       'style' => 'font-weight: bold; font-size: 110%;',
     ],
   ];
 }

 /**
  * {@inheritdoc}
  */
 public function getArgument() {
   // Init
   $campus_ids = [];
   $latlng = $_GET['proximity']['value']['latlng'] ?? NULL; //'45.1732394,-93.3030063';
   $distance = $_GET['proximity']['distance']['from'] ?? 16.09;
   $include_fully_online = $_GET['field_fully_online_programs']['True'] ?? NULL;
   $default_return = 'all';

   // Trap out if the user is not filtering by proximity.
   if (!$latlng) :
     return $default_return;
   endif;

   // Get all campuses within the geographic radius.
   $this->setCampusIdsByProximity($latlng, $distance);

   // Add 'Fully Online' campuses if the user selected that option.
   if (!empty($include_fully_online)) :
     // Init.
     $area_of_study_ids = $_GET['field_area_of_study'] ? implode('+', $_GET['field_area_of_study']) : 'all';
     $award_level_ids = $_GET['field_award_level'] ? implode('+', $_GET['field_award_level']) : 'all';

     // Update the Campus IDs.
     $this->setCampusIdsForFullyOnline($area_of_study_ids, $award_level_ids);
   endif;

   // Prepare the output that will be consumed by a view contextual filter.
   // Ideally, this is a string of Campus Location IDs, separated with '+'.
   $out = $default_return;
   if (is_array($this->campus_ids) && count($this->campus_ids) > 0) :
     $out = implode('+', $this->campus_ids);
   endif;

   // Good job!
   return $out;
 }

 /**
  * Find campuses by proximity and set them to the class property.
  *
  * Utilize Search API and a View to find all campuses within a geographic radius. Set
  * the IDs to a class property.
  *
  * @param string $latlng
  *   The latititude and longitude for the center point of the proximity search.
  * @param string $distance
  *   The radius of the search in kilometers.
  * @return false|void
  */
 protected function setCampusIdsByProximity($latlng, $distance) {
   // Init.
   $new_campus_ids = [];

   // Load view that will give us the valid Area of Study TIDs.
   $search_api_view = Views::getView('campus_location_search');

   // Select a display.
   $search_api_view->setDisplay('attachment_college_search_geo_only');

   // Add contextual filters.
   $search_api_view->setArguments([$latlng, $distance]);

   // Validate and trap out.
   if (!($search_api_view->execute())) :
     \Drupal::logger(self::$class_name . '->' . __FUNCTION__)->warning('Failed to execute Search API view.');
     return FALSE;
   endif;

   // Validate and iterate through the results.
   // Add to the array of Campus Location IDs.
   if (!empty($search_api_view->result)) :
     foreach ($search_api_view->result as $one_id => $one_row) :
       // Search API returns Items instead of the normal search result.
       // Validate. Skip to the next row as necessary.
       $one_item = $one_row->_item;
       if (!($one_item instanceof Item)) :
         continue;
       endif;

       // Get the field we need.
       // Validate. Skip to the next row as necessary.
       $one_field = $one_item->getField('civiserv_location_id');
       if (!($one_field instanceof Field)) :
         continue;
       endif;

       // Get the values in a local variable.
       $one_values = $one_field->getValues();
       if (is_array($one_values)) :
         foreach ($one_values as $one_location_id) :
           $new_campus_ids[$one_location_id] = $one_location_id;
         endforeach;
       endif;
     endforeach;
   endif;

   // Update the class property.
   $this->setCampusIDs($new_campus_ids);

   // Good job!
   return TRUE;
 }

 /**
  * Find campuses offering fully online programs and set them to the class property.
  *
  * Utilize a View to find all campuses offering fully online programs matching the
  * user's criteria. Set the IDs to a class property.
  *
  * @param string $area_of_study_ids
  *   A string of IDs for Area of Study entities, separated with '+' for consumption
  *   by a View contextual filter.
  * @param string $award_level_ids
  *   A string of IDs for Award Level entities, separated with '+' for consumption
  *   by a View contextual filter.
  * @return bool
  */
 protected function setCampusIdsForFullyOnline(string $area_of_study_ids, string $award_level_ids) {
   // Init.
   $new_campus_ids = [];

   // Load view that will give us the valid Area of Study TIDs.
   $fully_online_campuses_view = Views::getView('campus_location_with_program_information');

   // Select a display.
   $fully_online_campuses_view->setDisplay('default');

   // Add contextual filters.
   $fully_online_campuses_view->setArguments([$area_of_study_ids, $award_level_ids]);

   // Validate and trap out.
   if (!($fully_online_campuses_view->execute())) :
     \Drupal::logger(self::$class_name . '->' . __FUNCTION__)->warning('Failed to execute view.');
     return FALSE;
   endif;

   // Validate and iterate through the results.
   // Add to the array of Campus Location IDs.
   if (!empty($fully_online_campuses_view->result)) :
     foreach ($fully_online_campuses_view->result as $one_id => $one_row) :
       // Search API returns Items instead of the normal search result.
       // Validate. Skip to the next row as necessary.
       $one_item = $one_row->_entity;

       if (!($one_item instanceof Location)) :
         continue;
       endif;

       // Repack the ID into a local array.
       $new_campus_ids[$one_item->id()] = $one_item->id();
     endforeach;
   endif;

   // Update the class property.
   $this->setCampusIDs($new_campus_ids);

   // Good job!
   return TRUE;

 }

 /**
  * A set function to add Campus IDs to the class property.
  *
  * @param array $new_campus_ids
  *   An array of campus IDs that will be added to the class property.
  * @param bool $reset
  *   A flag to indicate if the class property should be cleared before adding new Campus IDs.
  * @return void
  */
 protected function setCampusIDs($new_campus_ids, $reset = FALSE) {
   if ($reset) :
     $this->campus_ids = [];
   endif;

   foreach ($new_campus_ids as $one_campus_id) :
     // Sanitize the campus ID.
     $one_campus_id = trim($one_campus_id);

     // Add to class property.
     if (!empty($one_campus_id)) :
       $this->campus_ids[$one_campus_id] = $one_campus_id;
     endif;
   endforeach;
 }

 /**
  * {@inheritdoc}
  */
 public function getCacheMaxAge() {
   return 0;
 }

 /**
  * {@inheritdoc}
  */
 public function getCacheContexts() {
   return ['url'];
 }

}