Every Blackfoot item featured on the Mootookakio’ssin website has a unique story to tell, and it is difficult to tell a story when you are hidden from the world, collecting dust in a museum. Bringing recognition and appreciation to these inaccessible Blackfoot items has been the underlying goal of the Mootookakio’ssin project; after all, that is what Mootookakio’ssin means: distant awareness. This project brought together a team of many talented researchers both locally and internationally, across a broad range of disciplines. The process of bringing these items closer to home has been a long and winding journey: from travelling overseas to visit these items in museums, to consulting with Blackfoot elders to gain a better understanding of them, and finally leveraging the power of web-technology to share them with you on Mootookakio’ssin. My name is Calvin Lloyd, and as the project’s lead web developer, it was my job to implement the vision of our designers and build the Mootookakio’ssin site from the ground up to provide a suitable home for these objects on the web. Are you curious about the frameworks and tools that were used to create the site and display the beautiful Blackfoot items found there? This blog will explain my contribution to the project and hopefully satisfy some curiosity about the Mootookakio’ssin website, how it works, and the steps involved in its creation.
What should it do, and how do we do it?
Before development even began, two crucial steps had to be completed, and several important questions needed to be answered. The first step was requirements gathering, in which the development team sat down and discussed what we wanted to achieve with the website, what functions we needed it to perform, and what expectations users will have for it. For example, we knew the site needed to be able to display 3D models without compromising the site’s performance. Second, we discussed the need to potentially support the addition of more Blackfoot items to the site after its completion. Third, we knew that a certain level of interactivity was required to keep users engaged with the content, avoiding the pitfalls of some historical information and museum sites where information is presented in a “wall of text” fashion. These three examples don’t represent the exhaustive list of requirements, but they were some of the most high-level and important requirements for the project, and they contributed greatly to development.
After the requirements were established, the next step was to analyse those requirements and find the optimal way to satisfy them. This was my first sizable contribution to the project. I set out to discover which tools, technologies, and frameworks could potentially satisfy our needs, and to test them on their own to determine their viability for the project. 3D web technologies were of the utmost importance, not only for displaying models, but also for using them as a means of navigation through the use of hotspots. Hotspots facilitate immersive and exploratory navigation through labelling locations on 3D Blackfoot item models with 2D buttons, which, when clicked, bring up relevant information about that point of interest. I chose to focus on this requirement first. I began coding a basic web page for testing and demonstration purposes. I started with a popular JavaScript 3D modelling library called Three.js, which implements a high-level system for WebGL-based rendering. Using Three.js to render a basic 3D model on a turntable took around 150 lines of JavaScript code to implement. At first glance, this may seem like a reasonable coding requirement, however, the more functions we needed the models to perform (e.g., supporting hotspots for navigation), the more complex the code would become to write and maintain. This complexity would make functional and aesthetic 3D model modifications too cumbersome after the site’s launch, which would not support the expandability requirement for the site. Despite the wealth of supporting documentation for Three.js, the notion of creating an entire system for rendering models that is both expandible and maintainable fell outside the scope of the project and could easily have become a substantial project on its own. Because of this, I began researching alternate means of web-based 3D rendering.
<Model-Viewer>: Why waste time say lot word when few word do trick?
After some searching, I came across <model-viewer>, a promising Web Component that offered easy web-based 3D rendering, without requiring the same meticulous and time-consuming process demanded by Three.js. At the time, the <model-viewer> project was still in development, but despite being unreleased, it was available for use with the caveat that it might not support all of the promised features or work correctly in all cases. Even in its unreleased form, <model-viewer> offered much of the functionality we needed from Three.js, such as hotspots and augmented reality (AR), while needing only a single line of code to be functionally equivalent to the Three.js model. I built a short demonstration that showcased what <model-viewer> had to offer and compared it to Three.js (Figure 1). The team was excited about its ease of use, feature set, and what it offered in terms of user experience, agreeing that it would allow us to focus on the design and functionality of the site as a whole, rather than spending too much time on first-party 3D rendering.
Figure 1: Comparing the difference between Three.js and <model-viewer>
To Web App or not to Web App?
We asked ourselves an important question: Is the Blackfoot Project aimed at building an informational website or an interactive web application (web app)? Answering this question helped to shape the future of the project and provide insight into what additional tools we might need to achieve our goals. We knew that we wanted to keep our users engaged with the content and give them an interactive and interesting experience, both in terms of the quality of the content and the navigational structure of the site. We wanted each Blackfoot item featured to tell a unique story about its past through interactions with the 3D model itself, rather than relying on more typical navigation structures found in the majority of sites. The notion of adding more Blackfoot items to the site post-release still required an adequate solution as well. With this in mind, the idea of a web app became more appealing, and I decided to look into front-end frameworks that would help us to satisfy our requirements for interactivity, performance, and expandability.
The front-end of a web app refers to what users see and interact with, often employing a front-end framework to facilitate the development of the user interface, especially when more advanced user interaction is needed. In general, there are three primary competitors in the front-end JavaScript framework space, namely Angular.js, React.js, and Vue.js. Because I had no prior experience with front-end frameworks for web development, I needed to be careful to choose the one that offered the most extensive supporting documentation and resources to facilitate learning and development. React.js was the obvious choice for me due to its popularity and foothold on the market of front-end development, as shown by Figure 2. Selecting React as our framework had the greatest impact on development overall, since it could be used to satisfy a great deal of our requirements. It enables an interactive design, flexibility in programming and templating (more on that later), and performance greater than what a standard web page can provide. Additionally, it facilitates atomic design principles by bundling related design elements and functionality into reusable components.
How React works in general (for a more technical reader)
The process of visiting a standard webpage on the internet is fairly straightforward. Every page that a user navigates to corresponds to a file located on a server somewhere on Earth. When you navigate to a page on the web, you are retrieving the file corresponding to the URL you visit. This file is then loaded into your browser, which creates an HTML Document Object Model (DOM) of that page. Each time you visit a new page on a standard website, a new file is retrieved, and a new DOM is constructed. React-based apps on the other hand work by populating a single DOM with content and replacing that content through the processes of rendering and re-rendering.
React Apps are comprised entirely of components which are small, self-contained, reusable elements, nested into a hierarchy of parent elements, and their children. Each component contains the code defining its own structure and functionality, in addition to state and lifecycle. State refers to a JavaScript object contained within a component that holds custom values relevant to it. When one of these state values is modified, the component responds by re-rendering. Lifecycle refers to a set of built-in React functions triggered when certain conditions are met, such as when the component is initially mounted, or rendered for the first time (componentDidMount()). Rendering also occurs when new values are sent from parent component to child, through props.
Props are the means through which components are able to communicate with one another through the sending of variables and functions from parent to child. Unfortunately, children cannot send props to one another directly and instead must communicate through their common parent element. This is potentially the biggest shortcoming of React, making it fairly convoluted to have two children within the hierarchy send necessary data to one another. This process of rendering and re-rendering, based on state and lifecycle results in the different views and interface elements that the user sees and interacts with when they visit a React application on the web. Figure 3 shows a diagram depicting the lifecycle of React. Since executing code is significantly faster than transferring files over the internet, this results in performance improvements over standard websites, since a new DOM is not required when navigating between URLs on a React site and only site assets like images (and in our case 3D models) need to be retrieved. If you are interested in learning more about React and its lifecycle, this is an excellent interactive resource.
Why didn’t you just use a Content Management System to build Mootookakio’ssin?
A Content Management System (CMS), like WordPress, is a powerful tool that allows for extremely easy and fast web development through choosing a template and populating that template with content. The benefits of a CMS include easy modifications to existing content, as well as the ability to add new content very quickly. Some modern CMSs even allow formatted content blocks to be drag-and-drop inserted into a webpage, and easily populated with custom content. While this does sound like it would satisfy several of our requirements, CMS are not as performant as React applications, nor would their use allow us to be as creative as we wanted in terms of design, functionality, and navigational structure, and for these reasons, we avoided using one. However, just because a CMS would limit the creativity of our designers does not mean that the benefits offered by a CMS should be discounted.
I was inspired by the ease of adding and modifying content, as well as the notion of formatted content blocks provided by a CMS. I wanted to leverage these same advantages within a custom system for Mootookakio’ssin, so I created a similar architecture in React. In addition to efficiently updating Mootookakio’ssin with additional Blackfoot items, I also wanted to ensure that website functionality and content were entirely separate so that modifying or adding content would not require modifying the code of any React components whatsoever. After rigorous testing, experimentation, and iteration, I was confident I had found a solution that satisfied all our needs.
Finally, we are going to talk templating
The Mootookakio’ssin React app is a custom templating solution consisting of two independent parts, the Blackfoot item configuration files, and the static structure of the site. The static structure of the site is comprised of React components, providing a blank slate to be filled with custom content, which comes from the Blackfoot item configuration files. Creating these components involved interpreting design elements and bundling related functionality into small portions that could easily be reused throughout the site, while simultaneously following atomic design principles. These components were then integrated into a single cohesive application, which accurately reflected the supplied designs. The process of interpreting designs and abstracting components from them was a challenge, considering no two Blackfoot items were the same and components needed to be built to support countless potential configurations and descriptions of items, both present and future. Building individual components and determining how they would be used together, was a heavily iterative process that required a great deal of trial and error, as well as an agile development mindset, given the constant modifications made to the aesthetic site designs and proposed functionality. When used in conjunction, these many small components comprise the static structure of the Mootookakio’ssin template. This static structure as a whole needed to be built to accommodate our list of requirements, while also facilitating the decoupling of functionality and content, allowing for easy expandability without advanced coding knowledge. To achieve this, content was entirely removed from the static structure and instead placed in separate Blackfoot item configuration files.
This is where the notion of templating comes in. Templating in this instance is when a web app automatically generates custom webpages by populating a static structure with dynamic content. Each Blackfoot item on Mootookakio’ssin has a unique configuration file related to it. This file contains a plethora of information related to the item, including various descriptions, names, translations, categories, hotspot info, 3D models, images, and meta information such as size and weight. All of the information related to a given Blackfoot item is localized in a structured JavaScript Object (not to be confused with a JSON) within the configuration file for that specific item, formatted in key-value pairs. The key is basically a variable within the JavaScript Object that expects a certain type of value to be associated with it. For example, within the JavaScript Object for the Beaded Belt, you would find a key “name” associated with a string value of “Beaded Belt.” You can also place basic unstyled HTML elements as the value of keys, as is the case with each item’s description. This allows all of an item’s information to be stored in one place and easily changed and updated with very little coding knowledge, providing our templating model with dynamic content that is both expandible, maintainable, and entirely separate from component functionality.
I created the static structure of Mootookakio’ssin to be both flexible and semi-modular, allowing for content to differ greatly between each Blackfoot item, including the ability to add various types of formatted content blocks, add sections relevant to each unique item, or leave out irrelevant sections. For example, because the Design section is not relevant to certain Blackfoot items, that portion of the item’s configuration file can be left out, resulting in that item’s design page not being rendered within the app. This flexibility also makes the static structure very forgiving, allowing it to automatically correct small errors within the configuration files by filling in gaps with default information where necessary or leaving out the missing data entirely.
Our custom templating model dynamically pulls in content from the Blackfoot item configuration files when needed and uses it to populate the static site structure with the required content by mapping the key-value pairs from our item configuration files. Not only does this enable highly custom content within a Blackfoot item, but also allows additional items to be customized and added to the site with ease post-release, given that only Blackfoot configuration files are needed. For example, the Explore view that shows all the Blackfoot items on the site will resize and evenly redistribute the item nodes around the ring as new items are added. Therefore, this complex view’s code does not need to be adjusted to add a new item, since the static structure was built to account for new additions. This custom templating solution decouples content from functionality and facilitates expandability and maintainability of content, allowing us to tell unique stories for each of the items that we feature without needing to redefine the structure of the site and modify the code that comprises it.
Mobile AND Desktop? Oh my…
The static structure of the app needed to be adapted into both mobile and desktop versions, which can differ quite dramatically in some cases. Fortunately, this did not require modifying the Blackfoot configuration files in any way. Typically, in responsive design, Cascading Style Sheets (CSS) handles the majority of the workload when creating a site or app that should work across a wide range of smartphones, tablets, and desktops, each of which has a different resolution and aspect ratio. This wide range of screen sizes can make responsive design a little tricky in some cases. CSS is the language used in web development to handle the aesthetic rules that govern the appearance of the site. These rules can be redefined based on the size of the viewport, and will be automatically selected based on the size of the window/display being used to view the app. This is the essence of responsive design. However, certain components require special care for the mobile and desktop views, sometimes even requiring an entirely separate component to be created with React. One example of this is Mootookakio’ssin’s Explore view where the desktop version of the site displays the Blackfoot item nodes in 3D while the mobile view is only in 2D, as shown in Figures 4 and 5. In this case, it was easier to build totally separate components for both of these views and leverage the MediaQuery class from the “react-responsive” module. MediaQuery within React allows one to specify rules that dynamically select components to render based on the size of the viewport, allowing the app to dynamically re-render with different components based on the sizes specified in the MediaQuery.
Tools in our Toolbox
In addition to <model-viewer> and React, several other tools deserve to be mentioned in this blog. First, to get up and running quickly with React at the beginning of the project, I used create-react-app to bootstrap the project and provide a starting point to jump into development right away. Create-react-app sets up a very basic single-page application in React and installs all the dependencies needed to get started. I would recommend create-react-app for anyone looking to jump into React and learn it with almost no barrier to entry. Another tool used hand-in-hand with React is Node.js, an open-source command-line tool used for installing libraries, running servers, building production versions of React apps, and much more. Node.js is very useful and is almost always used when working in React-based applications.
Web development often necessitates the use of benchmarking tools to ensure that webpages meet certain standards. While not a perfect benchmarking tool, Lighthouse was extremely valuable for learning about performance, best practices, accessibility, and Search Engine Optimization (SEO). Lighthouse generates reports on your site by scoring percentages in various categories and offering insight into what you should change to improve your score. This can be as simple as adding alt tags or aria-labels to elements that require them, or as complex as modifying site structure to improve load times. Performance was undoubtedly the most difficult score to improve given the size of the assets that our site displays. In order to improve this, I used code-splitting within React to reduce transfer sizes and improve load times by not importing components right away, and instead waiting until they are needed. This was done through dynamic imports and react.lazy, coupled with the suspense component. Throughout development, I was constantly auditing Mootookakio’ssin with Lighthouse, generating reports to ensure that I was earning excellent scores in all categories (Figure 6). The best part about Lighthouse is that it is built into Google Chrome and can be found in the developer tools panel.
Wait, I thought this was a website?
In addition to scoring your site, Lighthouse also provides a list of requirements for achieving the status of a Progressive Web App (PWA). A PWA is a web app that can be installed onto your device and used offline. While not every feature needs to be available offline, the more you can make installable, the better the user experience will be while viewing the site without an internet connection. Towards the end of development, I realized just how close we were to achieving this status, and I knew how important this level of accessibility would be, especially for our Blackfoot users living on reservations that often do not have access to highspeed internet. We were missing several icons needed to install as an App, as well as a service worker. The icon requirements were easy to satisfy, and the service worker, though slightly more involved, was not overly difficult either. Create-react-app creates a basic service worker for you, and with some slight modifications, it was ready for use. The service worker also provided a cache-first policy when viewing assets, further improving Lighthouse’s performance score for subsequent visits. If you did not know, Mootookakio’ssin is a fully fledged PWA, and can be installed onto your device and used offline. Practically every feature will work, with the notable exception of the Reflectance Transformation Imaging (RTI).
The trials and tribulations of Reflectance Transformation Imaging
Finally, we need to talk about the WebRTIViewer, the tool used to display the Reflectance Transformation Imaging on the Mootookakio’ssin site. RTI is an advanced imaging technique that allows the capture of a subject’s shape and colour, resulting in an interactive relighting of the subject in real time (Figure 7). This imaging method allows users to gain a better understanding and appreciation of the item and its materials by providing a unique experience that reveals the texture and depth of the subject. Due to the complexity of the technique, the use of RTI in our web app required a third-party solution, since constructing a first-party solution would have been incredibly intricate and time-consuming. Fortunately, in my research I came across a single solution, WebRTIViewer, which provides free support for RTI on the web. While finding a web-based RTI solution was fast, its integration and usage within React posed perhaps the greatest challenge I faced during development.
WebRTIViewer is an “HTML5-WebGL viewer for high resolution RTI images (PTM and HSH) and standard images (JPEG, PNG and TIF). It allows the visualization of your image in a standard web page without the installation of any plugin using a recent web-browser” (http://vcg.isti.cnr.it/rti/webviewer.php). Early in development, preliminary testing and preprocessing of images proved that WebRTIViewer’s usage within a standard web page was as simple and straightforward as their specifications suggest. However, the documentation for WebRTIViewer made no mention of its usability within React or front-end JavaScript frameworks in general, and I would have to go outside of its expected use case for Mootookakio’ssin. Using software outside of its specifications will often prove challenging, and integrating WebRTIViewer into a React application was no exception. I made many attempts to encapsulate the WebRTIViewer into a React component but had limited success. No matter what I tried, the canvas used to display the RTI would always fail to render properly within React. To my understanding, the underlying incompatibility between the WebRTIViewer and React is that the scripts required for the WebRTIViewer to function properly cannot be run asynchronously, which means that they must be executed at the time the page is parsing and cannot be run afterwards. Because the DOM within a React App is only generated once, and its contents are replaced through re-rendering in order to display new content, the required scripts for WebRTIViewer did not run, making it incompatible with the structure of React in general. Our options at this point were limited, given that I could not find any alternative RTI solutions for the web, and it did not seem to work with React. Our choices were to drop RTI as a feature or try to find a workaround. Due to the importance of the feature and the beauty of our Blackfoot RTI images, I was dedicated to finding the perfect method that would allow WebRTIViewer to work within React.
During my (extensive) time testing this feature and trying to make it cooperate, I created a WebRTIViewer component that worked in React, but only on a fresh load of the page that displays it, and only when the RTI is immediately rendered on that page without code-splitting. Using this clue, I was able to create a workaround by including the WebRTIViewer in an iframe and encapsulating the iframe within a wrapper component rather than the WebRTIViewer itself. Iframe is a tag used in web development to embed an entire viewable web page (DOM) within another web page. This solution works for two reasons. Firstly, the wrapper component containing the iframe can be subject to code-splitting, allowing the wrapper component to be dynamically imported and only rendered when needed. This means that if RTI has not previously been activated by the user, the RTI wrapper component is not found in the DOM at all, resulting in a sizable performance improvement to the site considering the size and quantity of RTI images required by WebRTIViewer. Once RTI has been selected by the user, the wrapper component containing the iframe will be rendered, and will live on in the DOM until the user navigates away from or reloads the page displaying the RTI. Secondly, iframe renders a totally separate web page, or in our case, a fresh instance of a route in our web app. Because the embedded document within the iframe is essentially a fresh page, it allows the required WebRTIViewer scripts to be run immediately as the contents of the iframe DOM are parsed, allowing the WebRTIViewer to correctly render its content and display the beautiful, interactive RTI feature.
While this solution did allow RTI to work within Mootookakio’ssin, it was not without several shortcomings related to the styling and responsive layout of the of the WebRTIViewer’s default buttons. Essentially, iframe behaves as its own independent viewport for the webpage it is displaying. A viewport is the user’s visible area of the webpage, varying by browser window size, device screen size, aspect ratio, and in this case the size of the iframe. Because the iframe is its own viewport, the CSS rules that govern the layout of the WebRTIViewer (which, again, are based on the media queries related to viewport size) would trigger unpredictably and sporadically relative to the size of the iframe and not to the size of the device/browser window being used to view the site. Thus, navigating between fullscreen RTI and sections of an item’s description would cause the CSS controlling the WebRTIViewer to trigger, even when the size of the browser stayed the same. Of course, we need the CSS to trigger only media queries based on the size of the browser window or device screen size, not the size of the iframe displaying the WebRTIViewer. This was an important issue to resolve given our need to support the RTI feature on both mobile and desktop versions of the app.
After some additional research and experimentation, I discovered a method to utilize the on-click functionality of the WebRTIViewer’s default buttons without relying on the buttons themselves. I was able to reach into the WebRTIViewer through the iframe using iframe.contentWindow and assign the functions found on the default buttons to buttons of my own located inside the RTI wrapper component. These custom buttons share the same functionality as the default buttons, with the advantage of them being relative to the wrapper, not the iframe. Because the buttons were no longer located inside the iframe, they were no longer subject to the CSS of the WebRTIViewer, and were instead controlled by the CSS of the wrapper, allowing for CSS rules to be applied correctly to the mobile and desktop versions of the app. After resolving this issue with responsive layout, the RTI feature was finally complete, allowing users a much closer look at the detail found in the Blackfoot items that support this feature. Without WebRTIViewer, there is no way the RTI feature on Mootookakio’ssin would have been possible. Despite the challenges of integrating WebRTIViewer with React, the end results speak for themselves, and it was definitely worth the effort.
Home again, home again, jiggety-jig
Once the aforementioned issues with the WebRTIViewer were resolved, I realized that all of our features for the website had been finalized, and our goals relating to user experience and expectations had been achieved. Looking back to the early days of planning, it is clear that the development of the Mootookakio’ssin app was not a straight path from our initial ideas to a functional implementation of them. Creating the site as it exists today was a lengthy and complex process, involving experimentation with many concepts for interface design, functionality, frameworks, and web tools that ultimately never made it to the final product. It was important in my development journey to never become too committed to a certain set of functions, design elements, or components, and to remain flexible so that additional features could be added, or alternative features could take the place of those that failed for some reason or another. The dynamic research process of the project and an agile web development strategy allowed us to present the Blackfoot items on the site in accordance with our continually growing understanding of them, and how they ought to be presented from a Blackfoot perspective. This flexibility, coupled with a desire to learn and experiment with new technologies was the key to success. In many ways, the web has become self-sufficient problem-solving machine by providing the tools and resources necessary to propagate its knowledgebase, allowing new problems to be solved, thereby providing additional insight for the problems of the future. This project offered me the freedom to learn and experiment with the React framework, and to discover new and exciting tools like <model-viewer> and WebRTIViewer, all whilst building an important and culturally relevant system for bringing Blackfoot items closer to home. To me, this is the most important aspect of the project. The Mootookakio’ssin system will allow the Blackfoot people, educators, and those with curious minds to experience this growing collection of historical items for years to come, an opportunity they might otherwise never have had.
