I just started building my first Gatsby project yesterday, and I’ve already ran into an issue that I’m having trouble finding existing solutions for. My goal is to have the directory structure of my source directory, which Gatsby is building from, closely mirror the generated site structure. This would include automated directory listing pages.
The issue is that Gatsby is excellent at creating static files on a 1-to-1 basis, but there is much less information out there on how to create virtual pages, especially when they need to be automated based on directory structure or folder content. This is hard to describe, so I’m just going to show you my exact issue:
The Issue:
I’m trying to use Gatsby to create a front-end for a Markdown-based cheatsheet repo I’ve been working on. Let’s say that the Gatsby build will be hosted at “output.test”. The repo is a work-in-progress, but currently the structure looks something like this:
- Cheatsheets
- JavaScript
- js-modules.md
- js-devops.md
- …
- bash-sh.md
- …
- JavaScript
- Random
- roku.md
- index.md
- …
If I follow the Gatsby tutorial on generating a static site based on Markdown (also see tutorial part-7), I’ll end up with a very close-to-usable result, where I can navigate to “output.test/cheatsheets/bash-sh/” or “output.test/random/roku/”. However, what about “output.test/cheatsheets/”? Well, unlike “random”, the “cheatsheets” directory does not have an index.md in it, so no corresponding static index.html file will be generated. On a server with disabled directory listings, this will result in a 404 error or directory listing denied error on trying to access it.
If you have not disabled directory index listings on your server, one option is that you can just use the default directory listing service offered by your server. For example, this is what it looks like on Apache:
However, there are many issue with this. The most important is that it breaks the whole PWA/SPA part of Gatsby – because this “index of” page is not handled by React/Gatsby, it can’t be pre-fetched and rendered by the system, and links to it will be broken. The second issue is that these automatic server directory listing pages are ugly, and not customizable. Here is how it could look a lot nicer, within the Gatsby system, just using default styling:
So how do we create these types of directory listing pages within the Gatsby ecosystem?
Gatsby-Node and exports.createPages
A good starting point for figuring out how to automatically create “index of” React pages with Gatsby, is with the official tutorial! Step 7 is “programmatically create pages from data”, and points us towards using “exports.onCreateNode” to listen for updates in source input (such as the filesystem), and “exports.createPages” to tell Gatsby to create new pages programatically.
That’s all fine and dandy, but again, we aren’t trying to create a 1-to-1 output of static files to input nodes. Here is how I approached this (pseudo code) in gatsby-node.js:
- use
exports.onCreateNodeto listen for MarkdownRemark files / nodes- As they come through, add a node field of “slug”
- So far, this is boilerplate code that matches the tutorial step 7
- CUSTOM: use
exports.createPagesto create directory listing pages- Use GraphQL to get allMarkdownRemark and allDirectory nodes
- Iterate through all the markdown files:
- Create a page for the markdown file itself (same as tutorial)
- Record what directory the page belongs to, and mark that this file is a child of it
- Iterate through all the directories that hold markdown files
- Record which directory the current one is a sub-dir of, and mark as child of it
- Check if the current directory already has an index page, and if it does not, add it to a “queue” of index pages that need to be generated
- Iterate through the queue of index pages that need to be created
- use
actions.createPageto generate the page, and pass a template file, and tell Gatsby to also send along “meta” information about the children of the index page, through context (which gets passed asprops.pageContext)
- use
- Directory Listing Template –
directory-index.js:- Combine the child directories and child file lists that were passed via
props.pageContextinto a combined array. - Use the combined listing array to generate links and icons on directory vs file
- Combine the child directories and child file lists that were passed via
Here it is in action:
Not bad! And that is with very minimal styling or tweaking beyond the Gatsby defaults.
The Code:
This is not the prettiest or most efficient code, but it is a good jumping off point for some approaches on how to handle this. If you are going to try to replicate these result for your own needs, you will probably need to tweak both these files to get things how you want them.
gatsby-node.js:
const path = require('path');
const { createFilePath } = require(`gatsby-source-filesystem`)
const AUTOBUILD_INDEXES = true;
/**
* Implement Gatsby's Node APIs in this file.
*
* See: https://www.gatsbyjs.org/docs/node-apis/
*/
exports.onCreateNode = ({node, getNode, actions}) => {
if (node.internal.type === 'MarkdownRemark'){
const fileNode = getNode(node.parent);
// Programmatically create slug field and value, and append to node to be consumed by page creator
// path should mirror md directory, and should be extracted
let slug = createFilePath({node, getNode, basePath: ''});
if (node.frontmatter && node.frontmatter.customPageSlug){
// replace parsedName with customPageSlug
slug = slug.replace((new RegExp(`\\/${fileNode.name}\\/$`)),`/${node.frontmatter.customPageSlug}/`);
}
actions.createNodeField({
node,
name: 'slug',
value: slug
});
}
}
exports.createPages = async ({graphql, actions}) => {
let subdirsWithIndexPages = [];
let subdirIndexesToCreate = [];
let subdirIndexPages = {};
const result = await graphql(`
query {
allMarkdownRemark {
edges {
node {
fileAbsolutePath
fields {
slug
}
parent {
... on File {
relativePath
base
name
}
}
}
}
}
allDirectory(filter: {sourceInstanceName: {eq: "cheatsheets-md"}}, sort: {fields: name, order: ASC}) {
nodes {
absolutePath
base
relativeDirectory
relativePath
name
}
}
}
`);
function recordAsChild(subdir, child, isDir){
subdirIndexPages[subdir] = typeof(subdirIndexPages[subdir])==='object' ? subdirIndexPages[subdir] : {children:{
dirs: [],
md: []
}}
const target = isDir ? subdirIndexPages[subdir].children.dirs : subdirIndexPages[subdir].children.md;
target.push(child);
}
result.data.allMarkdownRemark.edges.forEach(async ({node}) => {
// Get dir of file
const subdirAbs = path.dirname(node.fileAbsolutePath);
// Note that this md file is child of dir
recordAsChild(subdirAbs,node,false);
// Create page for file itself
actions.createPage({
path: node.fields.slug,
component: path.resolve(`./src/templates/generic.js`),
context: {
// Becomes available as GraphQL variables within page queries
slug: node.fields.slug
}
});
});
// Iterate over directories that contain MD, and check if they need an index page created
let directoryIteratorPromise = new Promise((resolve,reject) =>{
result.data.allDirectory.nodes.forEach(async (node, index, arr)=>{
const subdirAbs = node.absolutePath;
const subdirRel = node.relativePath;
const parentDir = path.posix.dirname(subdirAbs);
// Note child of dir
recordAsChild(parentDir,node,true);
// Create page for subdir that file is in, if it is missing an index page. Skip for homepage ('/'), or top level page (/test.md)
const alreadyHasIndexPage = (subdirsWithIndexPages.indexOf(subdirAbs)!==-1 || subdirRel === '');
if (!alreadyHasIndexPage){
// Check for index.md
const indexPath = path.posix.join(subdirAbs,'index.md');
const existResult = await graphql(`
{
allMarkdownRemark(filter: {fileAbsolutePath: {eq: "${indexPath}"}}) {
totalCount
}
}`);
if (existResult.data.allMarkdownRemark.totalCount < 1) {
console.log(`There is no index for ${indexPath}`);
subdirIndexesToCreate.push({
subdirAbs: subdirAbs,
subdirRel: subdirRel,
parentDir: parentDir
});
}
subdirsWithIndexPages.push(subdirAbs);
}
if (index === arr.length-1) resolve();
});
});
if (AUTOBUILD_INDEXES){
directoryIteratorPromise.then(()=>{
console.log('=============== BUILDING SUBDIR INDEX PAGES! ======================');
for (let x=0; x<subdirIndexesToCreate.length; x++){
const subdirPaths = subdirIndexesToCreate[x];
// Create index page!
actions.createPage({
path: subdirPaths.subdirRel,
component: path.resolve(`./src/templates/directory-index.js`),
context: {
slug: subdirPaths.subdirRel,
hasIndex: false,
meta: subdirIndexPages[subdirPaths.subdirAbs]
}
});
}
});
}
}
src/templates/directory-index.js:
import React from "react"
import { Link, graphql } from "gatsby"
import Layout from "../components/layout"
import Image from "../components/image"
import SEO from "../components/seo"
import {strMethods} from "../helpers/helpers"
function makeDisplayName(slug){
slug = slug.replace(/-/gim,' ');
slug = slug.split(/\//).map((e)=>{return strMethods.toTitleCase(e)}).join('/');
return slug;
}
export default (props) => {
const displayName = makeDisplayName(props.pageContext.slug);
const children = props.pageContext.meta.children;
// Compile listings - subdirectories and files
let listings = [];
listings = listings.concat(children.dirs.map(function(d){return {
path: '/' + d.relativePath,
display: '/' + d.name,
isDir: true
}}));
listings = listings.concat(children.md.map(function(m){return {
path: m.fields.slug,
display: m.parent.name,
isDir: false
}}));
return (
<Layout>
<SEO title={`${props.pageContext.slug}`} />
<h1>Generated Index Page - {`${displayName}`}</h1>
<div className="directoryListingWrapper">
<div className="directoryListingRow">
<i className="material-icons">folder</i>
<Link to={`${document.location.pathname + '/../'}`}>..</Link>
</div>
{listings.map((listing)=>{
return (
<div className="directoryListingRow" key={`${listing.path + ((new Date()).getTime())}`}>
<i className="material-icons">{listing.isDir ? 'folder' : 'description'}</i>
<Link to={`${listing.path}`}>
{listing.display}
</Link>
</div>
)
})}
</div>
</Layout>
)
}
