Joshua Tzucker's Site

What Makes Svelte So Great - Features and Comparisons

Joshua Tzucker — Thu, 07 Dec 2023 07:09:16 GMT

Intro

At this point, many people who have worked with me at my current job know I’m an (unashamed) Svelte fanatic; I spread the word about it constantly, use it on both professional and side projects, and am not afraid to throw jabs at React for falling short of what Svelte delivers.

In order to help spread the word even further, as well have a place I can point people to when they ask me why I like it so much, I’m creating this post as a ~~love-letter to Svelte~~, sorry, I meant to say carefully thought out list of reasons why I like Svelte (and I think you will too!).

Also, I worry it might feel like this post is constantly bashing React or might feel like a Svelte vs React showdown - while there is definitely some of that going on, I want to point out that:

I’m doing it out of a passion for web development and the hope that if we are honest about deficiencies, we can collectively work on addressing them. Competition is good, and React is making some good strides!
I still use React when it makes sense, both professionally and on hobby projects; it isn’t going anywhere anytime soon, and I’m OK with that.
Svelte is not a panacea - if you take poorly written React and re-write it in poorly written Svelte, it is still going to be poorly written. ~~Garbage in, garbage out~~ spaghetti code in, spaghetti code out 🍝

OK, now onto my ~~fawning over~~ careful analysis of what makes Svelte so great.

Performance

To get the ball rolling, I’m going to start with something objective, which is hard to argue against - Svelte performs faster and with less resources than React and many other frameworks. 🔥

This mostly comes down to how Svelte handles reactivity and DOM updates, which it has described as “surgical”. And this is without any hand-tuning or special code; no useMemo, useRef, or having to always be thinking about the render lifecycle - it just works.

If you don’t want to take my word for it, just check out these JS benchmarks. Right now, in December 2023, of the popular frameworks people are probably most familiar with, the rankings are generally (from best to worst): vanillajs -> solid -> svelte -> vue -> angular and react (pretty close together).

Simplicity

This is kind of jumping the gun, as this is the unifying theme of this post and how I wrap it up at the end as well, but if you walk away from this reading experience with only one takeaway about what makes Svelte so great, I think it should be that Svelte is simpler to use.

I’ll end this section with the same text I end this entire post with:

At the end of the day, I value simple, clean code that is easy to write, read, and maintain - for me personally, Svelte is the framework in which I find this the easiest to achieve, with the least amount of time and effort.

Bi-Directional Variable Bindings

Did you hear that sound? Its the sound of tears falling as this section is read by React devs who have to deal with form input listeners (hello formik users), callback hell, painful state refactoring, and everything else that comes with trying to pass values upwards in React and makes it such a chore.

Yes, you read that title right - Svelte support bi-directional data bindings / variable bindings, out of the box. Once again, that is data flowing back up from the child to the parent, not just down.

To paint a picture of what that means and how it saves so much time, imagine you have a form input for a username, and you want to keep your local variable in sync with the input as the user types.

In React, if you don’t use a 3rd party library, you are going to have to write boilerplate to listen for the input value change and then update the variable yourself; React does not support bi-directional bindings:

import React, { useState } from "react";

export const AgeEditor = () => {
  const [userAge, setUserAge] = useState(50);

  return (
    <div>
      <input
        type="number"
        value={userAge}
        onChange={(evt) => {
          setUserAge(parseInt(evt.target.value));
        }}
      />
      <p>Your age is {userAge}</p>
    </div>
  );
};

In Svelte, you can just bind the variable directly:

<script>
  let userAge = 50;
</script>

<input type="number" bind:value={userAge} />
<p>Your age is {userAge}</p>

BUT - that’s not all! If you order within the next 5 minutes, you get bindings for so much more than just <input>s. You get them for ANY property export from a child component. This is huge, as in other frameworks like React, you would either have to pass a state updater as a prop to the child (e.g., a useState setter or your own callback) or use something like useContext (perhaps with a shared reducer) to establish shared state - all this to say that it is far less code in Svelte and much simpler:

Parent component:

<script>
	import AgeEditor from './AgeEditor.svelte';
	let userAge = 50;
	$: userAgeDoubled = userAge * 2;
</script>

<!--
We could have used `bind:userAge={userAge}`, but this is
even _more_ concise!
-->
<AgeEditor bind:userAge />

<p>Your age doubled = {userAgeDoubled}</p>

Child component (AgeEditor.svelte)

<script>
	export let userAge = 50;
</script>

<input type="number" bind:value={userAge} />
<p>Your age is {userAge}</p>

Finally, you can also use bi-directional bindings with entire components, through the bind:this directive.

State, State, State (Did I Mention State Yet?) and Effectful Code

I’m going to be very honest here - I really don’t care for React’s built-in state management solutions and how it treats reactivity in regards to state variables, and I think Svelte’s approach is FAR superior.

In Svelte, you don’t need to do anything special to make a variable “stateful” - it just is.

<script>
	let count = 0;
</script>

<button on:click={() => count++}>
	Count = {count}
</button>

In React (and some other frameworks), this does not work, because regular variables are re-computed every time the component re-renders. So count would get reset to 0 unless you wrap it in a special useState hook.

Here is the same code in React:

import React, { useState } from "react";

const App = () => {
  const [count, setCount] = useState(0);

  return (
    <button onClick={() => setCount(count + 1)}>Count = {count}</button>
  );
};

For a simple example like this, using useState might not look so bad. But once you have multiple pieces of state, and some pieces are derived from other pieces, and there needs to be side-effects that happen when certain pieces change, it starts to become a headache to manage. Especially because of the “recompute on re-render” piece of how React works also means that things like const myFunc = () => {} are also recomputed on component re-render, unless wrapped in useCallback.

All this to say that it is not uncommon to see a single component in React with a bunch of complex useState, useEffect, useCallback, and useMemo hooks, for not that much resulting UX. And these hooks tend to have many easy-to-miss idiosyncrasies that make it a little too easy to write bugs with.

There is a reason why there is a joke that you could rename useEffect to useFootGun

In comparison, again, local variables are stateful in Svelte without any sort of special syntax. And for derived variables or effectful code, it just requires using a reactive declaration or reactive statements, respectfully. There is no need for useMemo or useCallback entirely!

This is far from my best work in either framework, but here is a slapped together example that shows how much more boilerplate and complexity is generally required in React than in Svelte for a reactive UX. Notice how not only does the simplified state management result in less code, but I’m also able to take advantage of bi-directional binding and other shortcuts in Svelte.

Large example, so click to expand:

Click to show / hide React Version

React:

import React, { useState, useEffect, useCallback } from "react";

// Mock function, invalidates coupon because special holiday is over
const checkCoupon = () =>
  new Promise((res, rej) => setTimeout(() => res(false), 1000 * 2));

export const App = () => {
  const [salesTax, setSalesTax] = useState(6.5);
  const [basketItems, setBasketItems] = useState([]);
  const [couponActive, setCouponActive] = useState(false);
  const [subTotal, setSubTotal] = useState(0);
  const [total, setTotal] = useState(0);

  // If we don't use `useCallback`, this function is recreated on every re-render
  const addItem = useCallback(() => {
    setBasketItems((basket) => [...basket, { cost: Math.random() * 100 }]);
  }, [setBasketItems]);

  useEffect(() => {
    let preTaxSum = basketItems.reduce(
      (running, current) => running + current.cost,
      0,
    );
    preTaxSum = preTaxSum * (couponActive ? 0.8 : 1);
    // Note the extra indirection here ^ and below;
    // we can't just assign directly to `subTotal` and reuse for total
    setSubTotal(preTaxSum);
    setTotal(preTaxSum * (salesTax / 100) + preTaxSum);
  }, [couponActive, salesTax, basketItems, setTotal, setSubTotal]);

  useEffect(() => {
    // useEffect does not support making the outer
    // function async, so we have to do this.
    // Another hook foot-gun 🙃
    const verifyCoupon = async () => {
      const verified = await checkCoupon();
      if (!verified) {
        setCouponActive(false);
        console.warn("Coupon is no longer valid");
      }
    };
    if (couponActive) {
      verifyCoupon();
    }
  }, [couponActive, setCouponActive, checkCoupon]);

  return (
    <>
      <label>
        Coupon?
        <input
          type="checkbox"
          checked={couponActive}
          onChange={(evt) => setCouponActive(evt.target.checked)}
        />
      </label>
      <label>
        Sales Tax ({salesTax.toFixed(2)}%)
        <input
          type="range"
          value={salesTax}
          min="1"
          max="50"
          step="0.5"
          onChange={(evt) => setSalesTax(parseFloat(evt.target.value))}
        />
      </label>
      <button type="button" onClick={addItem}>
        Add Item
      </button>
      <p>Item count = {basketItems.length}</p>
      <p>Subtotal = ${subTotal.toFixed(2)}</p>
      <p>Total = ${total.toFixed(2)}</p>
    </>
  );
};

Click to Show / Hide Svelte Version

<script>
  // Mock function, invalidates coupon because special holiday is over
  const checkCoupon = () =>
    new Promise((res, rej) => setTimeout(() => res(false), 1000 * 2));

  // Easy state variables!
  let salesTax = 6.5;
  let basketItems = [];
  let couponActive = false;
  let subTotal = 0;

  const addItem = () =>  basketItems = [...basketItems, { cost: Math.random() * 100 }];

  // Reactive / derived / side-effects
  $: {
     subTotal = basketItems.reduce(
      (running, current) => running + current.cost,
      0,
    );
    subTotal = subTotal * (couponActive ? 0.8 : 1);
  }
  $: total = subTotal * (salesTax / 100) + subTotal;
  $: (async() => {
    if (couponActive) {
      const verified = await checkCoupon();
      if (!verified) {
        couponActive = false;
        console.warn("Coupon is no longer valid");
      }
    }
  })();
</script>

<label>
  Coupon?
  <input
    type="checkbox"
    bind:checked={couponActive}
  />
</label>
<label>
  Sales Tax ({salesTax.toFixed(2)}%)
  <input
    type="range"
    bind:value={salesTax}
    min="1"
    max="50"
    step="0.5"
  />
</label>
<button type="button" on:click={addItem}>
  Add Item
</button>
<p>Item count = {basketItems.length}</p>
<p>Subtotal = ${subTotal.toFixed(2)}</p>
<p>Total = ${total.toFixed(2)}</p>

SFCs are SFC (so-frequently-cool) and Template Syntax vs JSX

For standard Svelte projects, each of your components are going to be built as a separate SFC (Single File Component).

These are composed of:

Any amount of regular HTML
Svelte templating code / syntax (not JSX)
<script> section (imports, exports, logic, assignments, etc.)
<style> section, for styling

It should be noted that you can omit any of the above and there are no constraints on how many root nodes a component can have.

For example, here is a valid Svelte SFC:

<script>
	let name = 'world';
</script>

<h1>Hello {name}!</h1>

<style>
	h1 {
		color: red;
	}
</style>

But, also so is this:

<p>Hello</p>

There are a few features of SFCs that are winners in my book:

Colocation! I find that having all three things - the logic, markup, and styling - all self-contained in one location, per-component, is a huge win for reducing complexity and keeping things clean.
<style> code is automatically scoped to just that component!
- This means you can do something like p { font-size: 30px; } and it only changes the font-size of <p> tags within that file.
- This is a game-changer if you are coming from React or vanilla HTML + CSS
- For many projects, this can eliminate the need for things like SASS, CSS-in-JS, Tailwind, or other libraries that try to help with CSS complexity
A lot of restrictions that exist in JSX / React - such as only having one root node (requiring <Fragment> to wrap if necessary), and having class be a reserved keyword, thus requiring className="" - none of these exist in Svelte’s templating syntax
Valid HTML is valid Svelte code - you can just drop entire blocks of HTML into a Svelte SFC and it should work. The whole thing feels very natural and a shift back towards using the actual HTML standard more 💪

Finally, this might be a personal preference, but I prefer Svelte’s logic blocks to JSX’s use of pure JS, as I feel it results in a flatter structure that is more readable.

Shorthand Syntax Everywhere

A powerful feature of Svelte’s templating / markup language is that it has a lot of shorthand syntax throughout different areas. Some examples:

Using variable name directly, as binding
- In JSX, you often see a lot of code like <Book author={author} />. In Svelte, if the variable name is the same as the prop, you can use a shortcut: <Book {author} />
- This works for bindings too: bind:author instead of bind:author={author}
preventDefault and other DOM event modifiers
- If you have done web development, you might be familiar with event.preventDefault(), to prevent the default event handler from running. Svelte has a shortcut to apply this, and other modifiers, to event listeners: on:eventname|modifiers={handler}
- Example: <form on:submit|preventDefault={handleSubmit}>
CSS bindings
- CSS variables / custom properties have become very powerful and useful in modern web development, and with Svelte, you can set them, inline with your markup, either on entire components with component style props (--my-css-var={val}), or with a style directive (style:--my-css-var={val})
- Classes can be conditionally attached to DOM nodes, with class: bindings. E.g., <div class="expander" class:open> to conditionally combine .open with .expander, based on a local open variable.
And tons of other Element Directives, Component Directives, Special Tags, and more!

Batteries Included

One of the common concerns or complaints I hear when it comes to considering Svelte is:

But there are more libraries out there for React!

True… and yes, sometimes this is a legitimate concern, but this also brings me to one of my favorite things about Svelte, which is that so much stuff is provided out of the box for state management, reactive bindings, form logic, styling, etc., whereas with React you pretty much just get… JSX and hooks.

This isn’t even a comprehensive list, but here are some random freebies that are included with Svelte that I enjoy using:

Per-component automatic CSS scoping (mentioned in the SFC section above as well)
Fancy transitions, animations
global state management / stores
Very strong TypeScript support, but not opinionated about it - with support for per-component opt-in if you’d like
Support for custom elements / web components

Plus, if you use the standard Svelte tooling - such as SvelteKit - you’ll also get:

Hot module reloading (aka HMR)
Advanced rendering options (SSG, SSR, hybrid, etc.)
Routing system, optional eager pre-loading, etc.
Extensible settings / configuration (does not require “ejecting”, unlike CRA, for full control)

Incredible Documentation and Tooling

The Svelte docs are, hands-down, some of the best I have ever seen.

In my opinion, they strike the perfect balance between detail and brevity, between being low-level and high-level, and between being beginner-friendly but also not leaving out the info you need.

📄 I’ve written about the importance of balanced documentation before - see High Level Docs Matter

In addition, Svelte has guided tutorials, live examples, and a free REPL.

It’s Fun to Use, Simple, and You’ll Write Less Code with Less Bugs

I’m ending with this one, because if I put it first, I have a feeling it would turn people off from this post. Saying a framework is “fun” and “simple” is not exactly an objective statement, and honestly usually I get irked when I read something like ”___ is magical”, but… y’all… honestly Svelte IS magical 🎉

Coding with Svelte, above all other UX frameworks I’ve used, is the closest I’ve felt to having my keyboard be an immediate extension of my thoughts - I can go from an idea to functional prototype so fast that it feels like I have a mind-link with my computer (OK, maybe slight exaggeration, but you get the picture).

With other frameworks, like React, I find that I have to add in a step to translate my mental model into the frameworks model for doing things - e.g., I know how I want to make this UX interactive, but how do I get this working with React’s useEffect hook and its other idiosyncrasies?

🐍 If you are familiar with Python and a fan of how it tends to orient itself (with pythonic approaches, the Zen of Python) towards what many would call “natural” or “simpler” coding, than I think you will like what Svelte has to offer.

I would also add that, by allowing for less complicated code that is more readable and comprehensible, Svelte code (on average) likely results in fewer bugs. Or at least bugs that are easier to fix.

Nothing’s Perfect

It feels dishonest to only talk about the good things, so here are some things that are less than perfect when it comes to Svelte.

Reactivity in Svelte Requires a Mental Model Shift

In general, “thinking reactively” in Svelte will take some getting used to, and IMHO, will be the most challenging part of onboarding into the Svelte world for someone new to Svelte.

In particular, Reactive Statements, i.e. the $: {} blocks in a Svelte component, can be tricky and have some “gotchas”.

However, that being said, this one is really not so bad because A) it is nowhere near as bad as it is for useEffect with React (sorry React, but you know its true), and B) The new Runes feature in Svelte 5 should solve this (the gotchas, that is).

The talent pool is smaller

Yes, there are far fewer experienced Svelte devs out there than there are React devs.

However, I would argue that it takes far less time for the average developer to learn Svelte and become productive with it than it does to learn React.

I would also argue that any developer that has taken the time to teach themselves Svelte probably has great taste and admires clean code (looks in mirror, does a cheesy wink to camera 😉).

The Svelte ecosystem isn’t as “rich” as React’s

Sorry Svelte, but yes, this one is categorically true. You are going to find far more libraries for “doing _ in React” than “doing _ in Svelte”.

This is probably the strongest case against Svelte, which is saying something becomes it amounts to a catch-22 of “people aren’t building libraries for your framework because it isn’t popular enough, and we don’t want to help it become popular because it doesn’t have enough 3rd party libraries”.

Sidenote: I’ve never been a fan of the “because its popular” argument. Battle-tested and robust? Sure, great arguments! “Because the popular kids are using it”? Less so.

Reminder: Facebook built React to solve Facebook-specific and Facebook-sized problems. The same idea applies to a lot of frameworks; always think carefully about the actual problems a tool is solving before picking it to apply to your use-case.

Conclusion / Wrap-Up

Conclusion: You are silly goose if you don’t use Svelte.

JUST KIDDING. My go-to sayings are “use the right tool for the job” and “strong opinions, loosely held”, so ultimately I think you should use whatever framework is the best fit for your specific project. Maybe it’s Svelte, maybe it’s React, maybe it’s something completely different like Vue or Angular.

But I also think people are coming around and I’m not alone in this; Svelte has consistently scored highly in developer surveys for being “desired”.

At the end of the day, I value simple, clean code that is easy to write, read, and maintain - for me personally, Svelte is the framework in which I find this the easiest to achieve, with the least amount of time and effort.

Thanks for Reading 👋

I know this was a lengthy post - one could say… it’s not so svelte (ducks tomatoes being thrown, audible boos).

Anyways, thanks for reading and I hope even if you don’t agree with all my points, you at least got something useful out of it.

Shell Heredocs - for Fun and Profit, in Bash and Beyond

Joshua Tzucker — Fri, 17 Nov 2023 23:22:11 GMT

Intro

One of features of shell scripting (e.g. with bash or zsh) that I feel is undervalued is heredocs (aka Here Documents). At their most simple level, heredocs are a way to embed and escape a multi-line block of text within shell code:

cat << "EOF"
This text is inside a heredoc!

I can put pretty much anything I want here, including whitespace and special characters:
  -> $
    -> 🎉
EOF

🔗 Docs: GNU Bash Reference - Here Documents

But wait - that’s not all! - bash heredocs can be so much more! And I’m here to show you why (and how).

💡 A lot of these tips about piping and redirection apply to shell strings in general; it is just that heredocs are the easiest to use for escaping and multi-line purposes.

🤩 On a personal level, there is something that really appeals to me about heredocs, both in shell scripting, and generally across all programming languages. There is something that feels really pleasant about the ability to write self-contained modules that can inline blocks of whatever you’d like, as opposed to having to break everything out into a separate file / component. Of course, you can overdo this too…

The Basics of heredocs

First, a quick lesson in heredocs (or a refresher, if you are already familiar). If you want to, you can also skip ahead to the use-cases / examples.

The general syntax for heredocs in shell scripts is:

[COMMAND] << "DELIMITER"
HEREDOC_CONTENTS
DELIMITER

or

[COMMAND] <<- "DELIMITER"
HEREDOC_CONTENTS
DELIMITER

Breaking this down:

The command is optional
- You often see cat being used, if you simply need to pipe to stdout
Double left redirect operators (<<) are always required
The hyphen / minus (-) after the redirect operators is optional; if you include it, it will allow for and ignore leading whitespace for the HEREDOC_CONTENTS
The DELIMITER string can be anything, but the most common convention is to use the string EOF (end-of-file)
The double (or single) quotes around the delimiter is optional; if used, it will prevent interpolation inside the heredoc
- For example, if you want to print the literal string $PWD without having it evaluated, you would want to quote the delimiter
Finally, the ending delimiter has to match the starting one, should never be quoted (regardless of the starting delimiter), and should never be indented

You can redirect and pipe around Heredocs just like other strings, which makes them very powerful. For example, you can mix interpolation, command substitution, and stdout redirection, all in one go:

cat > tee system-info.txt << EOF

Directory: $PWD
OS: $(uname -a)
EOF

And this is just scratching the surface; read on to see some fun and productive ways to use (or abuse) heredocs:

Fun and Productive Use Cases for Heredocs

Using heredocs to escape and store special text

One of the most common uses for heredocs, although not the most exciting (IMHO), is to use them to escape large messy text strings. Optionally also capturing as a variable.

For example, we might want to produce a string that contains a dollar sign and line breaks, without these causing issues with the shell interpreter / shell expansion. We can do this with a single heredoc, by making sure we quote the leading delimiter:

# Notice that we are quoting EOF to
# prevent expansion / interpretation of contents
receipt=$(cat << "EOF"
Item A: $20
Item B: $40
-----------
Total: $60

Have a great day!
EOF
)
echo "$receipt"

Using heredocs to store and execute non-shell code

This is probably my favorite use of heredocs, and the one that boosts my productivity the most.

Imagine you are working on a project with mixed programming languages (Python + JavaScript with Nodejs, for example) and you have a set of little utility functions that you like to run, but they are personal enough to your workflow that you don’t want to commit them to the shared codebase.

One option is to write your own CLI file, or set of CLI files, and .gitignore them. However, with mixed programming languages in the same project, you might end up with something like do_thing.py, do_other_thing.py, do_js_thing.js, and so on. This might be the approach that most people take, but I find it cumbersome.

However, another option is to inline them - as many different languages and utilities as you’d like - in a shell script:

python << "EOF"
from my_lib import do_thing
print("hello from Python!")
do_thing()
EOF

On the surface, this might not seem that useful, but trust me, this can come in very handy. Especially when you combine it with task runners, like my favorite (task). You can then execute multiple scripts, in multiple different programming languages, all from a single file / CLI ✨:

tasks:
  # Standard shell scripting
  clean: rm -rf ./logs/*
  test: npm run test

  # Python heredoc
  python_util: |
    python << "EOF"
    from my_lib import do_thing
    print("hello from Python!")
    do_thing()
    EOF

  # More advanced python heredoc, with variable from shell
  # NOTE: `{{.CLI_ARGS}}` is specific to `task` - outside of this file, you
  # would usually just use `$1`
  manage:delete_user_by_last_name: |
    cat << "EOF" | xargs -0 python3 my_backend/manage.py shell --command
    from my_app import models
    print("🗑 Deleting user with last name of {{.CLI_ARGS}}")
    models.User.objects.filter(last_name="{{.CLI_ARGS}}").delete()
    EOF

    echo "✅ User deleted!"

  # NodeJS heredoc
  node_util: |
    node << "EOF"
    const { myNodeUtil } = require('./src/utils.js');
    myNodeUtil();
    EOF

Now I can run task python_util to run my python command, task node_util to run my node script, and so on. Single CLI, single file to manage, and as many utilities as my yak-shaving heart desires.

The downside to this approach is that IDE syntax highlighting of code within the heredocs is generally not very good.

Taking Advantage of More Libraries From Your Terminal

This is basically an extension of the above section on mixing other scripting languages into shell scripts - the idea that you can use this approach to pull in more standard libraries (and 3rd party libraries) into your terminal, more than just the regular UNIX commands you might be used to.

Look, I like standard shell commands, the UNIX philosophy, and I get the importance of being comfortable with shell scripting, but at the same time, I think there is no denying that some of the core utilities might not have the best ergonomics. For example, I would argue that both NodeJS’s and Python’s standard library functions for dealing with regular expressions beat grep and sed by a large margin.

With heredocs, it becomes much easier to pull in various standard library functions from different scripting languages directly into your shell.

For example, if we have a JSON file, test.json that matches this format:

{
  "a": {
    "b": {
      "c": [1,2,3]
    }
  }
}

This is all it takes to get the length of the nested array with Node:

JSON_STRING="$(cat test.json)" node << "EOF"
console.log(JSON.parse(process.env.JSON_STRING).a.b.c.length);
EOF

Even if you disagree with me on the usability of some of these commands, I think it is worth considering the following:

Scripting standard libraries are often more portable than shell commands. Many developers have at least Python and NodeJS installed, and both of these languages have stable standard libraries that largely operate the same across OSes and are backwards compatible. For comparison, lots of shell commands have identical names across platforms but act completely different on Mac vs Linux.
- To put it simply, not everyone is going to have jq installed, but everyone with Node has JSON.parse() available.
The knowledge that you build working with a scripting language is more transferable than that of a specific shell command
- Knowing how to use your terminal in general is important, but at the same time memorizing the exact flags to a specific command is going to be less useful than working on your general coding skills
Why not take advantage of a battle-tested standard library, instead of rolling your own?

More advanced example (click to expand)

Here is a more complex, but slightly contrived, example.

Let’s say that I want to read in a JSON file, find any key matching the pattern of user_image_\d+, and then extract out the ID from the key (the digits), and the filename and extension from the string (filename.ext). Sure, I could probably get this done using a combination of cat, grep or sed, and jq, but it is going to be convoluted (capture groups are a pain in both grep and sed) and involve me learning a bunch of syntax that is hyper-specific to jq and will not benefit me in other contexts. For comparison, I can accomplish this easily with NodeJS or Python (this example will be with Node though):

get_user_data() {
	JSON_FILE_PATH=$1 node << "EOF"
const fs = require('fs');
const rawData = JSON.parse(fs.readFileSync(process.env.JSON_FILE_PATH));
const extractedData = [];
Object.keys(rawData).forEach((key) => {
	const userId = /user_image_(\d+)$/.exec(key)?.[1];
	if (!userId) {
		return;
	}
	const {fileName, extension} = /^(?<fileName>.+)(?<extension>\.[^.]+)$/.exec(rawData[key])?.groups;
	extractedData.push({userId, fileName, extension});
});
fs.writeFileSync('extracted.json', JSON.stringify(extractedData));
EOF
}

Using heredocs as real files and virtual files

We’ve already covered piping or redirecting heredocs to commands that are expecting standard input (stdin) , but what about commands that only take files?

The route often taken here would be to pipe the heredoc to an actual temp file that gets written out, then pass it to the command, like so:

TEMP_PATH=$(tempfile)
cat > $TEMP_PATH << EOF
Hello!
EOF
# Get filesize
du -sb $TEMP_PATH
rm $TEMP_PATH

However, with my obsession with seeing how much I can inline into my Taskfiles / shell scripts, I also went to the trouble of figuring out how to do this without needing the intermediate temp file.

There are two main workarounds.

The first is process substitution. We can use <() to get something that acts like a file descriptor (or is one), and pass our heredoc into it:

diff <(cat << EOF
Line 1
EOF
) <(cat << EOF
Line 1
Line 2 - I'm new!
EOF
)

If you are concerned about the readability of the above snippet, you could separate out the heredocs as variables first - I think this makes it more legible, but also adds more lines:

version_a=$(cat << EOF
Line 1
EOF
)
version_b=$(cat << EOF
Line 1
Line 2 - I'm new!
EOF
)
diff <(echo "$version_a") <(echo "$version_b")

However, there is actually one final trick up our sleeve to pass a heredoc to a command that only accepts file paths: we can pipe the heredoc, and then pass stdin as a file descriptor!

cat << EOF | wc -l /dev/fd/0
print("hello from Python")
print("Line 2")
EOF

# This also works
echo | wc -l /dev/fd/0 << EOF
print("hello from Python")
print("Line 2")
EOF

I’m using /dev/fd/0 instead of /dev/stdin. Either should realistically work in most OSes, but /0 and /1 (for /dev/stdout) might be more portable.

This might fall under the “hey is Josh taking this too far?” or “maybe we should cut off his coffee intake” side of things, as this is taking the “how many things can we inline” to the extreme, perhaps beyond where it should. But hey, I get a kick out of this stuff!

Here Strings

So far I’ve exclusively been talking about heredocs / here documents, but it is worth mentioning that there are also Here Strings. They use a triple redirection operator (<<<), but as they can only operate on a single quoted string (aka word), they have more limited utility.

Wrap Up

I hope you found this post useful, or at least entertaining.

If you did, you might also like my my bash / shell scripting cheatsheet.

And on that note,

cat << EOF
✨ Happy coding!!! ✨
   Sincerely, $USER
   $(date +%F)
EOF

Writing Slack Messages with Markdown (Multiple Options)

Joshua Tzucker — Tue, 14 Mar 2023 17:27:10 GMT

TLDR

The short answer to “how do I write a Slack message in Markdown” is that there are two options:

Option A (not very good): Turn your global setting / preference for message formatting to “Format messages with markup” (documented here)
- This is not very ideal, because it is not true Markdown, but rather their own markdown-adjacent syntax, and also requires that you globally disable the formatting toolbar / WYSIWYG (what-you-see-is-what-you-get) editing. More on this in my section on the pitfalls of this setting below.
Option B (my favorite!): There is a cool trick, where you can author your message in full Markdown, in any editor you chose and then get it into a Slack message. You need to first turn it into an HTML clipboard buffer, and then paste that into Slack. You can jump to that solution here, or read on for the background on how this works and why it is necessary.

Background

I want to be very clear: this post is about turning Markdown into a slack message that you send to someone, not the other way around, or anything to deal with their API.

Put another way: this is a solution for if you want to be able to copy and paste rich markdown into Slack and have it actually format the message correctly.

Why? Well,

I like writing in Markdown
For really long async messages, it makes me feel better to write it in a different editor (like VS Code), so the other person isn’t seeing “Joshua is typing…” for multiple minutes while I write (and sometimes rewrite) a message. I want to be able to paste from this other editor, into Slack.

Wait, I thought Slack Supported Pasting Markdown?

Ah, good question. Yes and no, but mostly no. Slack supports their “flavor” of a markdown like syntax (which they are referring to only as “markup”) which is a small subset of the typical Markdown syntax that most developers (like myself) are used to.

This alone wouldn’t be a deal breaker - many of the normal things you want regular Markdown syntax for (links, italics, etc.) are supported.

However - and here is my personal deal-breaker - in order to get the message editor to accept and parse Markdown (and again, not even full markdown, just their version / “markup”), you have to flip a global setting that then removes the formatting toolbar and WYSIWYG editing across Slack and forces you into only using “markup”, 100% of the time.

I haven’t tested this with every subset of their special syntax, but this is at least true for links.

To me, this is the worst of both worlds; I can get some markdown syntax in my editor, but it is an incomplete version of the Markdown spec, and at the same time, if I turn that on, then I can no longer do things like press CTRL + B to bold a section of text directly within the editor.

This post is the result of me searching for an alternative, and finding a “shortcut” that works surprisingly well!

Side note: One of the problems with Markdown itself is that the core original spec isn’t actually used directly all that much anymore; most of the time when people are talking about Markdown, they really mean a Markdown standard that is built on top of the original one - something like CommonMark, GitHub Flavored Markdown, etc.

The Solution / Shortcut

Here it is - the pro-tip - Slack doesn’t work well with pasting Markdown, but it does work well with pasted HTML!

By pasted HTML, I should also be clear that I don’t mean copying and pasting something like <b>Hello</b> with a standard copy-and-paste operation. Rather, it is pasting out of the special text/html buffer that clipboards can contain.

💡 Side note: If you weren’t aware, most operating system clipboards can contain a plain-text buffer, plus additional buffers with various file types, like RTF (rich text format), HTML, and even raw binary data. This is why you can do things like copy and paste images between applications. This isn’t documented very well - I have a WIP blog post about this that I’d like to publish as some point 😀.

💡 You can inspect your clipboard contents with this handy online tool - Clipboard Inspector

💡 This trick actually works with many different applications that don’t accept Markdown but do accept pasting, so it is a great one to keep in your back pocket. In fact, sometimes this even works with programs that act like they don’t support formatting at all, as a way to get things like bolded text when there isn’t a toolbar or keyboard shortcut to enable it.

Converting Markdown to HTML Clipboard Content

For how to convert your Markdown message to HTML and put it in your clipboard, there are a few options.

Personally, I often have VS Code’s Markdown Live Preview tab open as I author Markdown, so I can just copy and paste directly out of the preview window to get HTML to paste into Slack.

For a more automated solution, you can use scripting to A) Convert the markdown to HTML with something like marked, markdown-it, or pandoc, and B) Put the HTML in your clipboard with pbpaste (MacOS), clip (Window), or something like xclip (Linux).

Here is a shell script to do this on MacOS:

markdown_to_html_clip() {
	# Get markdown as arg, or fallback to clipboard contents
	md="$1"
	if [[ -z "$md" ]]; then
		md=$(pbpaste)
	fi
	html=$(npx marked --gfm -s "$md")
	html_hex=$(echo -n "$html" | hexdump -ve '1/1 "%.2x"')
	osascript <<- EOF
		set the clipboard to {«class HTML»:«data HTML${html_hex}», string:"${md}"}
	EOF
}

🔗 Functions for markdown conversion and clipboard manipulation, such as the above, are now in my dotfiles

Credit / Acknowledgments / Wrap Up

I’m definitely not the first person to realize this trick with Slack. Also, in particular the information about how to get HTML back into MacOS’s clipboard was hard to find, and I have this StackOverflow comment and blog post to thank for helping with that.

Also, I realize that my “shortcut” might not be the panacea that readers might be hoping for - obviously the most ideal solution would be for Slack to implement native Markdown support without requiring that it be an all-or-nothing setting; here’s hoping that they are actively working on that.

We Should Care More About Higher Level Documentation

Joshua Tzucker — Mon, 26 Sep 2022 20:57:26 GMT

I am going to reference several projects throughout this post as both positive and negative examples of following the guidance I’m laying out; these are illustrative examples and I hope it is clear that I’m not passing judgement on these projects. Documentation is hard!

Cover Image Photo by Kimberly Farmer on Unsplash

Intro

I’m going to be saying some opinionated things in this post, so to help get you on my side, I want to try a thought experiment.

Pretend that you suddenly need to get up-to-speed on a new framework or library, within a short amount of time and without guidance. Maybe you just got assigned to a new project at work or started a new job. This is minute zero, and you are sitting at your computer, unfamiliar with this new “thing” and looking to get started.

You type “{name_of_thing} documentation” into your preferred search engine and hit enter. You end up on the official documentation website and you are presented with the following three links:

Quick Start Guide
Handbook / User Manual
Complete API List / Low-Level Docs

Which one do you click on? Keep in mind, you are short on time and the pressure is on to get productive with this new tool quickly.

If you are like me, you are smashing that Quick Start Guide link as fast as possible.

Once I’ve started spending some time in the project, I’m probably going to be perusing that Handbook and/or User Manual. Finally, I might reference the full API docs from time to time.

Yet…

What I seem to consistently find is that too many projects focus on documentation in the reverse order. They have beautiful full API docs, generated through some bespoke automated process, they might or might not have a handbook or manual, and the “getting started” guide is often insufficient, outdated, or simply omitted entirely.

This should not be the case.

Thinking of Documentation Levels as a Funnel

I’m not an expert on this, but to me it seems like you can categorize most types of documentation by their technical level and degree of abstraction. E.g., on the high-level end of the spectrum, you have a step-by-step tutorial that a completely novice can follow. On the other end, you have the actual source code and any comments contained within.

I would argue that these levels of documentation are also closely correlated with how long you spend with a given piece of technology, so you can view the levels of documentation not just as a pyramid of abstraction level, but also as a funnel:

I am aware that this graphic is not an exhaustive representation of every type of documentation

Examples of Higher-Level Documentation

Again, I’m calling out these projects as illustrative examples, not to shame them or cast judgement in any way. Also, things change; these projects might have improved their docs since this post was published. Anyways, here are some links to documentation that, in my opinion, neglect the top part of the documentation funnel:

For an example of a project with (again, in my opinion) excellent higher-level docs, check out:

Why More Effort Should be Put On The Early Part of the Funnel

This post started with a hypothetical scenario, in which you are an employee looking to get started ASAP using a new framework or library. I’m hoping that this alone was pretty convincing of why high-level documentation matters, but here are some additional thoughts on the topic.

Jobs Have Changed

It used to be very common for developers to stay in a given job for many years, perhaps even spanning decades. Nowadays, for a multitude of reasons, you see more and more developers only spending a few years at a given company. And for those that stay longer, often they are switching teams and/or projects more frequently than before. With these changes in position often come changes in the developer’s day-to-day tech stack, meaning there are more developers trying to learn new things all the time.

On a related note, even when staying in the same role at the same company, it seems like technology stacks are still changing faster than ever (especially in the web space).

All this is to say that the funnel pictured above has been flipped 180 degrees from how it used to be. With more new developers entering the industry, more projects competing for use, more switching between stacks, and an increased urgency for getting “productive” with new things faster than before, the higher-level parts of the documentation funnel have become a bottleneck and core area of focus.

Your Documentation is a Form of Marketing

Those familiar with the term “Developer Advocate” probably already know this, but documentation is actually a form of marketing for most large projects. This is particularly the case when developers are trying to decide between multiple competing options.

Going once again back to the concept of a documentation funnel, if your high-level documentation is terrible, you might get developers abandoning your project before they have even tried it.

Source Code Is More Accessible Than Ever

This is more an argument for spending less time on the lower-level part of the tech documentation funnel, not necessarily more time on the upper part (although it should free you up to do so). My argument is that access to the actual source code of a given project is easier now than ever before and this access can often mitigate the need for (exhaustive) lower-level docs.

There have been many times where I have had a question that required a deeper understanding of a framework than the docs were able to provide; usually the way this went is I would spend 10+ minutes struggling to find the answer in the official documentation, give up and just open the source code on GitHub, use search + jump-to tools to navigate to the relevant code, and get my answer directly from the code.

In these instances, it often takes me only minutes to find the answer via source code, whereas I could have spent half-an-hour with the documentation and still not had a complete answer to my question.

I’m not saying that you should completely omit lower-level docs (such as auto-generated API docs), but that they are slightly less important than before and your time should be spent accordingly.

Why Are High-Level Docs Often Neglected?

It’s easy for me to say that more quality high-level documentation should be produced, but another thing to have it actually happen. Why is higher-level accessible documentation often neglected?

Writing at All Levels is Tough

The short answer is that writing quality documentation takes time, effort, and ability.

I would argue that writing high-quality high-level (abstracted) documentation tends to actually be more difficult than writing low-level documentation, which is a big part of why it is often neglected. This kind of falls into the same paradigm as the fact that often experts in various fields are not the best teachers - it is one thing to understand something at a very low level, but an entirely different can of worms to be able to translate those concepts and communicate them effectively to an audience with a range of different technical abilities and background.

Exploiting The Rise of Tech Blogs and Community Posts

This idea bleeds into the next section, when I rant about React’s documentation, but my argument here is that it seems like many large projects are neglecting their higher-level docs, because they know that tech bloggers (like myself) and communities like dev.to will pick up the slack and fill in the gaps. Why bother writing a “How to get started with ___” guide, if 500 different bloggers are all going to write their own version of it anyways?

Although I’m generally OK with this for projects that are not very profitable, I have a few issues with it:

It can lead to misinformation
- I have seen a number of community-written guides on various pieces of technology which contain information that is undeniably incorrect, and at times, dangerous.
It is using free labor
- Again, for smaller projects that aren’t turning a profit, I’m OK with this. But for large projects, with millions in funding, this feels super exploitative and just wrong. Hire more tech writers!

Don’t Overdo It

As much as I think focusing on the early part of the funnel is an important part of modern documentation development, I have also seen a fair number of instances where it has gone too far in that direction.

The best example I can think of for this is the official React docs (sorry Dan!). React has often sold itself as a beginner-friendly framework, so it is not that surprising that the docs reflect that (I have even heard stories about bootcamps teaching React before teaching JavaScript…). But I would argue that React has gone too far in focusing on the early stages of the funnel, to the detriment of those using the framework past the “getting started” stage of learning. The majority of their official documentation, even the “API Reference” sections, are written with a high-level of abstraction and tend to gloss over a ton of lower-level implementation details, which is great for getting started with React, but leads to too many developers not being aware of the nuances and (many) footguns that come with React.

I think that part of why this is the case with the React docs brings me to another point about insufficient documentation; I have a feeling that Facebook / React are exploiting the open-source community to write their docs for them for free, especially the lower-level docs. The best documentation I have read on React rendering has all been written by people outside the official React team. If you google things about React beyond the surface level, most of the results are not the official docs. I don’t want to go on too much of a rant here, but Facebook / Meta is basically a money-printing machine and it upsets me that they, and others, are exploiting free labor (although that is far from the worst practices - I’m not a fan of FB if you can’t tell 😉).

Throwing Stones

Look, no one is perfect and writing good documentation is hard. Writing “perfect” documentation that pleases everyone is likely impossible. I’m not perfect, or anywhere close to it, so I know I haven’t always followed my own guidance here. And I don’t expect every project to have the time or resources to do so either.

This post is mostly just some food-for-thought. Happy snacking.

Pytest Productivity - Tips I Wish I Had Known Sooner

Joshua Tzucker — Fri, 12 Aug 2022 13:27:57 GMT

Intro

Pytest is a very powerful test framework / runner for Python, but if you are new to it, the depth and surface area can be a bit overwhelming; there are lots of pytest docs, but once you are past the basics, it can be unclear what features you might be missing out on. One of my favorite types of documentation is a “getting productive with ___” guide or ”___ cheatsheet”, and I feel like pytest could use one - this is not that exactly, but at least lays out a few “thing I wish I had known sooner about pytest” type tips.

My last blog post was about the importance of high-level documentation and was inspired by situations like my experience getting up-to-speed with pytest. The pytest docs are comprehensive, but not the easiest to navigate and could use some improvement “at the top of the funnel” (see linked blog post for more on what this means).

Pytest Real-Time Usage

Similar to many other test frameworks, the default way that pytest acts is in sort of a black-box one-shot approach:

Input: tests, source code, pytest config files
Black box: pytest runs
Output: pytest spits out everything at once - which tests passed or failed, stdout and stderr, etc.

However, this doesn’t have to be how you use pytest; there are different ways you can interact with pytest as it runs and gain better visibility into how your tests are working.

You Can Stream Live Logs

Rather then get all your log messages at the very end of the entire pytest run, you can stream them as they run, with the log_cli live log feature: set log_cli to True (in the config file or via CLI).

By combining this with tee (in *nix shells), you can even see the output in real-time in your terminal while also saving it to a log file:

log_cli=true pytest 2>&1 | tee pytest_output.log

You Can See and Interact With Exceptions As They Happen

The very nature of test runners requires that they catch errors as they happen, but that same feature makes interacting with real-time errors generated by your code, as it gets executed through the test runner, tricky.

For example, if you tell your interactive debugger to break on exceptions and invoke pytest, you are going to end up looking at pytest’s own code, as it ends up handling errors in order to intercept them and determine the pass / fail outcome of a given test.

What I wish I had discovered sooner (it’s a bit buried in the docs) is that pytest exposes some lifecycle hooks that you can “register” functions for and hook into - including one for interacting with exceptions as they happen!

By hooking into the pytest_exception_interact hook, we can interact with an exception as pytest catches it and do whatever we want (including calling a third-party debugger, printing special messages, etc.).

The easiest way to register this hook is through the conftest.py file. Here is an example, complete with optional typings:

""""
@file conftest.py
"""
from typing import Any
from pytest import CallInfo, Item, TestReport

def pytest_exception_interact(node: Item, call: CallInfo[Any], report: TestReport):
	# This is just an example
	# You could invoke an interactive debugger here, call an API, or really do anything you want
	print(
f"""
=====
Intercepted pytest test failure!:
	Node: {node.nodeid}
	Marks: {', '.join([m.name for m in node.own_markers])}
	Duration: {call.duration}
	Exception: \n{report.longreprtext}
=====
"""
	)

You Can Debug at Any Level

There is a tendency when dealing with tests to think of them as running completely detached from your normal environment, and to a certain extent this is (and should be) true. However, that doesn’t mean that you should abandon time-saving tools like interactive debuggers.

Although the above section showed hooking into only exception catching in pytest, the truth is that you can invoke an interactive debugger anywhere in your tests, just like how you would normally do it in the rest of your codebase.

If you use a debugger that needs to establish a connection at the start of the session, like Microsoft’s debugpy, you can hook into the pytest_sessionstart hook to do so before any tests run:

"""
@file conftest.py
"""

from pytest import Session
import debugpy

def pytest_sessionstart(session: Session):
	debugpy.listen(("localhost", 5678))
	print("Waiting for client connection...")
	debugpy.wait_for_client()

Visual Studio Code (aka VSCode), along with pycharm, offers a test GUI that should have a one-click “debug test” button that should eliminate the need for this boilerplate code in most situations.

If you need help getting Python debugging setup with VSCode, I have a guide I’ve written that covers it!

Support for Flaky Tests

A flaky test is one that is inconsistent in whether or not it passes, even if no code has been changed, usually due to some subtle edge-case, accidental dependency on execution order, or timing-related quirk. If you have a flaky test, it should go without saying that the most optimal solution should be to fix it, but sometimes that is easier said than done. In those instances, it is worth knowing that pytest maintains a test cache by default, and has some built-in tools to help with flaky tests.

Some examples:

pytest --last-failed will re-run only the tests that failed in the last run
pytest --failed-first will re-run all tests, but start with the failed ones first
pytest --exitfirst or pytest -x will exit the entire test run on the first failed test
pytest --stepwise will exit on test failure and continue from last failing test next time

There is also an installable plugin that does configurable re-runs all in one command: the pytest-rerunfailures plugin

You Can Get Granular With Running Tests

Pytest has many different ways it can be invoked, including:

For specific files: pytest FILEPATH
For single test functions or methods: pytest "MODULE::MY_TEST"
For pattern matching: pytest -k "SEARCH_STRING"

📄 Full docs on invocation approaches

If you are trying to see which tests are matched against a given invocation command, you can use the --collect-only flag to get back a list of tests that will run, without actually running them.

Certain IDEs, like pycharm and VSCode, have built-in support for running tests via a granular GUI.

Magical Fixtures

Fixtures are a common part of many testing setups, across different programming languages and frameworks, but they are extra special with pytest. I won’t go into detail, but the basic “magic” of pytest fixtures is that they are auto-injected (as isolated copies) based on naming, so a lot of manual setup / teardown can be replaced with a decorated fixture function.

For more information, refer to the official docs.

Conclusion

I hope this helps someone out there! If you are a fan of to-the-point documentation focused on getting things done, you might enjoy my personal documentation / cheatsheet site.

How to Bulk Delete Album or Track Scrobbles on Last.fm

Joshua Tzucker — Fri, 29 Jul 2022 13:32:54 GMT

The solutions in this post cover bulk deleting albums or tracks in general on Tidal, but I’ll use a specific example, outlined below:

Recently I decided to give Tidal a try, which is an online music streaming platform. I signed up and started using a free account to give it a spin. I’ve also been a long-time last.fm user, a service that tracks what you listen to across multiple platforms, so connecting it to my Tidal account was one of the first things I did.

If you came to this page, you might already know this, but the term “scrobble” is what last.fm uses for a recorded play of a song. E.g., if you listen to a song once on Spotify and once on Tidal, but have both accounts connected to last.fm, you would see 2 scrobbles logged.

Unfortunately, I recently noticed that Tidal was “scrobbling” all the advertisements they play when you are using a free account. In fact, it recorded so many Tidal Advertisement (or Advertisement by Tidal) scrobbles that last.fm started saying it was the song I have listened to the most, across all time 😂

“What’s your taste in music?” … “It’s complicated”

Bulk Deleting Last.fm Scrobbles

How to Bulk Delete All Scrobbles of a Given Album in Last.fm

Steps:

Navigate to the album page, but via your profile
- Homepage -> Profile -> Albums -> find the album in the list -> click the three dots to open the extended menu -> Go to Album in Library
- The final URL should look something like: last.fm/user/{USERNAME}/library/music/{ARTIST}/{ALBUM}
- For deleting the Tidal Advertisement plays, it should be last.fm/user/{USERNAME}/library/music/Tidal/Advertisement
Click the link labeled Delete album From library
In the confirmation modal that appears, check the box to confirm that you indeed want to delete all plays from this album, thn click DELETE
You should be good to go!

How to Delete All Scrobbles of a Specific Track in Last.fm

Steps:

Navigate to the track, but via your profile
- Homepage -> Profile -> Tracks -> find the track in the list -> click the three dots to open the extended menu menu -> Go to Track in Library
- The final URL should look something like last.fm/user/{USERNAME}/library/music/{ARTIST}/_/{TRACK}
Click the link labeled *Delete track from library
In the confirmation modal that appears, check the box to confirm that you indeed want to delete all plays of this track, thn click DELETE
You should be good to go!

The Easier Solution for Tidal Advertisement Scrobbles

From a practical perspective, the easiest solution here would be to just upgrade my Tidal account to a paid account 😅

I probably will soon (I’m still evaluating different platforms), but this would still be something that needs to be fixed regardless.

Or, Tidal could fix this on their end. Although I don’t really blame them for not prioritizing a “bug” that only affects free accounts.

The Harder Solution

If you need to delete hundreds of different tracks and/or albums according to some specific logic, you might want to look into scripting a solution with the last.fm API.

YouTube History Bulk Deletion Script

Joshua Tzucker — Mon, 06 Jun 2022 07:20:17 GMT

Simple JavaScript snippet to manage YouTube history in bulk

Nook Simple Touch - Factory Reset & Bypass Network Error

Joshua Tzucker — Mon, 07 Feb 2022 07:59:00 GMT

🚨 WARNING: Due to the same reason why factory reset fails with a network error, if you factory reset your device, it will be unable to be re-registered after it has been wiped, and will require a workaround to become usable again. Also, if you have remapped buttons, deactivate the remapping before factory resetting (mine seemed to stick through the reset, which screwed up the registration skip hack).

I’m currently trying to declutter and came across my old Nook Simple Tough (BRNV300) in the closet. Feeling a pang of nostalgia (the Nook is, IMHO, one of the best designed e-readers, and the fact that it could run full Android was icing on the cake), I powered it up. Noticing that it had a bunch of garbage still installed on it, I figured I should do a factory reset. Unfortunately, I immediately ran into issue.

The normal way to run a factory reset on these devices is to go to Settings, then Device Info, then click Erase & Deregister Device. However, when trying to actually run this, I would quickly get an error message:

Network Error

You are currently not connected to the network. Network connectivity is required to deregister this device.

However, my Nook is connected to the internet, and the connection is working (I can browse webpages, download files, etc.). What gives?

My guess is that Barnes and Noble used to maintain an endpoint for device registration, that did something like track serial numbers against account IDs. With how old this device is, I also think it is likely that this endpoint was likely deprecated, meaning that the device is trying to “phone home” to an address that no longer has a recipient. The developers likely did not code for this kind of thing to happen (they rarely do), and so it simply blames the network connection, when the real error message should blame the endpoint (or better yet, fallback to just erasing the device if “derigestering” fails).

All that being said, here is how to work around this:

Solution(s)

Smash your device with a hammer. Kidding! Here is what you can do

Special Boot Mode

This is the most straightforward approach to factory reset:

Power off the device (press the power button on the top of the rear of the device, select the power off option)
Press the power button to start it back up
Once the screen changes to show the device is starting back up, immediately press and hold the two lower buttons (one on each side) until the “Factory Reset” menu comes up
Press home key to continue, follow prompts

Credit to this XDA Post

The 8 Failed Boots Method

If the above does not work for some reason, there is another approach you can use.

This workaround is so well known by the modding community it has a name you will often see: “The 8 Failed Boots Method”.

It is outlined in multiple places (including this post on XDA), but I’ll summarize below:

Power the nook off by pressing the power button (top button on the back of the device) and selecting the option to power off
Power the nook back on, by pressing and letting go of the power button
As soon as it starts to boot (the screen changes), long press the power button to interrupt the bootup process and shut it down again
Repeat steps 2-3 a total of 8 times to trigger a failsafe
At this point, the device should enter an automatic recover menu, with options for a factory reset

Support Me and My Content

Joshua Tzucker — Sun, 09 Jan 2022 03:04:54 GMT

The number one thing that anyone can do to support me and what I do is… go be kind, ethical, and caring to each other. If everyone did this, the wold would be a much nicer place.

Telling me that my content helped you in some way is also a great free way to show your support - and honestly means a lot to me. You can leave a comment on a individual post, email me, or @ or DM me on Twitter.

Monetary Support

I have enjoyed a decent amount of privilege in my life, and there are a lot of people out there that need your support more than I do, so I feel really strange accepting tips or donations, even when I have poured days or even weeks into a project that others find useful.

However, if you really want to send me a tip or donation, I finally have gotten around to setting up a Ko-Fi account:

Thank You

Finally, thank you, the reader, for visiting my site!

Writing for DigitalOcean - Gatsby Tutorial Series

Joshua Tzucker — Mon, 15 Nov 2021 01:23:28 GMT

I wrote several of the posts in this series, on how to build static web sites with the Gatsby.js framework, working with a Senior Technical Editor to get articles from idea to publication.

Canvas Hit Detection Methods - With or Without Tolerances

Joshua Tzucker — Sat, 25 Sep 2021 23:45:37 GMT

The HTML Canvas element often feels like a web development super power - you can render arbitrary content, take advantage of hardware acceleration (in some cases), and have full control over every pixel under your domain. However, all that power comes with some tradeoffs. One significant tradeoff is the inability to attach event listeners to visual content within the Canvas.

Unlike the regular DOM, where you can capture events at varying levels of granularity, with the Canvas you can only capture events on the Canvas element itself. For using the Canvas purely for output, this is usually fine, but if you need to add interactivity, things get more challenging.

Since it might be hard to visualize what I’m talking about, let’s start with a simple example: you are drawing a cloud on the screen, and when the user hovers over the cloud and/or the edge of it, you want to add some extra effects.

Pure HTML Approach

A refresher on event listeners and the non-canvas approach: in a pure HTML approach, without Canvas, we have per-element event listeners to make this easy:

See the Pen <a href="https://codepen.io/joshuatz/pen/abyRrpe"> Untitled</a> by Joshua T (<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

As you can see, we can use these native event listeners even on SVG elements, like path.

Canvas

Pretty much all intersection calculations with Canvas start the same way. Since we can’t attach event listeners directly to areas of the canvas, we start by getting the coordinates of the mouse cursor / touch point, and then checking if it is inside a hit-box or path.

For checking if the mouse coordinates are inside our area of interest, you have a few different options:

Manual tracking
- If you know exactly which pixels an object occupies (or can determine easily), you can check if the touch point is inside those pixels.
- This approach really only works well and is scalable for bounding-box situations, and does not lend itself well to curved paths, circles, etc.
Canvas point overlap check methods (isPointIn___):
- The context.isPointInPath method
- The context.isPointInStroke method
- These two methods are what you will usually want to use

The rest of this post will mostly focus on context.isPointInStroke, as our demo focuses on detecting intersection between the cursor and a line, rather than an entire shape.

Canvas - Getting Coordinates from a Mouse Event

The first complexity involved with Canvas point methods is that they want coordinates relative to the canvas element (distance from canvas top left), not to the DOM. This is an issue because rarely is it the case that your <canvas> element is going to be taking up 100% of the viewport, so you can’t simply use MouseEvent.clientX, MouseEvent.clientY as inputs into context methods.

Instead, you need to translate those coordinates each time, to adjust for the offset of the canvas element, and optionally for scaling as well.

Here is what that might look like:

/**
 * For a given mouse coordinate, translate to corresponding coordinate within original canvas element
 * @param mouseEvent {{clientX: number, clientY: number}}
 * @param canvasElem {HTMLCanvasElement}
 */
function getCanvasCoords(mouseEvent, canvasElem) {
	const canvasBoundingRect = canvasElem.getBoundingClientRect();
	const scale = {
		x: canvasElem.width / canvasBoundingRect.width,
		y: canvasElem.height / canvasBoundingRect.height
	};
	return {
		x: (mouseEvent.clientX - canvasBoundingRect.left) * scale.x,
		y: (mouseEvent.clientY - canvasBoundingRect.top) * scale.y
	};
}

Demo - Without Padding

Here is a demo without padding / increased hit-box:

See the Pen <a href="https://codepen.io/joshuatz/pen/yLPBJOz"> Canvas - Point Hit Detection Without Padding</a> by Joshua T (<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

You will likely notice that it is kind of hard to get the code to detect your mouse cursor over the cloud edge - the tip of the cursor has to be pretty much exactly over the line. Read on for how to add padding to our mouse collision detection so that our tolerance doesn’t have to be so tight.

Canvas Hit Detection - With a Tolerance Allowance

A minor issue with the demo above, and a common issue with detecting mouse interaction with a path, is that if the stroke of our path is not very wide, it can be hard for a user to actually get their mouse cursor directly over the path. In many applications, there is a tolerance involved with calculating overlap, so that you can still click on or mouseover elements that have very small regions.

To implement this for canvas, many forums and answer sites (like StackOverflow) will give (in my opinion) misleading and overly-complicated recommendations. For example, they might insist that you need to stroke your paths with a larger width before render, but this changes the visual output. Or an answer might involve complex transformation matrices and custom algorithms, which seems like overkill for simple path point intersection detection.

If you want to add a tolerance to isPointIn detection without visually changing the output, here is one of the more simple approaches:

Note: This works for both isPointInStroke and isPointInPath

Step 1: For the path that you want to perform hit detection on, convert methods that operate directly on context, to those that construct a path instance outside of it

For example:

// Before
const ctx = canvas.getContext('2d');
ctx.rect(10, 10, 30, 30);
ctx.stroke();

// After
const myRectPath = new Path2D();
myRectPath.rect(10, 10, 30, 30);
ctx.stroke(myRectPath);
// We still have access to `myRectPath` after it has been stroked

Step 2: Pass the path object to isPointIn___ methods, instead of relying on the current path
- Example: replace ctx.isPointInPath(10, 10) with ctx.isPointInPath(myRectPath, 10, 10)

Step 3: Instead of padding the path with strokeStyle or lineWidth before drawing it onto the canvas, only pad the path right before calling isPointIn___ and then reset it right after

Example:

/*
* ======== Before =======
*/
ctx.lineWidth = 1;

const myRectPath = new Path2D();
myRectPath.rect(10, 10, 30, 30);
ctx.lineWidth = 10; // Pad for bigger hit-target
ctx.stroke(myRectPath);

console.log(
	ctx.isPointInStroke(myRectPath, 12, 11)
);

/*
* ======== After =======
*/
ctx.lineWidth = 1;

const myRectPath = new Path2D();
myRectPath.rect(10, 10, 30, 30);
ctx.stroke(myRectPath);

ctx.lineWidth = 10; // Pad for bigger hit-target
console.log(
	ctx.isPointInStroke(myRectPath, 12, 11)
);
ctx.lineWidth = 1; // reset

Here is an updated demo that uses a much larger lineWidth for mouse detection, but does not visually change what is shown in the Canvas:

See the Pen <a href="https://codepen.io/joshuatz/pen/vYWBgeE"> Canvas - Point Hit Detection WITH Padding</a> by Joshua T (<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

`isPointInPath` vs `isPointInStroke`

In general, use isPointInStroke if you are trying to determine if a point exists directly on or inside the path itself (e.g., on the perimeter of a shape) and use isPointInPath if you are trying to determine if a point is enclosed by a given path (e.g., within a filled shape).

This also explains why the isPointInPath method has options for which fillRule to use for determination, but isPointInStroke does not.

If we wanted our demos to work when the cursor is anywhere inside or over the cloud, instead of only when it overlaps with the border of the cloud, we would need to update our code to use isPointInPath instead of isPointInStroke.

Other Approaches

As is frequently the case with Web Development, the only real limits here are your imagination. There are other ways I’ve seen to tackle this problem that have not been enumerated above, such as:

Stacking fixed position elements on top of the canvas
- Let’s say that you want to track clicks on a button that should appear as part of the UI being presented by the Canvas. Instead of rendering this as part of the Canvas, you could render it in the DOM, as a regular <button> element, and then position it over the Canvas element, which would let you attach event listeners much more easily
Use a full-fledge Canvas library with interaction support
- Some Canvas libraries, like fabric.js, provide flexible interaction tracking and event listeners out-of-the-box

How to Use and Customize Lighthouse - CLI & NodeJS

Joshua Tzucker — Mon, 12 Jul 2021 01:09:27 GMT

Introduction

Google Lighthouse is a free (as in both cost, and open-source) tool for evaluating the performance and capabilities of a website (or web app). Over time its capabilities have been expanded, and the “Lighthouse” product now encompasses multiple categories, including accessibility, 20+ category “groups”, 150+ audits, and hundreds of metrics, timings, and more.

With power, in this case, comes a little bit of complexity. This post covers a bit of the Lighthouse NodeJS package and CLI tool, getting started with using it, and some basics on customizing the output. As well tips as using it in a TypeScript project.

Let’s get started!

Different Ways to Use Lighthouse

Although this post will focus on the particulars of the CLI and a customized integration with the Lighthouse NodeJS package, published on NPM, I wanted to mention that there are many different forms that Lighthouse takes, and systems it is integrated into. Here are some of those:

Lighthouse CI
- This is a wrapper around the core Lighthouse library, meant for use with a CI environment. It exposes an altered CLI tailored to CI, report uploading (and/or serving), an integrated report viewer, and more.
- web.dev has a good introductory guide
- You can also find a pre-built GitHub Action for it
Lighthouse also powers some existing Google tools for measuring web performance, such as:
- PageSpeed Insights
- web.dev/measure
- Chrome DevTools Audits Panel (this is probably how many, if not most developers are first introduced to Lighthouse)
Lighthouse is also integrated into dozens of 3rd party services and projects.
- Some popular examples are WebPageTest and Treo
In addition to using in a NodeJS project, the NPM package can also be used directly via the CLI
- Example: npx lighthouse https://example.com

NPM Package

The “core” of Lighthouse is the GoogleChrome/lighthouse repo, which is also distributed as a published NPM package.

Lighthouse Package - CLI Usage

The Lighthouse package exposes a convenient CLI, which can be used to accomplish many standard Lighthouse related tasks, with or without customization, all from the command line.

The instructions for the CLI live in a few different places, but there are two main starting points:

A subsection of the main README
- Covers CLI syntax and options and usage examples
The configuration docs are also helpful, since the CLI can use a configuration file

If you are going to be using the CLI with a NodeJS project, you should install it as a local dependency, otherwise, you can install it globally with npm install -g lighthouse

A minimal example that shows running Lighthouse and then viewing the results is:

lighthouse https://example.com --view

Lighthouse Package - Customized Usage

Working with the Lighthouse CLI, there is a lot that you can do entirely through command line arguments. You can specify which audits to run, with --only-audits, or categories, with --only-categories, or even emulate a different screen type with --screenEmulation. However, not everything can be specified by argument alone, and at a certain point putting dozens of arguments into a CLI command also becomes unwieldy.

If you are looking to configure Lighthouse to your exact needs, going beyond the default arguments, here are your options.

Lighthouse Config File

If you want to customize Lighthouse to your specific requirements, you should be considering a Lighthouse configuration file.

You can use this to customize pretty much every setting for Lighthouse, and it can be reused across both the CLI and custom code that uses the Lighthouse package.

For example, here is a config file for only generating a PWA and performance report, emulating a mobile form factor:

my-lh-config.js

module.exports = {
	extends: 'lighthouse:default',
	settings: {
		onlyCategories: ['performance', 'pwa'],
		formFactor: 'mobile'
	}
}

To use this config with Lighthouse, you can pass the config file path via CLI:

lighthouse https://example.com --config-path ./my-lh-config.js

Or, you could import the file and pass it to the Lighthouse runner in a custom NodeJS script:

import MyConfig from './my-lh-config.js';

const results = await lighthouse('https://example.com', {}, MyConfig);

Lighthouse Config Files - Presets, Examples, and Extending

Rather than building your config file completely from scratch, you might find it helpful to start with an existing template, or even just completely reuse a preset config that ships with Lighthouse. This is not only an option with Lighthouse, but it is encouraged!

Presets: In the Lighthouse project, the CLI parameter of --preset is meant for specifying a config file by its special preset alias. As of right now, those are perf, experimental and desktop. Each of those actually just maps to the corresponding config file, with -config.js appended, such as perf-config.js

The pre-built config files that ship with Lighthouse can be found in the lighthouse-core/config/ directory. Because each config file follows the conventions of a JS module whose default export is a LH config object, you can use them with the CLI:

lighthouse https://example.com --config-path ./node_modules/lighthouse/lighthouse-core/config/desktop-config.js

… Or in custom JS:

import DesktopConfig from 'lighthouse/lighthouse-core/config/desktop-config.js';

const results = await lighthouse('https://example.com', {}, DesktopConfig);

Don’t forget that you can also extend the default configs.

🚨 Note that this is not done through the config.extends property - that is only for dictating if your config values should be applied on top of the default settings. This value should only ever be undefined or lighthouse:default.

To extend an existing config, you can import and then spread the config object into your own. For example:

const results = await lighthouse('https://example.com', {}, {
	...DesktopConfig,
	categories: {
		...DesktopConfig.categories,
		'installable-check': {
			title: 'Installable Check',
			auditRefs: [
				{id: 'installable-manifest', weight: 1, group: 'pwa-installable'}
			]
		}
	}
});

Custom NodeJS Scripting

Working with Lighthouse outside of the CLI, in a custom NodeJS script, can feel a lot different than the CLI, even though a lot of the same settings are shared. For example, in a NodeJS script, you are responsible for spawning the Chrome instance that will be used to host the Lighthouse execution, and passing the port number to the Lighthouse runner / main function.

There is less documentation about the NodeJS scripting side than the CLI side (part of why I’m writing this post), but I can still point to at least two documentation section to review:

Here is an example of using the Lighthouse package in a custom NodeJS script:

// Main lighthouse runner / fn
import lighthouse from 'lighthouse';

// Required for launching chrome instance
import chromeLauncher from 'chrome-launcher';

// So we can save output
import {writeFile} from 'fs/promises';

(async () => {
	// Launch instance of Chrome
	const chrome = await chromeLauncher.launch();

	// Gather results and report from Lighthouse
	const results = await lighthouse('https://example.com', {
		port: chrome.port,
		output: 'html'
	}, {
		extends: 'lighthouse:default',
		settings: {
			onlyCategories: ['performance']
		}
	});

	// Save report to file
	await writeFile('./lighthouse-report.html', results.report);

	// Kill Chrome
	await chrome.kill();
})();

There are examples like this, and more, in my TypeScript example repo (more on this below).

Flags vs Config

When using the lighthouse() function in NodeJS, the first three arguments are url, flags and configJson. I’ll call configJson just config to make it easier. The delineation between flags and config can be a little murky, because they share some common properties (such as onlyCategories), but the general idea is that flags:

If used, any values will override configJson, and are the highest level of configuration. Similar to what you would pass via CLI
Are less granular than the configuration; used for high-level settings, like report output format
Should (IMHO) be used sparingly

The config, on the other hand, is the main way to configure Lighthouse and is where you will most commonly be passing custom settings (other than the port for Chrome and output options).

Slimming Down Results

One common misunderstanding that I had about Lighthouse was the linkage between scores, audits, categories, and artifacts. It’s tempting to think that if you limit the report to just a single category that the size of the data collected should shrink drastically, but that is not necessarily the case. Each category can pull in dozens of audits, and the score is not based on a single data source - rather it is the composite of multiple audits and metrics.

Related explanation here on #11357, and here on #11885.

If you really need a smaller JSON export from the report, the key is to use onlyCategories and onlyAudits to slim down the collected metrics, and then go a step further and use JavaScript’s delete to trim items you don’t need off the result object.

For example, here is a small cleaner function I put together that focuses on collecting the standard browser Lighthouse report scores, removing a lot of data in the JSON that is no longer necessary once the score is generated:

const AUDITS = {
	FCP: 'first-contentful-paint',
	LCP: 'largest-contentful-paint',
	FID: 'max-potential-fid',
	CLS: 'cumulative-layout-shift',
	TTI: 'interactive'
};

const onlyAudits = Object.keys(AUDITS).map((key) => AUDITS[key]);

/**
 * @param {LH.RunnerResult} result
 * @returns {LH.RunnerResult}
 */
function cleanReport(result) {
	for (const key in result.lhr.audits) {
		if (!onlyAudits.includes(key)) {
			delete result.lhr.audits[key];
		}
	}
	delete result.lhr.stackPacks;
	delete result.lhr.i18n;
	delete result.lhr.timing;
	delete result.lhr.categoryGroups;
	delete result.artifacts;
	delete result.report;
	return result;
}

This is for the core result object, but could be modified to operate directly on the result.report string instead.

TypeScript Support

🚨 Warning: Due to some incomplete types, you might need to set skipLibCheck to true when running tsc with some of these approaches.

Lighthouse Config Types

Although supporting types for the overall module is complicated (more on this in a bit), adding types for just the custom Lighthouse config object takes just a few lines of code, depending on your approach.

First, you need to make the ambient types for the config object available. You can do so by using a triple slash directive:

/// <reference path="./node_modules/lighthouse/types/config.d.ts" />

Or, if you have a tsconfig.json file and prefer this approach, adding that file to the include array. You can add all of the Lighthouse ambient types with:’

{
	"include": ["./node_modules/lighthouse/types/**/*.d.ts"]
}

Once you have those types loaded, you can refer to the LH.Config.Json interface when using LH config objects. For example, in a TypeScript file:

const config: LH.Config.Json = {
	extends: 'lighthouse:default',
	// ...
}

module.exports = config;

If you want to annotate the config type in a regular JavaScript file, you can do that with the support of an IDE like VSCode, which supports advanced type-checking via JSDoc:


// @ts-check
/// <reference path="./node_modules/lighthouse/types/config.d.ts" />

/** @type {LH.Config.Json} */
const config = {
	extends: 'lighthouse:default',
	// ...
}

module.exports = config;

Lighthouse Module Types

Lighthouse is a huge repository, spanning thousands of files, and unfortunately the TypeScript situation is currently not exactly ideal. Although some types are explicitly exported, large parts of the codebase only use internal type annotations (JSDoc / Closure Compiler), and do not publish their types as part of the NPM package. The net result of this is that, in your own code, imported methods from lighthouse might appear half-typed (with lots of any inferred types), or not typed at all.

Luckily, between the types that are exported and what TS can infer, you can pretty much fill in the gaps if you are coding a TypeScript project that uses the Lighthouse package. You can take a look at how some existing TypeScript projects did this, such as Paul Irish’s pwmetrics, or, I’ve also put together a small sample repo to demonstrate: lighthouse-ts-examples. Of particular interest is probably the ambients.d.ts file, which is what provides the types for the main lighthouse() function.

Android WebView - Intercepting and Blocking Requests

Joshua Tzucker — Tue, 01 Jun 2021 21:40:02 GMT

Intro

One of the neat things about WebView on Android is the level of control that you are given over the WebView component. In addition to controlling settings and navigating to URLs, you can actually control the WebView at the network request level, intercepting network calls, modifying content, or even blocking requests altogether.

This post is about how to implement some network request blocking (and modification) with WebView and related classes, such as WebViewClient. Examples will focus on Kotlin, but would apply to Java as well with just a little tweaking.

Where to Hook into WebView Requests

There are a lot of different ways to customize your WebView within Android code; settings, callbacks, WebChromeClient, and WebViewClient.

For those wondering what the difference is between WebChromeClient and WebViewClient, the answer is mostly in separation of concerns. The WebChromeClient class is mostly concerned with long-lived browser-level things, such as the user’s visit history, whereas WebViewClient has more to do with the lifecycle of each individual webpage and things like hooking into when a page has finished loading.

For hooking into network requests, the easiest entry point is by subclassing WebViewClient and overriding two important functions:

These methods both do similar things; they let you, as a developer, intercept and modify how each browser request is handled.

The difference is a little subtle between these methods; although AJAX requests use URLs, the shouldOverrideUrlLoading is really only called for the URL of the webpage itself - e.g. the one that the user types in. AJAX and other requests will not come through this hook, but will flow through shouldInterceptRequest.

Writing our WebView Request Modifiers

Since we will want to use both hooks (shouldOverrideUrlLoading and shouldInterceptRequest), and both have identical method signatures, this is an ideal situation to keep our code clean and have a single function that can be called by both:

fun filterNetworkRequests(view: WebView, request: WebResourceRequest): Boolean {
	// By default, let all requests pass through
	var shouldBlock = false;

	// ....
	// code to determine if shouldBlock should be changed

	return shouldBlock
}

Although we can call this function from both hooks with identical arguments, how we use the response actually changes in each.

shouldOverrideUrlLoading: Returning true will abort / block the URL being loaded, false allows it to proceed as normal
shouldInterceptRequest: To block the request, we need to actually return our own custom response which should replace the actual one. Returning null will allow it to proceed as normal.

Knowing this, here is how we could use our common function within each hook:

override fun shouldInterceptRequest(
	view: WebView,
	request: WebResourceRequest
): WebResourceResponse? {
	val shouldBlock = filterNetworkRequests(webView, request);

	if (shouldBlock) {
		return WebResourceResponse("text/javascript", "UTF-8", null)
	}

	return null
}

override fun shouldOverrideUrlLoading(
	view: WebView,
	request: WebResourceRequest
): Boolean {
	return filterNetworkRequests(view, request)
}

Implementing Blocking Patterns

To take this a step further, I’m going to demonstrate how you could use these hooks with blocklists to prevent any requests from being fulfilled for certain domains or URL patterns.

First, let’s declare our block lists:

val BannedDomains: Array<String> = arrayOf(
	"facebook.com",
	"connect.facebook.net"
)
val BannedUrlPatterns: Array<Regex> = arrayOf(
	Regex("\\.taboola\\.")
)

Next, we can modify our reusable filterNetworkRequests function to take these into account:

fun filterNetworkRequests(view: WebView, request: WebResourceRequest): Boolean {
	// By default, let all requests pass through
	var shouldBlock = false;
	val reqUri = request.url
	val reqUrl = reqUri.toString()

	if (BannedDomains.contains(reqUri.host?.replace("www.", ""))) {
		shouldBlock = true
	}

	for (bannedPattern in BannedUrlPatterns) {
		if (!shouldBlock) {
			shouldBlock = bannedPattern.containsMatchIn(reqUrl)
		}
	}

	return shouldBlock
}

Request Blocking in WebView: A Complete Demo

Since sometimes it helps to see everything put together, here is a complete (although basic) working demo of request blocking with WebView, using the previous steps we built out.

Loading Github Gist: https://gist.github.com/joshuatz/e801edbe9f545ebddb2352f2d1941db3

Svelte Weather App

Joshua Tzucker — Sun, 30 May 2021 13:11:20 GMT

Demo weather app project, using Svelte, TypeScript, and Express to power a web interface with a location-based forecast, powered by the AccuWeather API.

Express - Selectively Using CSURF for CSRF Protection

Joshua Tzucker — Fri, 28 May 2021 12:03:13 GMT

Intro

This post is going to be talking about using the csurf CSRF library with Express.

More specifically, it is going to be talking about approaches for selectively applying CSRF protection to only certain routes, and how to dynamically extract the generated token during a request without enforcing CSRF for that request. I’ll start with the approaches that don’t allow for token extraction in non-protected routes, before getting to those that do.

Approaches Without Extraction

Separate Router

Starting with the approach recommended by the official docs, one way to disable CSRF protection on a subset of routes is to use a separate Express Router and mount it to the main app / express instance before attaching the CSRF middleware.

Again, the docs have a full code example, but a quick rundown:

import csrf from 'csurf';
import express from 'express';

const app = express();

function getNonProtectedRouter() {
	const router = new express.Router();
	// Non-protected routes...
	return router;
}

// Mount non-csrf protected routes first
app.use('/nonProtectedRoutes', getNonProtectedRouter());

// Now add middleware
app.use(csrf({cookie: true}));

// protected routes...

Note: With this approach, the non-protected routes do not have access to req.csrfToken().

Passing the CSRF Middleware Via Route Argument

Often, the CSRF middleware is applied by passing it very early on to the global express instance. Something like:

const app = express();
const csrfInstance = csrf({cookie: true});

// Add all middleware
// ...
app.use(csrfInstance);

// protected routes...
app.post('/newUser', (req, res) => {
	// ...
});

However, you can also pass middleware as an argument to each route, instead of at the app level. This lets you apply it to only some routes:

const app = express();
const csrfInstance = csrf({ cookie: true });

// protected route:
app.post('/newUser', csrfInstance, (req, res) => {
	// Protected by CSRF!
});

// Unprotected route
app.post('/checkPublicUser', (req, user) => {
	// No CSRF check here!
});

Note: The unprotected routes do not have access to req.csrfToken(), similar to the separate router instance approach. Theoretically, you could however combine this approach with two separate csurf instances (like this user) - one that ignores all methods and one that doesn’t - and use the ignoring one in routes where you want to extract the token, but not enforce it.

Selectively Calling with Wrapper Middleware

Since Express allows you to create your own middleware without much setup, it is relatively pain-free to create a “wrapper” middleware that only calls the csurf middleware when you want it to, otherwise just passing the request along.

For example:

const app = express();
const csrfInstance = csrf({ cookie: true });

// Custom middleware - wrapper around csurf
app.use((req, res, next) {
	if (req.path === '/checkPublicUser') {
		// This skips CSRF - route will be unprotected
		return next();
	}

	csrfInstance(req, res, next);
});

Approaches with Extraction

Using GET and ignoreMethods

The default value for the ignoreMethods option is ['GET', 'HEAD', 'OPTIONS'], which, as you might have guessed, actually means that CSRF is not enforced for any of those request types.

So, unless you have modified this option, you can actually use any GET route without CSRF tokens being checked. The token is still generated during these requests and accessible via req.csrfToken() - in fact, that is how you are supposed to pass the token to the front-end in the first place; your GET route passes the token to a view, which embeds it into the page (usually via hidden meta or input tags).

Hooking Into Next and Ignoring Errors

This method is one that I realized could be used, but didn’t find in most code examples.

As csurf is a form of Express middleware, it shares the same interface as all Express middleware - a function that takes arguments req, res, and next. Normally, when you attach the middleware via use(), Express will automatically be calling the middleware for you on each request, with those three arguments.

However, we can also manually call the middleware, even inside of our own middleware. And, if we pass our own function as the next argument, we can intercept the result from the csurf middleware, choosing to ignore or rethrow errors it generates while checking tokens.

By combining this idea with the wrapper middleware approach, we have a way where we can run the CSRF middleware for every request, which means all requests will have tokens via req.csrfToken(), but ignore the checking of tokens when we want to.

Here it is in action:

const app = express();
const csrfInstance = csrf({ cookie: true });

// CSRF protection, with excluded route that still captures token
app.use((req, res, next) => {
	csrfInstance(req, res, (err) => {
		// Within this function, and now within all routes that follow,
		// we have access to req.csrfToken()

		if (req.path === '/nonProtected' && e.code === 'EBADCSRFTOKEN') {
			// We will ignore the specific CSRF error for this specific path
			// Notice how we are not passing err back to next()
			next();
		} else {
			// Pass request along normally
			// If there is an error from CSRF or other middleware, it will be rethrown,
			// instead of being ignored
			next(err);
		}
	});
});

// Routes go here...

📄 According to the docs, you can rely on ‘EBADCSRFTOKEN’ to be the error code.

Info on WaveShare Passive NFC-Powered E-Paper Modules

Joshua Tzucker — Thu, 20 May 2021 22:59:25 GMT

This post comprises my assorted notes related to WaveShare Passive NFC-Powered E-Paper Displays. I recently bought a 2.9” module and ended up learning a good deal about it while building an Android app to interface with it.

Resources

Show Resources

WaveShare Store
- 2.9” Passive NFC e-Paper
- 296x128 2.9” E-Ink Display Module
WaveShare Wiki
- Homepage
- Images tagged “NFC”
Developer tools:
- Official
  - Android / Java SDK
    - Note: This is a bytecode / compiled JAR file, not raw source code
  - Android App (APK)
  - *ST25R3911B-NFC-Demo (C code)
    - Related to this WaveShare board
- Unofficial
  - Proxymark3
  - Examples using the JAR file:
    - joshuatz/nfc-epaper-writer (my app! 😀)
    - j796160836/EinkTagDemo
    - thomasleplus/android-nfc-timestamp
Blog posts
- The Digital Reader - *Hands on With Waveshare’s NFC-Powered E-Ink Screens
General WaveShare E-Ink Projects and Libraries
Other
- Web-NFC Explainer

What is Powering the Waveshare NFC EPaper Module?

Hard to tell - probably depends on model.

It looks like many use the ST25* (ST Microelectronics) based hardware for the NFC detection and reading. For example, the ST25R3911B board.. But, even more likely, they are specifically using the ST25 Dynamic chips, since those can support fast transfer mode, I2c, and energy harvesting.

Some models might also be using the ST M24LR for energy harvesting.

My Teardown

I was really curious on how these worked, so I took my 2.9” module apart and took some pictures of the circuit board. Disassembly actually made things more confusing, not less.

To begin with, the main integrated circuit that obviously handles the bulk of the operations has had its label etched out - so you cannot discern model number, or even the manufacturer. It also is a 40-pin chip, which strikes me as odd, as 40-pin chips appear far more rarely than 44-pin chips. However, there is a package type that might match - the HVQFN-40 (or for NPX, SOT618-1). Using this package type to narrow down options makes it seem likely that this is actually a NXP chip, because as far as I can tell, they are the only ones with NFC chips that comes in 40-pin packages - for example, the PN512 or PN532.

Teardown - Image Gallery

What NFC standards does it use?

These are based on my observations, with a 2.9” passive display

The answer is complicated.

It’s a Type-2 Tag (aka NFC Tag Type 2, aka NFC Forum Type 2), which is covered specficially by ISO/IEC 14443-3A (aka ISO14443A). Falls under NFC-A spec.

However, the way to talk to this tag type is also covered by the NFC Forum Digital Protocol Technical Specifications, which further details things like command sets (such as WRITE and READ), error checking, etc.

Although the data set maximum message size is reported at 46 bytes, the Capability Container (CC) reports a maximum NDEF data size of 2016 bytes, which would match the maximum allowed by the type-2 spec. ST25 app, which is probably more accurate, reports 2032 bytes.

Although the AAR uses NDEF, the actual image memory stuff appears to use lower-level NFC stuff, so you can’t use Web-NFC or anything else that only speaks NDEF. (see this, and this).

Capability Container

AFAIK, the Capability Container (CC) is basically a very specific area of memory for NFC chips that contains metadata about the capabilities of the chip itself. Thus, I believe it is usually programmed / written / burned in at the factory, and written as “read-only” memory.

Data Format

Stored Data

Reading the raw data off the chip, via NFC Tools, TagInfo, or the ST25 app - all of them say the same thing: that the only data on the NFC chip is some meta information, and the single NDEF AAR entry. The hex is as follows:

57:53:44:5A:31:30:6D:FF
FF:FF:00:00:E1:10:FC:00
03:27:D4:0F:15:61:6E:64
72:6F:69:64:2E:63:6F:6D
3A:70:6B:67:77:61:76:65
73:68:61:72:65:2E:66:65
6E:67:2E:6E:66:63:74:61
67:FE:00:00

The first 7 bytes (57:53:44:5A:31:30:6D) are WSDZ10m in ASCII, which is not random; this is the UID (Unique ID), which belongs the specific manufacturer and/or product (I’m sure the WS in the string is for WaveShare). In this case, the UID is a “double size UID”, which is 7 bytes.

Keep in mind: Not all NFC tags use UIDs (some use NUIDs), and not all are the same byte length - it can differ, but some are well-known or can be determined. For Type 1, 2, and 4 tags, this is governed by ISO/IEC 14443-3.

Troubleshooting

Flashing Issues

I ran into this with both the official WaveShare Android App and the one that I built with their library - flashing new content to the module tends to be really touch-and-go, and it often takes multiple tries of holding your phone against the module just right.

I came to these conclusions with my specific phone (a Pixel 4a), so your experience my differ, but here is what seems to work best for me:

Not holding the phone flush against the display, but with a tiny gap (about 0.25 - 0.5 inches) away.
The less data you are pushing, the more likely the flash is to succeed
Try to align the NFC transceiver in your phone with the circuit board behind the display of the module.

Problem: NFC tag contains “deeplink”

One issue I ran into very quickly is that the very first NDEF Record is an AAR (Android Application Record), which Android treats as a high priority signal to open the corresponding app.

type: "android.com:pkg"
data: "waveshare.feng.nfctag"

As constructed with JS (Web-NFC), it would look like this:

const encoder = new TextEncoder();
/** @type {NDEFRecordInit} */
const waveshareAAR = {
	recordType: 'android.com:pkg',
	data: {
		records: [
			{
				recordType: 'text',
				data: encoder.encode('waveshare.feng.nfctag')
			}
		]
	}
};

With an AAR, the only way to prevent the default app from taking over (or opening the Play Store to locate it, if not installed) is:

Web-NFC: Start an active scan, and handle the read event
Android: By intercepting the intent with the foreground dispatch system
Removing the AAR

NFC E-Paper Writer Android Application

Joshua Tzucker — Thu, 20 May 2021 22:54:31 GMT

Native Android application (Kotlin and Java) to update WaveShare Passively Powered NFC E-Paper Modules over NFC, and generate bitmaps for them.

WebViewAssetLoader and WebViewClient with Kotlin

Joshua Tzucker — Mon, 10 May 2021 16:10:33 GMT

Intro

For fun, I’m trying to hack together a little Android app for myself. Because it potentially will use low-level system calls, I wanted to try my hand at building the app with a native Android approach - Kotlin, Java, and Android APIs - as opposed to React Native or Flutter. This is a considerable challenge to take on, given that up to this point I have had zero experience with Java, Kotlin, or native Android development.

This post is about a particular challenge I ran into - trying to use WebViewAssetLoader and WebViewClient to get around the ERR_ACCESS_DENIED issue with local HTML files.

Issue

My scenario is fairly standard - I have some local HTML files that I want to load into a WebView. The first attempt, loading via file:///android_asset/my-file.html failed with ERR_ACCESS_DENIED.

At first glance, the easy way to solve this seems to be through one or more of the following (depending on API level) (based on these responses):

setAllowFileAccess(true)
Adding android:usesCleartextTraffic="true" to the manifest

However, after further digging, it became clear that both of the above options are not best practice, and actually reduce the security of our app. The better approach, which is mentioned in the setAllowFileAccess docs, is to use androidx.webkit.WebViewAssetLoader.

However, the docs for that class only show examples in Java, even when you select Kotlin as the language. Given that I had no experience with Java, and they were using more advanced syntax, this is where I got stuck for a tiny bit (and is the reason for this post).

Solution

Installing the WebViewAssetLoader Dependency

First, before even working on the Kotlin code, something that I had to figure out was that WebViewAssetLoader is an extra dependency - it is part of androidx not android. Note the import path:

import androidx.webkit.WebViewAssetLoader

So, before moving on, go ahead and install the required dependency. You can do this in one of two ways.

Recommend: Use Android Studio: Project Structure -> Dependencies -> Add Dependency (+) -> Library Dependency and then search for and add androidx.webkit
Alternatively, you can add the dependency directly to your gradle file. For example:
- implementation 'androidx.webkit:webkit:1.4.0'

WebViewAssetLoader + WebViewClient

OK, so now that we have the dependency installed, how do we go about actually using WebViewAssetLoader with a webview? In trying to re-build the example from the docs, I got tripped up on some fancy syntax and how to emulate it in Kotlin:

// Java code
webView.setWebViewClient(new WebViewClient() {
	@Override
	public WebResourceResponse shouldInterceptRequest(WebView view, WebResourceRequest request) {
		return assetLoader.shouldInterceptRequest(request.getUrl());
	}
	// ...
}

Coming from web development, that syntax for both extending and initializing a class instance in the same section (and as an argument!) is something I had rarely ever seen before. In Java, this is known as Anonymous Class syntax.

To replicate this same syntax in Kotlin, there are several options, but the important part is mostly the object : MyClass() { } syntax..

Here is a consolidated example, showing an activity that uses WebView with a local HTML file, via WebViewAssetLoader:

MyActivity.kt:

package com.joshuatz.demo

import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import android.webkit.WebResourceRequest
import android.webkit.WebResourceResponse
import android.webkit.WebView
import android.webkit.WebViewClient
import androidx.webkit.WebViewAssetLoader

class MyActivity : AppCompatActivity() {
	override fun onCreate(savedInstanceState: Bundle?) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_my_activity);

		val webView: WebView = findViewById(R.id.myWebView);

		// Setup asset loader to handle local asset paths
		val assetLoader = WebViewAssetLoader.Builder()
			.addPathHandler("/assets/", WebViewAssetLoader.AssetsPathHandler(this))
			.build();

		// Override WebView client, and if request is to local file, intercept and serve local
		webView.webViewClient = object : WebViewClient() {
			override fun shouldInterceptRequest(
				view: WebView,
				request: WebResourceRequest
			): WebResourceResponse? {
				return assetLoader.shouldInterceptRequest(request.url);
			}
		}

		webView.loadUrl("https://appassets.androidplatform.net/assets/my-file.html");
	}
}

💡 There is another important change between using WebViewAssetLoader and standard WebView - notice how the path changed fromfile:///android_asset/... to https://appassets.androidplatform.net/assets/.... The new domain is less important (it’s a non-existent subdomain), but the /assets/ is very important for our WebViewAssetLoader path handler - it is how it matches the URL.

Globlist Packer - NodeJS Powered File Packing Tool

Joshua Tzucker — Thu, 29 Apr 2021 01:54:52 GMT

CLI and module for packing files into directories or archives, based on ignorelist files and glob patterns. Built with TypeScript and NodeJS, and highly configurable.

Using TypeScript with WordPress Gutenberg Blocks

Joshua Tzucker — Sat, 17 Apr 2021 10:42:55 GMT

Intro

For those unfamiliar, Gutenberg Blocks, or now known simply as Blocks, are the backbone behind the new reusable component-driven paradigm for WordPress.

For more info on Gutenberg, you can check out WordPress’s landing page.

Now, if you are reading this, there is a good chance that you are a WordPress developer, maybe even already familiar with Blocks and how they work. There are lots of dev guides out there on how to use Blocks (including my own notes, here), but I was recently having some issues using Blocks with the TypeScript programming language (which gets converted into JavaScript) and found much less out there to guide my way.

This post covers some steps I had to take to get TypeScript working with WP Gutenberg Blocks, as well as some issues that I ran into.

Steps

The first step, whether or not you end up using TypeScript, I would recommend to be to install and start using wp-scripts (source code, docs) for orchestrating your build setup. You don’t have to do this, but it makes things a lot easier while still being extensible via configuration files and CLI arguments.

Now, assuming that you are using the wp-scripts, or something similar, here are the general steps I had to take to get TypeScript working

Install TypeScript and ts-loader locally
- npm i -D ts-loader
- npm i -D typescript
Create a tsconfig.json file and adjust it (see below)
Modify the WebPack config to use ts-loader (see below)
Install types for packages that distribute them separately (i.e. through the @types monorepo)
Ambient type-declare for packages that are missing them (see below)

TSConfig

Since your WebPack config will override a lot of settings of your TSConfig, there is less to worry about, but still some settings that really matter.

{
	"compilerOptions": {
		// Note: Webpack config overrides this anyways
		"outDir": "./dist",
		// React compatibility
		"jsx": "react-jsx",
		// We want ESM supported in the output, because WebPack is going to bundle this anyways,
		// and we want the tree-shaking support and some other benefits
		"module": "ESNext",
		"target": "ES5",
		// == Now for some settings that are not *strictly* required, but I recommend:
		// Embrace strong-typing!
		"strict": true,
		// Not strictly necessary, but nice
		"allowSyntheticDefaultImports": true,
		// Recommended
		"moduleResolution": "node",
	}
}

Modifying the WebPack Config

📄 WebPack has a guide on using TypeScript with WebPack.

For actually generating our final code that will be distributed, we are still going to be using WebPack, not the TypeScript compiler (tsc). However, we have to tell WebPack about the TypeScript code we have added and give it a tool to load it - that is why we added ts-loader as a devDependency.

Luckily for us, wp-scripts not only allows us to use a custom WebPack config, but also exports the base config so that we can extend it instead of replacing it.

webpack.config.js

// This comes from https://github.com/WordPress/gutenberg/blob/trunk/packages/scripts/config/webpack.config.js
const wpConfig = require( '@wordpress/scripts/config/webpack.config' );

// Modified config
module.exports =  {
	// Since whatever object we export from this file will *replace* the default config, we need to
	// make sure to constantly pull in the default values from WordPress's config file
	...wpConfig,

	// Point to your main file, wherever it is, which should now be TS
	entry: `./src/index.ts`,

	// We need to add a new rule to process `.ts` and `.tsx` files with `ts-loader
	module: {
		...defaultConfig.module,
		rules: [
			...defaultConfig.module.rules,
			{
				// Notice that this regex matches both `.ts` and `.tsx`
				test: /\.tsx?$/,
				use: [
					{
						loader: 'ts-loader',
						options: {
							// You can specify any custom config
							configFile: 'tsconfig.json',

							// See note under "issues" for details
							// Speeds up by skipping type-checking. You can still use TSC for that.
							transpileOnly: true,
						},
					},
				],
			},
		],
	},
	// This merges the extensions setting with whatever WP has in their config, but also adds TS
	// extensions, and puts them first, to indicate preferred resolving over .js files
	// @see https://webpack.js.org/configuration/resolve/
	resolve: {
		extensions: [ '.ts', '.tsx', ...(defaultConfig.resolve ? defaultConfig.resolve.extensions || ['.js', '.jsx'] : [])],
	},
}

Issues

Other than getting the configuration setup, there were a few issues that I ran into that I feel would be helpful to share.

Untyped packages

Some WordPress packages offer type definitions (either bundled directly, or, very frequently, distributed via @types, such as @types/wordpress__blocks). However, there are still a bunch that are completely “untyped”, which will throw all kinds of errors in TypeScript, especially if you have strict rules on.
To address this, you either need to wait for types to be added upstream, do it yourself and make a PR to merge them (help the open-source community!), or, if you are in a rush, you can augment the types by using ambient declaration files.
- I’m not going to go into detail on how to do that here, since it is complicated, but I do have a comprehensive section on this in my TypeScript cheatsheet!

TS-Loader: Refuses to Stop Type-Checking node_modules

One frustrating issue that I ran into was that I simply could not get my build command to stop type-checking node_modules - this is a big deal because A) it slows everything down, and B) it actually finds errors and will throw an exception. But it’s not my code!

Trying all the common settings had no effect, such as explicitly excluding node_modules through tsconfig, excluding through the WebPack config, and passing explicit inputs.

However, I did eventually find the culprit, as well as two solutions.

The culprit for me was that I was using "checkJs": true in my tsconfig.json file, which apparently causes ts-loader to ignore the exclude rules and start checking node_modules. Although this issue is marked “Closed” on Github, I can assure you it still exists (even with TypeScript version 4.2+).

Solutions:

Recommendation: Run ts-loader with the transpileOnly setting changed to true.
- This stops WebPack from type-checking your code entirely, which means that you should run tsc separately with tsc --noEmit when you want to type-check
- This also speeds up build and reload times!
You can create a separate TS config file just to use with ts-loader, such as tsconfig.webpack.json, which has "checkJs": false
- Instead of copying and pasting the other settings, you can use TSC’s support for extends
- Pass the selected config to ts-loader via the options.configFile setting

Using TypeScript Changed The Output Filenames

The last interesting issue that I encountered was that changing the entry point of my WebPack config to point to a TypeScript file changed all the output filenames from index.* to main.*. I’m not sure exactly what caused this internally (my guess would be something hidden in the WordPress Webpack base config that handles chunk names), but the fix was easy enough.

Just make sure that in the PHP side, when you register and enqueue assets for your block, you use the updated filenames that show up in the build output.

If you really want to keep the original filenames, you can override the output settings in the WebPack config: see how this sample repo solved it.

If you are looking for more resources on TypeScript + Blocks, there aren’t many guides out there, but this blog post by Kazmierczak is a nice succinct explanation, and they also have a sample repo.

Feel free to let me know if I’ve missed anything or you have tips of your own to share! Hope this helps someone!

HTML Canvas - Filled Quarter and Partial Circles with Arc

Joshua Tzucker — Thu, 25 Mar 2021 07:14:55 GMT

Quick post today - I thought I would share some tips on drawing rotated partial circles (e.g. quarter-circles, or slices) with HTML Canvas and JavaScript. It is actually slightly harder than one might think, as there are some “gotchas”.

Things to Note

Here are the two things that I ran into that made me pause when trying to do this:

The Canvas Web API uses radians for angle measurement, not degrees.
- If you are new working with Canvas, this might be easy to miss, as the function arguments are often just named things like angle or startAngle, not indicating units unless you read through the docs
If you are trying to create a filled segment of a circle (a.k.a. a slice), you need to start an explicit path before drawing your arc, and you need to make sure to set the starting point at the arc origin, even though you also pass that origin to the .arc() method.
- Use ctx.beginPath() method to start the path
- Use ctx.moveTo() method to move “cursor” / starting point, and explicitly start sub-path
- Technically, I believe you could just use moveTo() without beginPath(), since moveTo() starts a new sub-path automatically

Wrapper Function - Quarter Circles

Putting these important pieces together, I created a simle wrapper function that I can call whenever I want to draw a quarter circle:

const drawQuarterCircle = (origin, radius, startAngleDeg = 0) => {
	const startAngle = startAngleDeg * (Math.PI / 180);
	const endAngle = startAngle + Math.PI * 0.5;
	ctx.beginPath();
	ctx.moveTo(origin.x, origin.y);
	// Draw actual arc
	ctx.arc(origin.x, origin.y, radius, startAngle, endAngle, false);
	ctx.fill();
}

startAngleDeg is in normal degrees, to make conceptualizing what you are drawing a little easier.

startAngleDeg is also what controls the rotation of the rendered quarter circle. See demo below for a clear use-case

And here is an embedded demo, showing how to use it to draw rotated quarter circles:

See the Pen <a href='https://codepen.io/joshuatz/pen/OJWymdm'>Canvas - Fill Quarter Circles</a> by Joshua T (<a href='https://codepen.io/joshuatz'>@joshuatz</a>) on <a href='https://codepen.io'>CodePen</a>.

Arbitrary Partial Circles

To draw arbitrary slices of a circle, you can adapt the above code; startAngle and endAngle are the arguments of the .arc() method that you want to use to control the slice of the circle that gets rendered.

Importance of Setting The First Point on the Path

Just for fun, here is what happens if you forget to explicitly set the first point of your path as the origin point of the arc:

See the Pen <a href='https://codepen.io/joshuatz/pen/yLgeoOB'>Canvas - Arc With and Without moveTo()</a> by Joshua T (<a href='https://codepen.io/joshuatz'>@joshuatz</a>) on <a href='https://codepen.io'>CodePen</a>.

As you can see, without the first point on the path being the origin of the arc (or center point), the .fill() method only fills the area between the two end points of the arc itself, drawing a direct line across the curve.

Why Every Developer Should Care About Documentation

Joshua Tzucker — Mon, 01 Mar 2021 13:57:10 GMT

I’ve wanted to write this post for a while now, as with every day that passes, I believe its message more and more: truly, every developer should care about documentation. And I really do mean every developer.

Personally, documentation is something that I care about and is closely related to my passion for the sharing of knowledge, self-education, and the belief that coding should be accessible to all.

However, I realize that not everyone shares these passions, so I’ve set out to compile a list of practical reasons why developers should care about documentation (spoiler: documenting your code is often as much a benefit to yourself as it is to others!).

Clarification
Part A: It’s For Your Own Good
Part B: For the Good of the Team, Users, and Everyone
- Developer Experience and Relations
- As a Form of Backup
Final Thoughts:

Clarification

Before getting started, I should clarify for whom and how this post might apply. Starting with the term documentation:

The term documentation can refer to a lot of different things; README files, Wikis (e.g. Confluence), inline code comments, or even documents that are auto-generated directly from the source code!

Within the scope of this post, I’ll be talking about documentation in general, although the focus will be somewhat on more verbose forms of documentation, such as README files.

And for “whom” do the points in this post apply to? Well, I stand by my article title - yes, every developer should care about docs - but, at the same time, I acknowledge that some specific points (such as using doc writing as motivation) might apply less to those in highly regulated industries or working in a role that is already driven by tight requirements and things like BDD.

Part A: It’s For Your Own Good

Since I often see that it is difficult to get some developers to care about documentation, since they claim they don’t see any personal benefit from spending time on it - I’m going to start by challenging that fallacy. Even if you are a team of one, only working on your own solo project that only you use, there are still very important reasons to care about documentation.

Documenting Code Uncovers Bugs, Vulnerabilities, and Issues

This goes hand-in-hand with my point about “dogfooding” (more on this later), but often a underappreciated way to find bugs, security concerns, and issues that need to be raised is to write documentation, and / or working on keeping it up-to-date.

I’m not going to go deep into discussing things like requirements and specification languages, but they are an obvious tangent to this point, and can have a similar effect.

The reason why I say that writing docs can help a developer uncover bugs and issues is because the act of writing them often challenges the assumptions they might have about how their code works, and forces them to see their system in a new context.

When you are writing and viewing code, unless your code base is extremely tiny (or your monitor is extremely large), you are never seeing the entire thing at one time. Your brain conveniently fills in the gaps - you know that when you call method X, it is going to do thing Y and Z.

The same thing generally goes for thinking about the user experience (or developer experience) - the model of what the experience looks like is too large to “see” at a single moment, so you either consider a tiny part (“What does it look like for a user to request an authentication token from us?”), or for the macro level, you brain goes back to its old tricks of filling in the gaps and abstracting into broader levels.

Reading the above, you might see how this could lead to issues. Assumptions are dangerous in software development, and although tests might catch bugs, or even visual regressions, they don’t generally check experience or usability, they check that something works and meets preset expectations. An application might compile and run without crashing, but fail to be functional in a practical sense.

Writing documentation encourages you to slow down, see your code in a new context, and challenges assumptions you have about how your own codebase works.

Writing While “Dogfooding”

One of my favorite “side-effects” of writing documentation is that it often forces a developer to “dogfood” their own product (or area of the product that they are working on).

For those unfamiliar, the term “Dogfooding” usually refers to the practice of using your own product or service.

There is often a disconnect that happens with software development, where people end up working on features that they never use themselves. In my opinion, this is also a large contributing factor to how we end up with scenarios where a feature is released or updated in a way that makes the user experience worse for many users, and they are left wondering: “Why did no one realize how unusable this is?”

Writing documentation usually forces the person writing it to, at the very least, run through the basic steps of using the thing they are writing about. This, in turn, has a couple huge benefits:

The author might discover new features that can be added in the future. This might come about in several different forms of realizations:
- “Wait, our product doesn’t do ____? We should add that!”
- “Huh, what I’m documenting sounds useful, but I have a feeling it could be better if ___”
- “This can’t be all there is to ____!”
The author might discover shortcomings in existing features
- “Wow, this should not take this many steps to do! Our users put up with this?”
- “This can’t be right…”
- “Wait, how does this work again?”
The author might realize ways in which their mental model of the application has diverged from what actually ends up getting delivered to the end-user
- “I could have sworn it worked this way!”

Improve Productivity

Comprehensive documentation can have a massive impact on developer productivity, when you consider that it can have all the following effects:

Reduce employee onboarding time
- “Getting started” docs, even if only internally used, can greatly cut down on time to get a new dev setup
Reduce redundant questions / issues
- I love people asking questions - it is a necessary part of growth. However, if you have multiple people all asking the same questions over and over again, that is a good sign that the knowledge they are seeking is important enough to belong in shared documentation
Reduces delays / “knowledge latency”
- No one wants a project held up because everyone is waiting on one person with secret knowledge to reply to a long email chain
- Docs don’t take vacations, they aren’t going to burn out from everyone pestering them, etc.

Documentation Driven Development

Starting with Docs is Motivating

This point is especially true if you are a “indie hacker”, solo coder, entrepreneur, etc. - writing documentation can be extremely motivating if you approach it with the right perspective(s).

In explaining this point, I’ll start off with what is probably the wrong approach if your goal is self-motivation: don’t leave writing documentation as the last thing to do, as a final step when all the code has already been written.

Instead, to use documentation as motivation I would suggest writing and updating docs actively as you work, or even writing ahead - describing features that you aspire to one day implement (making sure to either keep these out of public-facing docs, or clearly marking them as “roadmap” items).

This might not work for everyone, but I often find that if I have already written down the basics of a feature that I’m personally excited about, having that documentation in front of me is highly motivating; even if it is just a single bullet point, having it written down helps me visualize the end-goal and focus on my desire to make it become reality.

Starting with Docs Is a Legitimate Pattern

I’m far from the first person to think that starting with the docs is an approach worth exploring. Those who have been in software for some time might already be familiar with a lot of related topics:

The “Waterfall Model” of Software Development

“Behavior-driven development”

Part B: For the Good of the Team, Users, and Everyone

Developer Experience and Relations

Developer Experience / DX

If the product that you are working is something that is developer focused (e.g. an API, SDK, Framework, PaaS, etc.), then you might already be aware of how critical documentation is in eliciting a positive developer experience (DX).

However, something that often gets overlooked is that, in the same way that having a lot of tests is not necessarily the same as having high test coverage, having a lot of docs is not necessarily the same as having documentation that properly covers the developer experience of your product.

To be fair, there is a very fine balance between too much documentation (which can overwhelm the user / dev), and too little or sparse documentation. But at the end of the day, there is a truth which is hard to ignore - you can build a dev product that beats every competitor out of the water with features, stability, and performance - but without docs, without a way for developers to know how to use your product - you can’t expect people to jump on board.

"You're saying this inspirational post-it note isn't enough instruction for potential users?"

Developer Relations (DevRel), Advocacy, Evangelism

The role of a developer advocate, if you want to boil it down into the core purpose, usually comes down to “helping developers (use your product)”. In the context of this definition, I don’t think anyone will argue with me that docs are a big part of helping developers, but sometimes there are more subtle influences that documentation has on developer relations aka DevRel):

Evangelism and driving product growth
- Usually, under the umbrella of DevRel, one of the objectives is spreading word about the product to developers and helping to drive growth. (“Hey, it’s a business after all!”)
- People often focus on blog posts, viral tweets, and landing pages as the content that drives further adoption, while neglecting to talk about docs
- Good docs can help drive growth, especially if attention is paid to proper SEO (Search-Engine Optimization) and organization
  - There are a number of products that I have found, and started using, only because I ended up on their documentation while researching something related.
  - Conversely, there are also some products that I have avoided entirely, or hesitated trying, due to what I perceived as sub-par documentation
- Some products are renowned for the quality of their docs, and that itself can become a driving force behind “word-of-mouth” growth
For open source: driving contributions and community interest
- If your product is open-source, and you are looking to get the community more involved and grow the number of contributors, documentation is great way to do so
- Documentation is often the entry-point for a developer, before pretty much anything else, so it becomes an easy place for developers to contribute back to the project by addressing shortcomings in the documentation as they on-board themselves
  - Example: Developer opens a Pull-Request (PR) to add a section to the “Getting Started” section, with added instructions for Windows (current docs only have instructions for Mac or *Nix)

📄 Further reading:

“The Developer Relations Explainer”

“What is Developer Advocacy”

Empathy

The best documentation is inclusive and meant for as many people as possible. Writing documentation with this outcome as the goal is a great way to challenge yourself to approach your codebase with greater empathy for other developers.

As a Form of Backup

Just like code and data, knowledge needs protecting and backing up too! A bunch of code without the information on how to run or update it does not hold much value in practice.

If you mention the words “backup”, “recovery”, or “redundancy” in a room full of software industry professionals, you are likely to get swarmed by multiple people who are all too enthusiastic to explain how their company’s systems are built to handle any sort of catastrophe or incident.

"Is this the line to talk that person's ear off about backup systems?"

You’ll hear things like how they use multiple redundant backups, perhaps even replicating onto physical long-term storage, like tapes. They might backup every version-controlled repository daily, refusing to rely entirely on a third-party host like Github or Perforce, in case of a failure. They might boast about how fast they can rollback the data itself - reverting a DROP TABLE or rm -rf type incident with lightning speed.

You might also get an ear-full about how easy deployment and scaling are for them. Perhaps they have reached less than a minute of downtime per year (aka Six Nines, 99.9999% uptime).

However… try asking this same group of individuals how knowledge is backed up within their organization. Often, the answer is either “it isn’t”, or the efforts to organize, safeguard, and streamline documentation is miniscule in comparison to the systems it belongs to.

Why Does Knowledge Need Backing Up?

One way to think about this, which is unfortunately morbid, is “what would happen if ____ employee suddenly passed away”.

Or, if it makes you feel better, they were suddenly “made redundant” after they spilled coffee all over the server racks.

In many companies, information tends to get siloed - within a team, or even limited to a single person. You might ask someone “How do I change X in system Y” and get a response like “Oh! Morgan, the architect of system Y is the only one that can answer that.”. Or maybe you’ve been a participant on a message chain that goes something like: “I’m not sure about that… I’ll have to CC ___ in on this…”, until it turns out you need Morgan again.

Hmm... I think you *actually* want to talk to Morgan... I think they went that way...

Notice how these examples are also instances in which developer productivity is reduced, because we are now waiting on knowledge to be available (something I touched on earlier)?

Going back to our morbid question: what if we suddenly lose Morgan? Did they ever document how system Y worked, in a way that others can understand? If not, we might still survive - but it might take quadruple the resources - in time and money - to decipher their code and make sense of everything. Even if we can hire someone of exactly the same skill level, they aren’t going to be able to hit the ground running without having that system properly documented and the knowledge accessible.

🚍 The software industry actually has a term for the idea of how many people it would take suddenly dying (e.g. getting hit by a bus) before it brings down a project - the “Bus Factor”

Also, nothing against anyone named Morgan!

Final Thoughts:

Refuting Common Arguments

There seems to be a certain subset of developers that, the second someone brings up the importance of documentation, will (loudly) interject with objections and various claims of why they don’t need to write docs.

This is already a very long post, so I don’t want to spend too much time digging into these (maybe in a future post?). But I’ll very quickly give a response in list format

“My code is self-documenting!” (or self-explanatory, or something like that)
- Nice try, but no. Even if you are using a strongly-typed language, your code can answer the question of how, but not why (that is what comments are great for). Well-name variables and readable code do great things, but they need to be part of a larger picture.
- ALSO: readability !== maintainability
“It’s just me! I know my own code!”
- If I we had time travel, there are a non-trivial number of developers that would all have bruises from when their future self went back in time to punch their past self for saying this. Yes, you know your own code now, but will you really feel the exact same way in a year? 5 years? 10?
“Too many comments in code are a form of code-smell”
- People love to throw this argument around. They will use examples where a code comment will basically be identical to the code - but this is a contrived example, and not how comments should be written in the first place
- Again, comments should be mostly about the why, not the how. Docs such as READMEs can be all of the above (+ what, when, etc.)
- See responses in this discussion
“We are already using auto-generated docs from [methods, JSDoc, etc.]”
- Auto-generated docs serve a purpose, and are good for things like describing API endpoints, but they often don’t add that much value on-top of the code they are generated from.
“Docs get easily outdated. Code does not.”
- This is actually a reasonable point. See below

Preventing Out-of-Date Docs

The one argument against spending much time on documentation that I feel actually has some merit is the tendency for docs to get out of date (or out-of-sync) with the code they belong to. Critics will claim that bad docs are worse than no docs at all, which I don’t agree 100% with, but as someone who has been burned by bad documentation, can attest to the frustration that is felt when you waste time and effort because no one bothered to update materials.

This is a complex problem, worthy of its own in-depth discussion, so I’ll just provide what I find as the best counter-measure; the closer your docs live to the source-code the more likely they are to stay up to date.

This is where systems like Confluence, Notion, or shared Google Docs can potentially cause issues. In my honest opinion, those should be used for knowledge base (e.g. wiki) stuff, and documentation is better suited to things like Markdown files that live near the actual code (aka colocation).

Another important aspect is how easy is it, from a procedural standpoint, for new people to edit the docs? Can any new employee or contributor submit an update? Or does it require some lengthy approval and merge process, that is likely to discourage frequent edits?

I’m Not Perfect

Finally, I’m not asking that you follow every tip I’ve laid out here, or buy into the points I’m making without any objections. I’m not perfect, and some of the realizations I’m sharing here I have only because of painful learning experiences.

I’m only asking for a little more mindfulness around documentation, for everyone’s benefit.

Strongly Typed Web and Service Workers with TypeScript

Joshua Tzucker — Tue, 09 Feb 2021 06:10:48 GMT

I’m writing this post because I had trouble finding a straightforward answer to this question: what are the current ways to use TypeScript with Service Worker files? And, to take it a step further, how to use those types in JavaScript, with JSDoc comments?

The examples in this post are specifically for a Service Worker script, but in general the solutions and issues in this post apply to web workers in general as well.

The Basics of Where Things are At

First, before getting into the solutions, I want to bring up a few things that I found in my research that are important to note. Keep in mind that these things might change, so this is kind of a “here things are now” section:

You no longer need external type definitions for web workers / service workers
- A lot of older posts and StackOverflow answers will suggest things like copying and pasting code out of a Gist, using @types (DefinitelyTyped), or some other external type defs. These used to be necessary, but Web Worker types are now shipped by default with TypeScript, in the lib file lib.webworker.d.ts. How to use this file will be discussed in the solutions below.
  - For relevant issues / PRs on how these made their way in:
    - TS DOM-lib-generator #439
    - TS #11781
There are some big open issues with Web Worker types that are blocking easier solutions for strong-typing service worker files.
- These will be discussed again, throughout this post, but the major ones are:
  - #14877: The definition for self is not resolved correctly
  - #20595: Conflicts with DOM vs WebWorker
    - Related - #11093: Intersection of libs
    - If this issue is resolved, depending on how it is fixed, the solution to using TS with web workers might become greatly simplified

Solution A: Triple Slash Directives

Since a Service Worker often lives in a single file, some users prefer to pull in its lib types via a triple slash directives instead of modifying the tsconfig.

/// <reference no-default-lib="true"/>
/// <reference lib="ES2015" />
/// <reference lib="webworker" />

// Default type of `self` is `WorkerGlobalScope & typeof globalThis`
// https://github.com/microsoft/TypeScript/issues/14877
declare var self: ServiceWorkerGlobalScope;

// Example:
self.addEventListener('install', () => {
	//
});

// We need an export to force this file to act like a module, so TS will let us re-type `self`
export default null;

lib=ES2015 should be changed to whatever you are using, but it should be an es5+ choice

🚨 Uh-oh! Triple-slash directives don't play nice with other files that use the regular DOM!

If you have noticed that using triple-slash directives to bring in Web Worker types in your service worker file also messed up your regular TS files, you are not imagining things. For a pure TS solution, skip to solution B, which explores multiple scoped TSConfigs.

But, for JS users, we can bypass this tricky issue; read on below!

Triple-Slash Directives with JavaScript and JSDoc

Here is the same approach as above, but carried out with JavaScript, using JSDoc for types:

// @ts-check
/// <reference no-default-lib="false"/>
/// <reference lib="ES2015" />
/// <reference lib="webworker" />

// Using IIFE to provide closure to redefine `self`
(() => {
	// This is a little messy, but necessary to force type assertion
	// Same issue as in TS -> https://github.com/microsoft/TypeScript/issues/14877
	// prettier-ignore
	const self = /** @type {ServiceWorkerGlobalScope} */ (/** @type {unknown} */ (globalThis.self));
	self.addEventListener('install', (evt) => {
		// ...
	});
})();

💡 Ah-ha! In implementing this solution I have found a trick around the issue of triple-slash directives polluting other files.

The trick is to exclude the JS file from your tsconfig and then add back checking by using //@ts-check at the top. This prevents the triple slash directives in the file from “bleeding” into the other files that are not excluded by the tsconfig. This will prevent things like window becoming undefined or all instances of globalThis.self getting changed to the Service Worker Type!

lib=ES2015 should be changed to whatever you are using, but it should be an es5+ choice

Solution B: TSConfig Libs

Triple slash directives are not always the best approach, especially since they can “bleed” into other files anyways when you are using them with libs (they essentially act like global lib options - see docs). So, there is not much benefit over using the --lib flag or lib option in tsconfig.

Relevant issue: #20595

In fact, the best option might be to have a different tsconfig for your service worker, versus for all other files. For example:

Base tsconfig.json:

{
	"compilerOptions": {
		"target": "ES6"
	},
	"exclude": ["service-worker.js"]
}

Service-Worker tsconfig.service-worker.json:

{
	"extends": "./tsconfig",
	"compilerOptions": {
		"lib": ["WebWorker", "ES2015"]
	},
	"files": ["service-worker.js"]
}

If you are looking for a more detailed breakdown of this approach, check out this StackOverflow answer.

Nested Service Worker Folder

The above option works if you are only using TypeScript via CLI, and not for Intellisense, as VSCode does not resolve more than one tsconfig per folder. If you wanted to use the above solution with VSCode and get full in-IDE type-checking, you would need to move service-worker.js to a sub-directory, with its tsconfig file located alongside it.

However, it should be noted that this causes its own issue, and in fact, a big one. Service workers by default are granted a scope based on file location. So a worker located in the root directory:

.
├── index.html
├── tsconfig.json
└── service-worker.{ts|js}

Can access any request. However, if we move to a subfolder:

.
├── index.html
├── tsconfig.json
└── sw/
	├── service-worker.{ts|js}
	└── tsconfig.json

… now our service worker has a default scope of /sw, and declaring a higher scope actually requires configuring our server to send a unique HTTP header with any request for our worker script!

The easiest fix is probably to use outDir or outFile to make sure that the transpiled SW script ends up in the hosted project root, regardless of the source code directory structure.

Auto Compiled Svelte Tutorial Docs - Single Page Version

Joshua Tzucker — Thu, 04 Feb 2021 03:36:07 GMT

NodeJS-powered automation that extracts all the tutorial docs out of the Svelte repo and combines them into a single formatted HTML document.

Light Screen Display - Virtual Fill Light

Joshua Tzucker — Tue, 02 Feb 2021 16:34:38 GMT

This was a fun project that I built the first version of in a few hours, on a whim. I have multiple extra monitors pointing down at my face that are empty real-estate while I participate in video calls, and I had been noticing that the light they cast actually has a subtle effect on the image captured by my webcam.

I was curious how much a difference it would make if I filled up my extra monitors with solid colors, so I built this app as a quick way to test out different fill styles and colors.

Spoiler: I made this for fun, and the end result was that, although it makes a small subtle difference, using OBS filters or buying a dedicated external light will make 100x the difference that this does. But it is still a fun experiment!

Where to Get It

🔗 App: displaylight.netlify.app

💾 Source code: github.com/joshuatz/light-screen

Demo

Sorry, your browser doesn't support embedded videos.

Features

Multiple modes
- Ring Light
  - LEDs
  - Diffused LEDs
  - Solid Ring
- Solid Full-Screen Fill
Camera preview
Auto-hiding settings panel (can be turned off)
Keeps screen on while in use
Custom color picker, or use presets
Works offline, and as a PWA

What I Used

JavaScript (Vanilla)
HTML, CSS
Service Workers
TypeScript (with JSDoc)
HTML Canvas

Google Sheets - How To Split a String Into Characters

Joshua Tzucker — Sun, 31 Jan 2021 13:53:46 GMT

I’m writing up this post because this is now the second time I have run into this issue, and the solution I finally ended up with is not as simple as one might guess.

For splitting text into columns in Google Sheets, or even with a known delimiter, the approach is not that difficult - “How-To Geek” has a good step-by-step guide for those situations.

What this post is about is splitting text in Google Sheets by character, with an unknown input length and text content. For example:

Input	Expected Output
`\| hello \|`	`\| h \| e \| l \| l \| o \|`

This is harder than it might seem for several reasons:

The SPLIT() function in Google Sheets does not allow splitting by an empty delimiter (e.g. "")
- This might be surprising to you if you are a JavaScript developer, since const charArr = 'hello'.split(''); is not only permissable in JavaScript, but it does exactly what we want
  - In fact, one solution to our issue could be to simply create a custom function with Google Apps Script, called JS_SPLIT(), and allow it to be called from Sheets
Inserting a delimiter into the input string, between each input character, to workaround the above limitation works, but has its own set of nuances

Solution

First, I’ll provide the solution, and then break down how it works, and why it might seem more complicated than necessary.

Formula:

=SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))

Easier to read:

=SPLIT(
	REGEXREPLACE(
		REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127))
	,"'","''")
,CHAR(127))

Breaking this down section by section, from inside to out:

REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127))
- What: This replaces every character in the input string (contained in cell A2) with itself, plus a special delimiter character
- You’ll notice that I use CHAR(127) as my inserted delimiter, whereas many answers on StackOverflow use a plain text character, like ~. There are two reasons why I use CHAR(127) instead of ~.
  - CHAR(127) is the Delete Character, which is an ASCII control character (invisible in this case) that is highly unlikely to ever show up in arbitrary input text. The Tilde character (~), on the other hand, is much more likely to show up in text input, which would cause issues with a formula that uses it as a delimiter
  - The delete character is one of the very few control characters that Google Sheets does not strip out of input / output
- Another really important part of this step that a lot of StackOverflow answers miss is the (?s) part. This is a RegEx flag (in the RE2 flavor) that allows . to match any character as well as new line (\n). If we were to leave off this flag, then our formula would fail to work properly with input cells that contained multiple lines of input
REGEXREPLACE(..., "'", "''")
- Here I am escaping any single quotes in our input string, by replacing them with '' (two single quotes).
- This is important because if we don’t do this, then any single quotes in the input string will turn into empty cells when we split by the delimiter in the next step. ' is a special leading escape character in Sheets.
=SPLIT(..., CHAR(127))
- Finally, I am splitting the string with delimiters inserted, by the delimiter itself (CHAR(127)). The result will be an array, which gets expanded out, starting in the cell you place this formula in

Emoji Issue

Despite this formula working with line breaks, symbols, special characters, and even quotes, there is one instance I have found where it might still produce unexpected results: emoji input. I’m reluctant to call this a problem or “bad” behavior, because you could also view it as the formula working correctly.

Here is the issue: certain “emoji” are displayed as one character, but actually comprised of multiple emoji joined together, by a “zero-width joiner” (ZWJ), into a ZWJ Sequence. These tend to be emoji that have a common base (such as a face), but are a variation on that base (maybe a different skin color, hair color, etc.), and especially for gendered emoji combinations. A list of some of these can be found on emojipedia.

Without getting too much into the details, here is an example and how it plays out with my formula.

This emoji - 👩‍🦰 - is Woman: Red Hair, and as long as you have an up-to-date operating system, you should see it as a single character on your screen. However, look at how the Google Sheets formula splits up that character when it appears in our input string:

What has happened is that the formula has treated 👩‍🦰 as three characters, not one, which is technically correct since it is a ZWJ Sequence comprised of 👩 (Woman, U+1F469) + ZWJ (U+200D) + 🦰 (Red Hair, U+1F9B0).

The 👋 (waving hand emoji) is split as a single character, since it is not a ZWJ sequence, but rather a regular single glyph / emoji.

Note: This same behavior can be seen in JavaScript - try copying and pasting this into your console for a fun experiment: console.log([...'👩‍🦰'])

ZWJ Sequences also completely break formula that rely on using length tricks, such as this alternative solution for splitting by character.

Brain Breaks Browser - Online Web App for Fun Activities

Joshua Tzucker — Wed, 20 Jan 2021 00:08:44 GMT

Brain Breaks Browser is a web app that lets you create, store, track, and organize “brain break” activities. These are fun tasks and exercises that you can use to split up work time into more manageable chunks, and exercise your brain while enjoying a break.

Where to Get It

🔗 brainbreaksbrowser.netlify.app

Demo

Sorry, your browser doesn't support embedded videos.

Features

Create multiple boards, share activities between boards
Create custom activities with an easy-to-use form
Use activity types for extra automatic features
- Videos: Task automatically marked as complete and popup closed when video reaches the end of its duration
- Website / URL: Embed an entire webpage, securely, into your activity
- Timed: Multiple activity types, even the completely customized one, supports an integrated timer component
Track activity completion, reset progress, or disable repeat limits entirely
Randomize activities, or even mask activity icons

Why?

These types of activities used to be common to do in person, and were often the type of thing to be written on flashcards or popsicle sticks and picked out of a cup. With the combination of COVID, advancement of technology, and renewed focus on remote learning and professional work, these offline systems simply are not feasible for many users.

I thought it would be a fun project to create a simple web portal to organize these activities, and provide extra features like built-in timers and progress tracking, so that not only will users not miss their stacks of activity flashcards, but they might even prefer the new system.

Although the focus is on remote learning, there are many use-cases:

Classrooms, tutors, remote education settings
Business Meetings / Standups
Web conferences
Yourself! Control your break time with preset activities and auto-closing tasks

What I Used

TypeScript
Preact
Preact / React Hooks
React Player
Chakra UI
Dexie.js (IndexedDB wrapper)
Javascript, HTML, CSS
And more!

I also published a writeup around a specific issue I ran into with this app - “How to Deploy Preact from a Subdirectory”.

How to Embed, Inline, and Reference SVG Files in Svelte

Joshua Tzucker — Tue, 19 Jan 2021 14:40:39 GMT

I recently needed to work with SVGs in Svelte, and had trouble finding comprehensive guides on solutions, especially those that allow for the SVG to be completely inlined, so that you can use things like fill: currentColor, or CSS rules. This post tries to cover some of the most useful solutions I found during my research, complete with step-by-step instructions and code snippets.

Placing in Public and Avoiding Bundling
Using a Bundler Plugin
Directly Inlining
Saving as Svelte Files
Side-note: Dealing with TypeScript
Svelte-Inline-SVG and other AJAX Based Solutions
Other Solutions

Placing in Public and Avoiding Bundling

If you don’t care about inlining the SVG, and are OK with an external embed, the easiest way to get an SVG into a Svelte component is to simply bypass the bundler and use the /public directory.

For example, you could place your file at: /public/assets/dog.svg, and then your Svelte file could be as simple as:

<img src="/assets/dog.svg" />

In general, there are some important drawbacks to this approach however:

Paths are not checked by the compiler / bundler, or even the IDE
- This makes typos a little too easy, or situations where a file gets renamed and no one catches the change
Assets are not optimized
- Any file you stick in /public is going to end up in the build output. By contrast, if you use the bundler (see other approaches I outline), assets that are not used will not get bundled into the build output
You cannot inline SVGs this way
- This method allows you to embed SVGs via <img> tags, but it does not allow you to inline the SVG contents directly into the DOM
- This prevents a lot of direct manipulation of the SVG from working, which might be a deal-breaker if you need to animate parts of it, change nested colors, or pass through CSS variables

Using a Bundler Plugin

As is common with many frameworks, one way to solve this problem is to have the bundler take care of reading in the image files, transforming them, and letting them be imported into components. Since the Svelte starter template ships with Rollup by default, that is the bundler I’ll be discussing.

To solve this with Rollup, there are three plugins I’ll suggest as solutions.

rollup-plugin-svg or rollup-plugin-inline-svg
- These are both similar, and both are good if you need to inline a lot of SVGs
@rollup/plugin-image
- This supports more image formats, but is less friendly for inlining

I’m not going to cover it in-depth in this post, but for webpack, check out their docs on asset modules.

rollup-plugin-svg [or] rollup-plugin-inline-svg

These two plugins are similar in how they work, although rollup-plugin-inline-svg has more configuration options.

Once you have installed the plugin you have chosen and updated your rollup config, for actually importing and using the SVG, both require similar code.

Inline SVG embedding (this code works for either plugin):

<script lang="ts">
	import dogSvgDecoded from '../assets/dog.svg';
</script>

<main>
	{@html dogSvgDecoded}
</main>

Embedding as external embed, via image tag + data URI:

For embedding via data URI:
- rollup-plugin-svg
  - Change base64 config option to be true
  - In component, import and then bind as <img src={importedSvgStr}>
- rollup-plugin-inline-svg
  - Since the plugin is focused on inlining, there is no config option for escaping / base64 encoding.
  - The workaround is to decode the string inside the component - this is essentially the reverse of the @rollup/plugin-image workaround:
```
<script lang="ts">
	import dogSvgDecoded from '../assets/dog.svg';
	const dogSvgEncoded = `data:image/svg+xml,${encodeURIComponent(dogSvgDecoded)}`;
</script>

<main>
	<img src={dogSvgEncoded} />
</main>
```

@rollup/plugin-image

The @rollup/plugin-image is the most popular of the plugins, and can support more formats than just SVGs (also supports JPG, PNG, etc.), but also comes with an important limitation.

The limitation is that it converts the contents to a base64 string (or HTML encoded entities string, in the case of SVGs) and then expects you to use it as a Data URI - for most image formats this is no problem (and actually preferred), but for SVGs, this also prevents them from being directly inlined, which subsequently prevents CSS rules from working. If you are fine with that limitation, or want to use the workaround (more on this in a bit), it is fairly simple to set up.

Modify your rollup config to pull in the the new plugin
Activate the plugin within the config (plugins: [image()])
Now, in any Svelte component where you want to use an SVG:
1. Import the file, while assigning it to a variable
2. Bind the variable you assign it to, as src={myVar} on an image element

Example component:

<script lang="ts">
	import DogSvgEncoded from '../assets/dog.svg';
</script>

<main>
	<img src={DogSvgEncoded} alt="Cute dog graphic" />
</main>

@rollup/plugin-image: Inline SVG Workaround

As noted above, a limitation with @rollup/plugin-image is that it is designed to be used with the src attribute on image tags, not for embedding the raw text of a SVG into the DOM. This is why it Base64 encodes the value of binary image files, and escapes / encodes HTML entities in the value of SVG files.

However, with some extra work we can turn the encoded value it gives us back into the original raw SVG string:

<script lang="ts">
	import DogSvgEncoded from '../assets/dog.svg';

	const dogSvgDecoded = decodeURIComponent(DogSvgEncoded).replace('data:image/svg+xml,', '');
</script>

<main>
	{@html dogSvgDecoded}
</main>

This works, but should be avoided (in favor of any of the alternatives outlined in this post), for three reasons:

You are shifting computational work from a build-server onto the users’ browser, which is… not great.
This is adding extra steps - why escape characters only to decode immediately?
It actually increases bandwidth used; it takes extra characters (longer string, more bytes) to escape HTML entities
- For a tiny test SVG, the original version used 3,988 characters, whereas the encoded one grew to 4,285.

Svite / Vite

Currently, using SVG bundling plugins with Vite or Svite is a little tricky.

If you are using Svite, I would recommend just using svelte-inline-svg components, or directly inlining SVGs into Svelte SFC. If Svite is updated to internally use Vite 2.0+, instead of v1, then the below will apply.
If you are using Vite (2.0+), SVG inlining should eventually work out of the box; it currently has an open issue against it (#1204), but there is a PR in progress that should address it.

Directly Inlining

If the SVG file is small enough and it makes sense for your needs, you can actually just copy and paste the raw SVG contents into a Svelte SFC. This is similar to the another solution in this post, renaming entire SVG files to .svelte files.

Make sure you remove any declarations, like <?xml version="1.0" encoding="UTF-8"?>, before inlining

Saving as Svelte Files

As suggested in the comments on this S/O question, one option for importing SVGs and rendering them inline is by renaming the .svg file to .svelte and then importing and using as a normal Svelte component. This works because of how flexible Svelte is with its templating / SFC system.

However, there are some catches. The main one is that if you try this with some SVG files, you might see this error:

ParseError: Expected valid tag name

This might happen because your .svg file contains (at the top) something like:

<?xml version="1.0" encoding="UTF-8"?>

This is an XML declaration (a subset of an XML Prolog). The short summary of what is going on here is that:

Svelte doesn’t like the prolog
It shouldn’t be allowed anyways if the SVG is going to be inlined into the DOM - HTML spec allows / recommends only one declaration per doc, at the top of the page
It is not even necessary to include in the SVG, even if the file is going to be used as a normal SVG external embed

As the declaration is not even necessary for standalone SVG files, the easy fix is to simply remove the declaration line from your file before or after changing the extension to .svelte.

The other solutions outlined in this post deal with omitting the declaration automatically, which is another reason to prefer them over converting your SVGs to Svelte files

Side-note: Dealing with TypeScript

If you are using TypeScript with Svelte, you might see the following error when trying to import an SVG, even if you have properly configured your bundler:

Cannot find module ’../dog.svg’ or its corresponding type declarations.ts(2307)

If our bundler is going to give us a string when we import the SVG path, we can tell TS that this is the case by creating an ambient declaration file and typing the file pattern.

ambients.d.ts:

declare module '*.svg' {
	const content: string;
	export default content;
}

Svelte-Inline-SVG and other AJAX Based Solutions

The plugin svelte-inline-svg plugin, and some similar plugins, solve the inlining issue while avoiding bundlers, by taking a somewhat unusual approach. The package is essentially a Svelte component that requests your SVG file as a network request, then parses the response and inlines the nodes.

This can work for a lot of situations, but also comes with an important caveat; the SVG files must be able to be fetched by the component at run-time, or in the case of SSR, at build-time. In practicality, this means that SVG assets would need to be placed in /public, and not /src.

Guide: How To Deploy Preact from a Subdirectory

Joshua Tzucker — Sat, 02 Jan 2021 09:54:45 GMT

Intro
How to Deploy Preact from a Subdirectory
Summarizing: A Checklist
Alternate Approaches

Intro

I’ve been working on a Preact project recently, which is a single-page-application, and I decided that I wanted to host and deploy the landing page from the same project repository, on the same domain. This means that instead of serving my generated Preact files from server root (by deploying /build), I want to serve them from a subdirectory:

___	Old	New
Build output:	`/build`	`/static-root/app`
Homepage (example.com) gives you:	The app	Custom static LP (`/static-root/index.html`)
App URL Path:	`/` (homepage)	`/app`

My first thought was simply to change the destination of the build output to the nested folder and change my deploy settings to host the parent folder, but, without changing anything else, this basically breaks the entire app:

Breaks assets, script loading, etc.
Breaks Preact router
Breaks webpack bundle output, ESM chunk loading, etc.

When I searched for the right way on how to deploy a Preact app from a subdirectory, a lot of the results I got were not very straightforward, and not necessarily even the right things to do.

I’m making this guide as a reference point for myself, and anyone else that wants to see the basic steps that are necessary to get this working.

It can be done! 😄

How to Deploy Preact from a Subdirectory

Most of these steps are not in any particular order, and they are somewhat specific to my setup; if you are using a different bundler, router, etc. - you will need to slightly tweak things. But, still a good starting point for the right things to adjust.

However, this first step should probably be done first, since it will help later:

Prepare for Changes: Setup Shared Base Path

My first step, which I’m glad I had the foresight to think through, was setting up a shared base URL / path value, which can intelligently change based on the environment the app is launched in. For example, if I launch my app in dev mode, with hot-reloading, I don’t want the app to try and use /app/* URLs, because the dev server launches the Preact app directly - not in a subdirectory.

You can set this up however, you please, but the way I did it was first adding a value to my package.json for easy configuration changes:

{
	"publicPath": "/app/"
}

This is important, because the package.json is an easy root-level spot to share values, and publicPath matches the name used by webpack (more on this later).

Next, in a common export point (I used src/constants.ts), I export a wrapper around this value:

import pkgInfo from '../package.json';
/**
 * The base URL where this app lives. Used for routes, assets, etc.
 *  - Needed for dealing with serving out of subdirectory
 */
export const APP_BASE_URL_NO_SLASH_END: string =
	process.env.NODE_ENV !== 'production'
		? ''
		: pkgInfo.publicPath.endsWith('/')
		? pkgInfo.publicPath.substring(0, pkgInfo.publicPath.length - 1)
		: pkgInfo.publicPath;

export const APP_BASE = APP_BASE_URL_NO_SLASH_END;

The above code uses the package.json value, which means that right now it either returns '' (empty string) if running in dev mode, or /app if actually running in production, with the nested subdirectory.

Now, I can import APP_BASE in various parts of my app, and automatically prefix links with the correct base, by doing something like ${APP_BASE}/user/edit/.

Shared Base Path: Setting Environment Variable

One issue that I quickly ran into with the above code is that, unlike most other frameworks I’ve used, Preact CLI does not actually change the value of process.env.NODE_ENV based on the command that is run. In order to have process.env.NODE_ENV = 'production' when preact build is ran, and = 'development when dev is ran, I had to install cross-env as a dependency, and then manually add those values to my package.json scripts.

For example:

{
	"scripts": {
		"dev": "cross-env NODE_ENV=development preact watch",
		"build": "cross-env NODE_ENV=production preact build"
	}
}

Configure Bundler

This is one of the most important steps; you need to update the configuration of the JavaScript bundler (e.g. Webpack) so that all the distribution files end up in the right places, with the right files pulling them in.

I won’t provide instructions for every bundler out there, but I will show how to do it for Webpack.

If you are using standard Preact template, there is a good chance you are using Webpack, with configuration controlled via the preact.config.js file. To update this config to handle a subdirectory deploy, you want to mutate the config to add a publicPath option - this tells webpack where to place generated files. Here is how I modified my preact.config.js:

// We need the shared path value!
import pkgInfo from './package.json';

export default {
	webpack(config, env, helpers, options) {
		// Rest of my config omitted
		// ...
		config.output = config.output || {};
		if (process.env.NODE_ENV === 'production') {
			config.output.publicPath = pkgInfo.publicPath;
		}
	}
}

Fixing Router

This step can take the most work, especially if you are using preact-router and have lots of routes and links.

I highly suggest using an IDE that supports advanced Find-and-Replace searching (such as VSCode), so you can bulk update links and paths

Patching Preact-Router

Preact-Router does not currently support a base path option, although there is an issue and open PR to add it. In the meantime, it is feasible to patch the functionality in-place.

First, I created a wrapper method around the normal route() function. This is in TypeScript, which makes it little more verbose in dealing with function overloading signatures (route() is overloaded in its exported definition):

// Notice: I need the automatic sub-dir path from where it is exported
import { APP_BASE_URL_NO_SLASH_END } from '../constants';
import { route } from 'preact-router';

/**
 * Wrapper around preact-router `route()`, since they have not implemented a
 * base URL option, needed for subdirectories, or custom base paths
 */
export const appRoute: RouteFuncSignatures = (
	input: string | { url: string; replace?: boolean },
	replace?: boolean
) => {
	const inputUrl = typeof input === 'string' ? input : input.url;
	let patchedUrl = inputUrl;
	if (!patchedUrl.startsWith(APP_BASE_URL_NO_SLASH_END)) {
		const sep = !patchedUrl.startsWith('/') ? '/' : '';
		patchedUrl = `${APP_BASE_URL_NO_SLASH_END}${sep}${patchedUrl}`;
	}

	if (typeof input === 'string') {
		return route(patchedUrl, replace);
	}

	return route({
		url: patchedUrl,
		replace: input.replace,
	});
};

interface RouteFuncSignatures {
	(url: string, replace?: boolean): boolean;
	(options: { url: string; replace?: boolean }): boolean;
}

Now, I can simply find all uses of route() and replace them with my appRoute() wrapper function (which takes identical arguments, so I can literally find and replace all instances).

I also need to fix how I have defined my route matches. Luckily, this is a lot easier; I just need to prefix all the paths with my shared prefix:

Diff:

<Router onChange={handleRoute}>
	<Route
-		path="/user/create"
+		path={`${APP_BASE}/user/create`}
		component={User}
	/>
</Router>

There are other ways you could patch this - e.g., you could create a wrapper around <Router></Router> instead.

Fix Hard-Coded Links

Anywhere that you have hard-coded links, that are not relative, you need to update them to either be relative (which should work with the subdirectory move), or prefix with the shared subdirectory base.

Examples:

- <a href="/dashboard"> Dashboard </a>
+ <a href={`${APP_BASE}/dashboard`}> Dashboard </a>

- <Link href="/dashboard"> Dashboard </Link>
+ <Link href={`${APP_BASE}/dashboard`}> Dashboard </Link>

PWA Support: Double-Check manifest.json

If you are deploying your app as a PWA, you will probably want to check your manifest.json file, and make sure that the URLs within it are relative and not absolute, as well as update the start_url value to point to the subdirectory and not the root path (/).

Update Deploy Config

This step is unrelated to Preact, and would be required for any project where you have changed what folders should become the root of the host / deployment server.

This step is also going to be unique to your deployment situation. For example, if you using an automated system like Netlify, it should just require changing some values through an Admin UI (Example: Netlify - change the publish directory option).

Summarizing: A Checklist

Here are all the steps I’ve listed, as a checklist that you can scan through if you are still having issues getting things working.

Setting up a shared base path variable
Configure your Bundler
Fixing Router
Fixing Links
PWA Support: Checking manifest.json

Alternate Approaches

Through all the steps I listed, my approach was to store the subdirectory path string in my package.json, but that is not the only option. If we wanted an even more flexible system, where the same repo could be deployed to multiple different subdirectories, we could pull the subdirectory path from an environmental variable instead of hard-coding it into the package file. Then, in each deployment, we would just need to make sure the build server had that environmental variable populated with the correct value for its target.

I’m sure there are many more approaches I haven’t discussed; as always, use what works best for you and your requirements. And happy coding! 👩‍💻

Google Sheets - Faster Data Exports Options, Including CSVs

Joshua Tzucker — Wed, 30 Dec 2020 06:04:29 GMT

Intro

I recently came across this thread on Twitter, which captured my interest:

If you have Twitter embeds blocked, the Tweet thread is a discussion of an issue with Google Sheets, where the public “published” CSV feed URL does not return up-to-date data when fetched via curl, or at least not consistently and quickly.

I fired up a demo sheet, and lo-and-behold, I could easily reproduce the same issue they were seeing; Google Sheets was:

Taking a decent amount of time to update the published CSV
The published data would sometimes fluctuate, and rows would randomly disappear from the CSV feed
Both of the above issues were occurring randomly; I could not determine a consistent pattern

Now, I used to use the published CSV/TSV feature all the time and rarely ever saw this, but was also not relying on real-time data feeds, so my guess is that it is a replication issue that goes away with time; James is looking for data ASAP, and the Google servers are likely still passing around the data as it is added (could be multiple layers, like CDN + replication + sharding).

But, if you are in James position, and you are looking for the ability to have a CSV feed that always returns an up-to-date fresh export from your Google Sheet, is there an option for you? Yes! Several!

Option A - Use the Official Sheet APIs (v4)

This (using the Official v4 Sheets API) was suggested pretty quickly in the Twitter thread:

But, if you follow the linked StackOverflow answer, you’ll see that there are some limitations with the new v4 API:

You need an API key (which requires using Google Cloud Platform)
Data is always returned as JSON

However, this is your best bet for a more robust, best-practices, “future-proof” choice, as v3 of the API is being deprecated very soon.

Option B - Google Sheets Public Feed URLs (v3)

To start with, if you have a Google Sheet and want a public CSV feed, you can use the Publish to the Web option, and get a feed URL that looks like this:

https://docs.google.com/spreadsheets/d/e/{UNIQUE_ID}/pub?gid={SHEET_ID}&single=true&output=csv

… where {UNIQUE_ID} is not the normal document ID, but instead a unique random publication ID. {SHEET_ID} does however, correspond to normal sheet (pages of the spreadsheet) IDs.

As already noted, an issue with the above feed can be a delay in accurate data being reflected after the source is updated. However, that is not the only feed URL that is available to us without an API key. Until Google deprecates v3 of the Sheets API, there are still v3 API endpoints that you can use to get fresh data, without needing an API key:

🚨 WARNING 🚨: All these URLs will likely stop working on June 8, 2021, which is when Google is planning on shutting down the v3 version of the API. They are not likely to extend the deadline, as they already did so once already.

All values on a sheet
- XML: https://spreadsheets.google.com/feeds/cells/{DOC_ID}/{PAGE_NUM}/public/full
- JSON: https://spreadsheets.google.com/feeds/cells/{DOC_ID}/{PAGE_NUM}/public/full?alt=json
Specific range of values
- Use the same endpoints as above, but control range with:
  - min-row=
  - max-row=
  - min-col=
  - max-col=

There are a lot of other advanced feeds you can get with the v3 API - see Docs: “Manage List-based and Cell-based Feeds”.

📄 You can find a more “step-by-step” guide here.

Option C - Google Apps Scripting

All this research into Google Sheets got me thinking about Google Apps Scripts (aka GAS), which I have used many times before, and wondering if it could be fitted to this task. GAS can live with a spreadsheet document, access cell values, and can even be published as a web app and respond to web requests!

I spent a little time and threw together a fun mini-API that can export just the last X number of rows of data, or the entire document, as CSV, TSV, or JSON! If you don’t want to require a Google login, but also don’t want it 100% public, you can even lock it down with an authentication key, entered into a named range of your document.

Of course, after finishing my script, I found that someone had already put together a similar solution. Feel free to check that out as well: gist.github.com/ronaldsmartin/47f5239ab1834c47088e.

MedTimer - Shared Medication Timer and Tracker Web App

Joshua Tzucker — Tue, 29 Dec 2020 09:44:28 GMT

Intro

I’m writing this project post in late 2020, but I actually coded and built this out in early 2018. It was one of my earliest PHP projects, and also an experiment in starting from scratch (no frameworks, no starter package). I’m making a writeup for it years later, because, looking back I actually really like what I built and feel it deserves one. I kind of wish I had done more with this project, and perhaps built it into a larger-scale SaaS. As it is though, it served its purpose for a long time.

This project came about at the request of a friend of the family. Her son, who has a chronic health condition, was taking special medication to help his body with ingestion. He would take a pill, and then have 90 minutes to eat. Once the 90 minutes were up, he would need to take another pill if he wanted to eat more, even for tiny snacks. The part that I was asked if I could help with was how to track this; at the time he was really young, in elementary school, and because he was often moving between environments (home, school, friend’s houses) it was hard for the adults in his life to have a central place to know:

A) Had he recently taken a pill?
B) And, if so, were the 90 minutes up yet?

I thought this sounded like a great opportunity for me to learn some more full-stack development, and build a tiny web app that could have a shared login and “virtual” timer.

Features

Core Functionality

The core functionality of the app is fairly simple:

Each adult is given their own “passphrase” to log into the system. This is so that, if the timer is started, everyone can see which adult was the one to administer the pill and start the timer.

The main admin account has the ability to add and delete these passphrases whenever they want. New school nurse? Give them their own login!

Show / Hide Passphrase Demo

Sorry, your browser doesn't support embedded videos.

After logging in, there is a giant status indicator area, which tells everyone:

If the timer is running, and he can currently eat
If he can eat, for how much longer (how much time is left of the 90 minutes)

Sorry, your browser doesn't support embedded videos.

After the 90 minutes are up, the countdown automatically ends and the status changes. Additionally, if the timer was started on accident, it can be manually cancelled (which still gets saved to history). Or, if you forgot to start it and the pill has already been taken recently, you can offset the start time manually.

All entries are automatically recorded and summarized, under the “History” page:

Show / Hide History Demo Video

Sorry, your browser doesn't support embedded videos.

Extra Feature - Medication Scheduling

The person I built this for was so pleased with the timer feature, they asked if I could add on another feature; something to replace their paper-and-pen solution for tracking medication and therapy that was being administered. I said yes, and quickly put together a “schedule” feature.

You can configure medications and therapies, specify time of day, and for therapies, even specify and track the required duration (e.g. 30 minutes of hand grips).

👇 Tracking Medications Sorry, your browser doesn’t support embedded videos.

👇 Tracking Timed Therapies Sorry, your browser doesn’t support embedded videos.

Tech Stack

At the time, I wanted to get a bit more familiar with PHP and general full-stack basics, so I used this project as an exercise in using some PHP, HTML, CSS, JS, and SQL fundamentals.

I built it with zero frameworks, and only a tiny bit of third party for front-end stuff; backend is all vanilla PHP. For tying together the front and backend, I wrote a tiny API in PHP to handle the requests.

Backend
- PHP
- PDO + SQLite
- Apache
Frontend
- HTML
- CSS
- JavaScript
- JQuery (just a little 😅)
- DataTables

Outcome / Wrap-Up

I received very positive feedback from the family that this was built for, and it served its function hundreds of times; to me, that is what matters most.

Overall, I’m happy with what I built, although I might revisit this at some point in the future as a fun project to rebuild at-scale.

Common CSS Color Variables and Properties in Theming

Joshua Tzucker — Fri, 25 Dec 2020 01:23:27 GMT

Collected Framework Results
Examples of Custom Variable Sets
Some Common Patterns
Summary
More Resources

Intro

As I’ve dipped in and out of different front-end theming frameworks, something that I have noticed is a lack of standardization when it comes to global theme color variables.

For the unfamiliar, many front-end frameworks provide global (e.i. shared, or root) color variables that can be easily re-used across your app, and often support customization and hot-swapping of values for instantaneous theme switching (e.g. between “dark mode” and “light mode”).

This blog post is the result of a small research and classification attempt by myself, to see if I can identify some common trends and tools that might be useful within this area of theme development.

Collected Framework Results

Ionic Framework
- Variables:
  - Primary
  - Secondary
  - Tertiary
  - Success
  - Warning
  - Danger
  - Dark
  - Medium
  - Light
- Tools:
  - Color Generator
  - New Color Creator
Boostrap 4
- Variables:
  - Primary
  - Secondary
  - Success
  - Danger
  - Warning
  - Info
  - Light
  - Dark
- Tools:
  - Theme builder and customization
Material Design
- Variables:
  - Primary
  - Primary Variant
  - Secondary
  - Secondary Variant
  - Background
  - Surface
  - Error
  - On Primary
  - On Secondary
  - On Background
  - On Surface
  - On Error
- Tools:
  - Color Tool
  - Theme Builder
Vuetify (Material framework for Vue)
- Variables
  - Primary
  - Secondary
  - Accent
  - Error
  - Info
  - Success
  - Warning
- Tools:
  - Color Palette Generator

Examples of Custom Variable Sets

As noted in the intro, the whole reason behind this post was because I noticed how many different combinations of global CSS variables I was seeing. In addition to noting the variance between design frameworks (see above), I also wanted to callout some interesting examples of highly customized sets of global CSS variables.

Twitch (twitch.tv)

Twitch has, literally, hundreds of :root variables, some of which are pretty standard, such as --color-accent, while others are pretty specific to their use-case, such as --color-fill-channel-status-text-indicator-offline.

There is a built-in theme switcher, to switch between dark and light. Switching between them swaps a class in the top level element (e.g. tw-root--theme-light), which in turn changes which values are assigned to the root-level variables. This is a common CSS-variable-based theming approach.

StackOverflow (stackoverflow.com)

StackOverflow makes use of a lot of patterns covered in this post, as well as some advanced patterns that shift more theme processing towards native CSS and away from pre-processors.

For example, they define the primary color in separate chunks (both HSL and RGB), and then dynamically compute derived colors by combining those values and optionally applying calculations. Here is how they compute a derived color, which is a shade of the primary color darkened by 30%:

:root {
	--theme-primary-color-h: 26.53846154;
	--theme-primary-color-s: 90.43478261%;
	--theme-primary-color-l: 54.90196078%;
	--theme-primary-color-darken-30: hsl(var(--theme-primary-color-h), var(--theme-primary-color-s), calc(var(--theme-primary-color-l) - 30%));
}

Some Common Patterns

Here is a breakdown of some common recurring patterns I saw:

Color Wheel based names

Although the values people pick might not always correspond with it, these variable names seem to be based on the standard color wheel.

Examples:
- --primary: red
- --secondary: orange
- --tertiary: orangered

Named scaled color palettes / swatches

Since there is no such thing as just “red”, many themes maintain their own palette of colors, with different weights / intensities:

Examples:
- --orange-400: #f9872c;
- --orange-500: #f4740d;
- --yellow-400: #d8ba45;
Notes:
- This does not appear to be standardized either. Many sites and frameworks use a scale that seems to correlate with font weights (100-900), where others might use 0-10, or -a# for properties using alpha values.
- I think part of the reason why this is so common is because creating “shades” in CSS is a lot more complicated than in SASS (where you have access to native functions like lighten() and darken()). Another consideration is performance; having these values pre-populated is probably has significantly better performance than a bunch of calls to calc(), especially if you have dozens of colors and derivative shades / variations.

Contrast Variations

When specifying a color, you have to consider all the different ways it will be used, and that includes how it contrasts with the colors around it. Thus, thus some systems like to explicitly declare contrasting color values.

Often, the contrast value is used for text or icons; for example, the Material Design system has an entire section devoted to what they call “On Colors” - this is a color that should be used for text or iconography when it appears “on” another surface.

:root {
	--primary: #261ff2;
	--secondary: #ea80fc;
	--on-primary: #fafafa;
	--on-secondary: #000504;
}

The main concern with these values should be of proper contrast and accessibility; you don’t want yellow text on a yellow background.

In some systems, these contrasting colors might be automatically derived based on calculated minimums (e.g. WCAG / WebAIM)

You can also see these variables reflected in Material’s custom theme color docs. And the Ionic Framework places particular importance on contrast in color variables

Semantic Named Colors

Although some might call primary and secondary semantic variables, I think the best examples of what one might classify as semantic color variables are things like:

:root {
	--success: #28a745;
	--info: #17a2b8;
	--warning: #ffc107;
	--danger: #dc3545;
}

These come directly from Bootstrap 4.1’s root variables

For those unfamiliar, the term semantic, within the context of variable naming, usually means that the name itself conveys the meaning and/or application of the value held, rather then relying on said value.

The benefit to these types of values is largely in readability and ease of refactoring. For example, imagine that you have a bunch of components that show errors. You might improve refactor-ability and ease of reading by changing:

background-color: var(--red-600);

background-color: var(--danger);

Brand Colors

Although primary and secondary colors usually are the main brand colors, a brand will often have a large palette that makes up their branding guidelines, and you might see this reflected in CSS variables.

For example, if you open up a new empty Chrome in tab, you are likely to see variables like --google-blue-refresh-700. Or, on twitch.tv, you might see a smattering of --color-amazon throughout the site, as well as things like --color-brand-accent-cherry.

If you have a really massive application, and especially if it has third party assets or different themed areas, it might be helpful to clearly differentiate between variables that are part of the application theme, vs part of your actual overall brand (and stricter brand guidelines).

Color Parts / Calc Helpers

I’m not trying to diss CSS (I love CSS!), but the truth is that there are not a whole lot of built-in functions for manipulating hex based color values (especially compared with things like SASS).

One really neat trick to work around this fact is to store color values as their individual parts (either R-G-B or H-S-L), and then put them together and manipulate them in the process.

Here is an example of this in action:

:root {
	--theme-primary-color-h: 242deg;
	--theme-primary-color-s: 56%;
	--theme-primary-color-l: 62%;
	--theme-primary-color-darken-30: hsl(var(--theme-primary-color-h), var(--theme-primary-color-s), calc(var(--theme-primary-color-l) - 30%));
}

Note: The hue value in CSS can be written as either a number (242) with deg unit implied, or with angle unit (e.g. 242deg). See docs for details.

This writeup by Una is amazing guide to using this approach to dynamically generate derived colors from your CSS variables.

Before you dismiss this approach, I’ll note that several high-volume production sites (such as StackOverflow) use this technique in CSS

Differentiating Variable Types

You might notice that some applications / sites are very careful to distinguish color properties, and they will always use things like --border-color: instead of just --border:.

A good reason to do so is if your app stores (or could in the future) non color values in CSS custom properties. Not everyone always remembers this, but CSS custom properties can actually store all types of values beyond just colors; you can store strings, numbers, or any CSS value.

Thus, it might be wise to differentiate color values from the start, since that will make refactoring and readability easier (“Is --border for border color or border value, or border px?”).

Summary

Although I enjoy consistency and standardization, I think seeing a moderate level of variance in how CSS custom properties are getting used and their naming conventions is actually a good thing. It speaks to the flexibility of CSS and what I believe is important part of web development, which is to use what works best for the situation and your users, not necessarily what the most popular framework dictates.

More Resources

Prototypr.io - Anna Molly: Basic UI color guide
- Excellent breakdown of the different types of colors in a color system
Sebastiano Guerriero: Create your design system, part 3 - colors
Generic color palette creators:
- Coolors
- Colormind
- More listed here
Material Design Docs - “The Color System”
- The Material Design docs are a fairly comprehensive resource
Examples of theme variable sets
- DiscordStyles/Slate
- Materialize CSS: _variables.scss
- Vuetify: Default theme
- Bootstrap: _variables.scss

Cover image background - Photo by Keila Hötzel on Unsplash

Git SHA Badges for Deploys - Know Which Commit is Live

Joshua Tzucker — Wed, 09 Dec 2020 06:00:10 GMT

Demo Repo

I’ve taken the approaches covered in this post and created a fully-functional demo repo to demonstrate them in action:

💾 github.com/joshuatz/git-sha-badges.

Intro

When building a project that lives in a Git repository and auto-deploys elsewhere, a common concern is how to be 100% absolutely sure that the live “deployed” version is up-to-date. If I go to the live, production application, how can I trust that what I am seeing is the latest pushed code in my repository?

Just because my CI/CD pipeline passed does not necessarily mean that the latest commit is live yet. Furthermore, if something goes bad, I want to be able to know at a glance exactly what code is live in production. A simple “pass / fail” badge cannot tell me that.

Close, but Not Good Enough

Many automated deploy systems provide a build status badge, but this doesn’t tell us much:

Standard Netlify Status Badge:

Many apps also display in the deployed UI the current version string of the app (e.g. 1.8.2-beta.1.10), but this actually still leaves a lot of room for ambiguity; what if a developer used --force to override the head of the main branch? What if there have been commits added to the main branch, but no new release authored? Etc.

Instead of a version string, a nice alternative (or better yet, an addition) is to use the actual SHA of the latest commit as a status indicator. This makes it very easy to compare against the latest commit in version control, and verify that the correct set of changes are live.

Furthermore, no two commits can have the same SHA, even if --force has been used, so if I know the SHA that is “live”, I know exactly what code is running, and can even quickly “checkout” that exact version of code in my local environment.

💡 Similar to how you can use git checkout {branch_name}, you can also use git checkout {SHA} to checkout your entire repo at that commit. It kind of feels like time travelling!

Methods for Creating SHA Badges

The general concept that we are going to employ is that our build step, which runs on every deploy, is now going to save the current SHA of the branch (aka the tip) in an accessible format that can be used for visual indicators.

Two easy approaches that build off this concept, and which I’ll explore here, are JSON files and raw SVG files.

JSON File

The basic idea with the JSON file approach is that, at build time, we stash the value of the latest SHA in a shared JSON file. This makes it easy to reuse across our codebase, possibly even at runtime, and in a widely compatible format.

The JSON file approach is my favorite, for a couple of reasons:

It is easier than dealing with SVG string building, but can be combined with that approach easily (more details further below)
The JSON file can be re-used, and even imported into runtime code (assuming they share access)!
Usable by APIs and 3rd party services (including Shields.io for badges)

JSON File - Getting and Saving the SHA

The implementation of this task could be accomplished in an endless number of ways, and the best option probably depends on the specifics of your build environment, project type, and operating system.

For a generic example, here is how I how I have integrated this approach into a NodeJS based project, within a package.json file:

{
  "scripts": {
    "build-badge": "$NETLIFY && git rev-parse --short HEAD | xargs -I % printf '{\"sha\":\"%\"}' > ./static/build-info.json",
    "build": "(yarn run build-badge || true) && gatsby build"
  }
}

The most important thing about the above command is that it modifies our build command, which used to be just gatsby build, to include a step that saves the current SHA to a JSON file (./static/build-info.json). If you wanted to, you could replace a lot of the bash stuff with a NodeJS or python script that does the same thing.

If you want a breakdown of the above command, you can view it below

Open Command Explanation

To break this down further, the build-badge command is composed of:

$NETLIFY is an environmental variable that only exists within my build environment (Netlify servers).
- You could use any value that evaluates to true in your build system
- Or, if you always want this file generated regardless of local vs production, you could omit the whole $VARIABLE && part
git rev-parse --short HEAD
- This is a git command to get the short version of the last commit SHA
- You could use the full SHA instead; just omit --short
- If you are looking for a lot of nifty git commands like this one, here is a shameless plug to check out my Git Cheat Sheet (this command is on it!)
xargs -I % printf '{\"sha\":\"%\"}'
- Not going to get too much into this, but it is a fancy way of passing the SHA from the last section into a stringified JSON representation
> ./static/build-info.json
- Save the generated JSON string to a file. This could be any file, and it doesn’t even necessarily need the .json extension
- You should make sure that whatever directory you save it to is going to be accessible after the build is complete, either internally and/or externally (hosted)

And I’ve changed my build command from gatsby build to (yarn run build-badge || true) && gatsby build:

This is an easy way to make sure that gatsby build, the true build command, always runs regardless of the success or failure of the build-badge command.

Again, as an alternative to a long bash command you could alternatively extract and save the SHA entirely in your favorite scripting language of choice (NodeJS, Python, etc.).

JSON File - Using the Stored SHA

As-is, once we have our JSON file, that alone is enough to be able to add badges or pull the SHA value into front-end code. For example, we can use the dynamic badge feature of Shields.io to easily get a SVG URL that displays our SHA. Something like:

<img src="https://img.shields.io/badge/dynamic/json?color=blue&label=Deployed%20SHA&query=sha&url=https%3A%2F%2Fgit-sha-badges.netlify.app%2Fbuild-info.json" alt="Deployed SHA" />

Notice the importance of:
 - query=sha
 - url=HOSTED_SITE/build-info.json

If you wanted to pull it into your code, in CommonJS, that is as easy as:

const buildInfo = require(`${BUILD_DIR}/build-info.json`);
console.log(buildInfo.sha);

SVG File

Rather than displaying your SHA in front-end code by pulling it in via AJAX or inside framework components (e.g. a React component), you could also pre-generate SVG badges at build time, with just a little bit of code.

The easiest approach is to have 99% of the SVG badge ready-to-go, stored as a string. You can use whatever method you prefer for designing your badge. The main thing to do is leave room for the SHA to be inserted at build time, as a <text></text> element inside the SVG.

At build time, you combine your SVG template string with the latest SHA (this is where it is handy to have that build-info.json already generated), and then write out the resulting string to an actual .svg file, which can be served.

You can checkout how I put this all together in build.js in my demo repo.

In pseudo code, this approach might look something like:

// Compose full SVG string
const svgStr = '<svg>' + svgTemplateStr + '<text>' + sha + '</text></svg>';
// Save to SVG
saveFile(svgStr, 'my-badge.svg');

And don’t forget, you can get as fancy as you want with your generated SVGs 😄

Bonus: Github Commit Badges

If all you want is a badge that shows the current HEAD of a particular branch in Github (i.e. the most recent pushed commit) - you can actually do this entirely with just Shields.io and the public Github API (no credentials required).

Caveat: this requires that your repo is published as public

First, you need get the API endpoint that returns HEAD info, for your specific repo and branch combo. The syntax follows:

https://api.github.com/repos/{USER}/{REPO}/git/refs/heads/{BRANCH}

So, for example, to get the currently pushed tip for my main branch in my demo repo for this post (joshuatz/git-sha-badges), I can use api.github.com/repos/joshuatz/git-sha-badges/git/refs/heads/main.

If you try that URL, you will get a JSON response, including:

{
	// Other stuff
	"object": {
		"sha": "____",
		"type": "commit"
	}
}

Now, we can plug that endpoint directly into Shield’s amazing dynamic badge generator, which accepts a JSON endpoint and lets us control badge settings through query string parameters. By tweaking the parameters, we can get an orange badge that displays the object.sha value in the endpoint:

API Endpoint:
  https://api.github.com/repos/joshuatz/git-sha-badges/git/refs/heads/main

Shield Badge Base Endpoint
  https://img.shields.io/badge/dynamic/json

Shield Badge URL (raw)
  https://img.shields.io/badge/dynamic/json?color=orange&label=Github SHA&query=object.sha&url=https://api.github.com/repos/joshuatz/git-sha-badges/git/refs/heads/main

Same URL, encoded:
  https://img.shields.io/badge/dynamic/json?color=orange&label=Github%20SHA&query=object.sha&url=https%3A%2F%2Fapi.github.com%2Frepos%2Fjoshuatz%2Fgit-sha-badges%2Fgit%2Frefs%2Fheads%2Fmain

---

Final badge code

---

Markdown:
  ![Github SHA](https://img.shields.io/badge/dynamic/json?color=orange&label=Github%20SHA&query=object.sha&url=https%3A%2F%2Fapi.github.com%2Frepos%2Fjoshuatz%2Fgit-sha-badges%2Fgit%2Frefs%2Fheads%2Fmain)

HTML
  <img src="https://img.shields.io/badge/dynamic/json?color=orange&label=Github%20SHA&query=object.sha&url=https%3A%2F%2Fapi.github.com%2Frepos%2Fjoshuatz%2Fgit-sha-badges%2Fgit%2Frefs%2Fheads%2Fmain" alt="Github SHA" />

And, here it is (this is a live badge!):

Wrap Up

I want to wrap up this post by pointing out that, although I gave some specific examples and code samples, the approaches throughout this post are fairly generic in nature and could be extended to solutions beyond just adding Git SHA badges.

For example, if you maintain several build and deploy targets (e.g. dev, test, staging-alpha, production, etc.), you could re-use most of the solutions through this post to add visual indicators to each environment that makes it clear what system you are looking at.

Hope this post helps someone out there! (And kudos to anyone that understands and appreciates what my fancy SVG example is referencing).

Rendering Non-Video Content in PiP - Experimental Project

Joshua Tzucker — Wed, 02 Dec 2020 20:30:45 GMT

What Is It?

This was a project to explore creative ways to use the experimental Picture-in-Picture Web API to render non-video content in a floating always-on-top window. Eventually this ability might be offered natively by the browser (e.g. by PiP v2, or a different Web API), but I wanted to showcase some use-cases for why we should want it in the first place and the value it could bring.

I believe that giving webpages the ability to draw content off-page, in windows they could control, opens up some amazing possibilities, and would be a step towards parity between native desktop apps and web apps. This project is an exploration of that idea.

Where to Get It

💾 Source Code: GitHub

🚀 Live Demos: pip-rendering-fun.netlify.app

Demo

Here is a demo showing how it could be used to display real-time stats, in a persistent floating display:

There are quite a few demos that I built out, including using PiP for:

A time tracking application
Displaying info about itself, through the web API
Displaying the user’s own display / desktop
Mirroring a drawable surface, in real-time
The analytics / stats demo, as shown above

To see all the demos above, and try them for yourself if you have a compatible browser, they are deployed at pip-rendering-fun.netlify.app.

What I Used

TypeScript
Web APIs
- HTML Canvas
- MediaSource
- PictureInPictureWindow
- MediaDevices
Netlify (deploying the demos)
Documentation! (Both consuming and writing my own)
Markdown
Vanilla HTML and JavaScript
Browser “quirks”

A lot of the web APIs I used are considering “experimental”; in fact, I had to use TypeScript’s ambient declaration support to augment the Document interface, since the types for those APIs have not yet made it into the lib.dom.d.ts that ships with TypeScript!

This was also a great opportunity to learn more about web standards, and how different browsers end up implementing the same specification (surprise, surprise - it is not always the same!). I also spent a fair amount of time writing my own documentation on the topic; covering some of the issues I ran into, discrepancies in how APIs were implemented, and browser quirks.

Investigating Stalled Video Segments in MediaSource Buffers

Joshua Tzucker — Fri, 27 Nov 2020 18:00:18 GMT

I just finished finding the culprit to a very vexing problem, and I want to share my findings in the hope it will help someone.

I’ve been working on constructing an HTML video dynamically, using MediaSource APIs / MSE and the methods outlined in my recent post on MediaSource approaches. I’m trying to build a seek-able video, out of distinct WEBM files, as opposed to byte chunks out of stream. Loading the files directly via src=file.webm or one at time with MediaSource was working fine, but appending multiple segments together was triggering some bizarre behavior.

Issue

After loading multiple segments via MediaSource’s SourceBuffer.appendBuffer() and closing off the stream (MediaSource.endOfStream()), there would be no fatal errors in the console, but the video element would exhibit the following behavior (not always all at the same time):

Player element would let me press play, but then would “stall” - time does not move forward, and UI shows video as “playing”, but it clearly is not
- Sometimes it would play for a very brief amount (milliseconds) before stalling
Player element would be seek-able, and I could scrub through timeline with correct preview. However, manually scrubbing position and then trying to resume playback would still result in stalled state
Occasionally I would get the well known play() request was interrupted by a call to pause() error, but this would be despite any calls pause()
- I think this was actually caused by the stalled buffer internally (not my code) requesting a pause to fire. This seems to be backed up by this comment: “Say you have audio+video in a single <video> tag. If just audio underflows, chrome will immediately pause playback”

These issues occurred across both Chromium and Firefox, but again, not always together or consistently.

Investigating

I’ll admit it; my first step at investigation did not get me very far. I tried a bunch of trial-and-error, searching through docs on MediaSource, and switching between the sequence mode of SourceBuffer and segments mode. Nothing helped.

Chrome Media Inspector

I next stumbled across an article that mentioned that Chrome has a “Media” inspector tab in DevTools. Holy smokes is this thing awesome! Firing up the Media inspector and running my code again gave me much better insight into what to focus my investigation on: audio buffering.

Looking at the timeline view, this is what I pretty consistently saw when the video stalled:

Uh… that’s not good 😰. Let’s look at some of the events that are firing when it starts stalling:

kBufferingStateChanged

event = {
	audio_buffering_state: {
		reason: "DEMUXER_UNDERFLOW",
		state: "BUFFERING_HAVE_NOTHING"
	}
}

kBufferingStateChanged

event = {
	pipeline_buffering_state: {
		for_suspended_start: false,
		reason: "DEMUXER_UNDERFLOW",
		state: "BUFFERING_HAVE_NOTHING"
	}
}

Hmm. “DEMUXER_UNDERFLOW” doesn’t exactly have a nice sound to it… Also, a message like this would occasionally show up in the “Messages” panel:

Skipping audio splice trimming at PTS=11626999us. Found only 1us of overlap, need at least 1000us. Multiple occurrences may result in loss of A/V sync.

That definitely seems related, and might also point to an audio buffer issue. What’s the deal?

To make this more confusing, there was a lot of things I could rule out:

Buffer / Memory overflow / auto eviction: SourceBuffer is not an endless tube you can fill; it is more like a temporary reservoir. As such it has limits and is subject to garbage collection. However, I could rule this out because my tests were using very small files, and the overall buffer should have been well below the 12 MB audio / 150 MB video limit.
Mismatched codecs: Although I couldn’t rule out issues with my source files (more on this shortly), I could rule out that I was maybe using codecs that the browser did not support, as:
- There were no errors to that effect
- Calling MediaSource.isTypeSupported(mimeStr) showed support
- My codecs matched the MSE specification / WEBM specs
- Video playback worked just fine if videos were loaded directly, bypassing combined appendBuffer calls
Network issue
- Although there were no indicators of issues with fetch() or any network requests, to be safe I even was trying loading videos directly via base64 strings in the JS. Same issues.

To rule out something wrong with my actual JavaScript code that was doing the loading (no errors in console does not rule this out), I swapped out my source files for some others that are commonly used throughout the web as WEBM samples. Issues immediately went away, signaling that this was indeed an issue with my source files.

At this point though, that still didn’t clear up a huge amount. These files played just fine in standard media players, and to reiterate, also played just fine in <video> elements when loaded one at a time.

Checking Files with FFMPEG

I should have tried this from the very start, but at this point I realized I absolutely needed to inspect the video files themselves. Although I had never done this kind of thing before (other than viewing what VLC provides in their info popup, and that sort of thing), FFMPEG makes it super easy to do, with the -i inspect flag, or with the ffprobe tool.

To inspect a file, use:

ffmpeg -i FILE_NAME

# Or, with ffprobe:
ffprobe FILE_NAME

For more advanced file inspection, check out ffprobe tips.

When I inspected my source files, I saw something reallllly funky:

Duration: 00:00:01.65, start: -0.001000, bitrate: 22 kb/s
Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv, bt709/unknown/bt709), 144x96, SAR 1:1 DAR 3:2, 10 fps, 10 tbr, 1k tbn, 1k tbc (default)
	Metadata:
		DURATION        : 00:00:00.207000000
Stream #0:1: Audio: opus, 48000 Hz, mono, fltp (default)
	Metadata:
		DURATION        : 00:00:01.648000000

So, not only is it a little strange that each stream has its own duration metadata (I don’t think thats required), but it is alarming that the duration for the video segment completely does not match up with the audio! It sounds like these numbers don’t have to match exactly, but those numbers are so wildly off it seems suspect.

Note: A negative “start time” is actually an OK thing

Fix

I first tried the command suggested here, to patch the duration with FFMPEG:

ffmpeg -i broken.webm -c copy -fflags +genpts fixed.webm

However, that didn’t work.

At this point, it seemed likely that the file was just seriously malformed. If there are chunks of missing data or corrupted bytes, that is not something FFMPEG copying is likely going to be able to fix.

The file was created with a video editor (Shotcut), which has a lot of control over video output encoding, but rather than tweak every setting under the sun, my first thought was to update the program to the latest version. That… actually ended up fixing the issue. My guess is internally they upgraded their bundled version of the VP9 (or Opus) encoder, or how they were using it, and this in turn resulted in a “more valid” export file for my project.

Here is a sample of the inspection of the “fixed” file:

Duration: 00:00:01.71, start: -0.001000, bitrate: 10 kb/s
Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv, bt709), 144x96, SAR 1:1 DAR 3:2, 10 fps, 10 tbr, 1k tbn, 1k tbc (default)
	Metadata:
		DURATION        : 00:00:01.707000000
Stream #0:1: Audio: opus, 48000 Hz, mono, fltp (default)
	Metadata:
		DURATION        : 00:00:01.708000000

Other Remarks

I also want to point out that in my previous MediaSource post and set of demos, I ran into a variant of this issue that only occurred in Firefox, and ffprobe results looked fine. The issue was specifically around multi-file appends, and I ended up just switching all my input files from VP9 to VP8, which seemed to resolve the issue.

Appending Videos in JavaScript with MediaSource Buffers

Joshua Tzucker — Wed, 25 Nov 2020 16:56:29 GMT

Intro
- Link to All Code Demos
Getting Dummy Data Ready
- Loading Pre-Generated Dummy Data
General Approach
Multi-File Support
- Source Buffer Modes: Segments vs Sequence
- Better Option: Streaming Formats
Research and FAQ
- Existing Tutorials and Examples
- Issues I Ran Into

Intro

In my previous post about using raw data in JavaScript, I alluded to the concept of dynamically loading data into video elements using the newer Media Source Extensions API (aka MSE) and its collection of interfaces, buffers, and sources.

I just finished brute-forcing my way through some of the basics (and I do mean basics) of MSE, and wanted to share my findings on dynamically loading data into videos, specifically by using the MediaSource interface, SourceBuffer, and the appendBuffer method. MSE is a really complicated topic, and distinctly separate from my normal area of programming, so my examples are going to be a lot more basic level than what I normally post. Codecs and math-heavy coding are just not my bread-and-butter 🤷‍♂️.

Anyways, I’m posting this because although my examples are simple, there is not nearly enough info out there on the web on how to even start with MSE / MediaSource. I’m hoping this helps someone.

Link to All Code Demos

I have coded several fully-functional and well-commented examples that use the approaches from this post to dynamically load video. These will be discussed further on, but if you want to jump right to the source code, you can find it at:

github.com/joshuatz/mediasource-append-examples

Getting Dummy Data Ready

First thing; I need some video clips that are going to work with MediaSource (see “Research” section for more info on file formats). For my demo, I’m going to stick with Webm as the container, and VP9 as the codec. I could use FFMPEG to convert videos I want to use to the right format directly, but in my case, I used Shotcut to handle both the trimming and conversion process.

If you don’t have your own files to work with, I compiled a list of resources I found as I worked on this.

💡 I was a little lost on the state of video support across different browsers, until I found this awesome writeup by @Vestride: Encoding Video for the Web. Mozilla also has a great guide. General idea is AVC-H.264/MP4+AAC offers widest compatibility, but is commercially licensed, whereas a better option with less support is VP8 or VP9 with Opus audio.

Loading Pre-Generated Dummy Data

I know that I’m going to need the video files that I want to load accessible from within my JavaScript code. This leaves me with two main options.

The first is pretty standard. Upload the video clips somewhere (or find already hosted clips), and from JavaScript, use fetch() to retrieve them via network request and get the binary blob.

The other approach is something I want to do for fun, and to reuse some concepts from my last post on binary formats. I want to store my video clips, directly in JavaScript!

To do this, first we need to get the binary data into a “stringified” format. You can’t just open a video file with Notepad and copy and paste the text into a JS file. However, Base64 encoding is a fast and easy way to store binary data:

base64 --wrap=0 sample_vid.webm > sample_vid-base64.txt

If you don’t have access to a local base64 converter, you can also use an online converter tool

Now, I can store the base64 string directly in JS:

const vidClip = `GkXfo59...`;

However, when I’m ready to append the clip into a source buffer with MediaSource, I’m going to have to convert it back into a binary blob (more on this later).

General Approach

For dynamically loading raw video data into a <video> element using the MediaSource API, the primary method we need to be focused on is the sourceBuffer.appendBuffer(buffer) method. This takes a chunk of raw data (as an ArrayBuffer) and appends it to an existing SourceBuffer instance.

However, to actually get to the point where we can use this to append to a video, there is a bunch of setup necessary, since our SourceBuffer should belong to a MediaSource instance, which in turn should be connected to a <video> through an Object URL.

The general steps to put it all together look something like this:

Get a reference to a <video> element (in the DOM)
Create a new instance of the MediaSource interface (const mediaSource = new MediaSource())
Create an Object URL that points to that raw source. Point the src attribute of the video element to it
- Example: videoElem.src = URL.createObjectURL(mediaSource)
Attach a listener to the mediaSource for the sourceopen event (or define as the callback property onsourceopen)
- We need this to fire before we can attach a buffer
- You can also check the mediaSource.readyState property, to see if it == 'open'
Once sourceopen has fired, create an instance of the SourceBuffer interface, which will hold data, and attach to to mediaSource with the addSourceBuffer method:
- Example: const sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp9,opus"');
Now that we have the buffer, we can start loading data with the sourceBuffer.appendBuffer(arrayBuffer) method
Every time that we append a buffer, we also need to listen for the updateend event that will fire after the browser is done with the operation
- If we have more chunks to append, we can start a new append operation, and repeat the listener cycle over again
- If we don’t have any more chunks, we need to close out / signal the end of the stream with the mediaSource.endOfStream() method. We also might want to call videoElement.play() at this point, if the video is not set to autoplay (note: video must be muted or else this will fail due to autoplay rules).

- These exact steps are used in my "Standard" approach example

Loading...

Multi-File Support

In discussing multi-file support, it is important to make a clear distinction between multi-file video loading vs streamed “chunks”, as these often get confused. A single video file (sample.webm) can be split into multiple chunks (e.g. ArrayBuffers), but you can also have multiple video files (sample_a.webm, sample_b.webm), which each get their own buffer (and/or are split into even smaller chunks for each file).

You can use AppendBuffer for both data chunks of a single video file, or entire disparate video files.

This section is going to be discussing multi-file appends, as opposed to single-file chunk appends, as there are some special caveats that apply when appending completely separate files.

For chunks of the same file, the standard approach above for loading buffers should work just fine (as a basic example).

Source Buffer Modes: Segments vs Sequence

One of the primary caveats that apply to multi-file appends is how they are handled depending on the sourceBuffer.mode property (spec).

In sequence mode, you are forcing new appends to be treated as adjacent to the previous, regardless of timestamps held within the files themselves. This works to our advantage with multiple file appends, but would be bad if we were appending chunks in a random order and relying on their internal timestamps for placement.

- Sequence Mode Example

In sequence mode, you will also see this warning in Chrome when using it with multiple files:

Warning: using MSE ‘sequence’ AppendMode for a SourceBuffer with multiple tracks may cause loss of track synchronization. In some cases, buffered range gaps and playback stalls can occur. It is recommended to instead use ‘segments’ mode for a multitrack SourceBuffer.

In addition to the likelihood of sync bugs that the above warning is pointing out, there is also a chance that support for using sequence mode with multiple files might be deprecated in Chrome. Their recommendation would be to switch to segments.

In segments mode, the internal timestamps of the content of the buffer determine placement; this means that you can append chunks of a video in any order you want, and as long as they have “coded frames” with timestamps. This is great for handling chunks passed over a network connection, as you cannot guarantee the order in which they will be returned from the server. However, this works to our disadvantage for multiple-file appends, as relying on internal timestamps for placement makes no sense if appending multiple files as chunks. The files do not know about each other when they are encoded (why would they?), so their timestamps only describe their own chunks, not their relation to other files on the timeline. In practice, this usually means that, without tweaking your code, if you append multiple files with mode set to segments, they will overwrite each other in the buffer, and you end up with only one video getting played back.

So, to use segments mode with multiple files, you need to manage the timing offset between file appends yourself. My demo, here, shows this in action, using sourceBuffer.timestampOffset to move the pointer to where the next append call will place data.

- Segments Mode Example

Better Option: Streaming Formats

For live video feeds, or just a really robust video loading approach that can handle things like adaptive bitrate streaming, resolution switching, etc. - you probably want to switch to using a streaming video format paired with a well-supported player component.

The two most popular streaming formats are DASH (aka MPEG-DASH, Dynamic Adaptive Streaming over HTTP) and HLS (HTTP Live Streaming). There are tons of differences between these two formats, better explained by those more qualified to do so than myself. However, the general gist is that DASH is newer, but gaining in native adoption and power, whereas HLS is older and not growing as fast, but has a lot of legacy support.

These are some good starting places for learning more:

MDN: Live Streaming Web Audio and Video
Cloudflare: HLS vs DASH
Eleven-Labs: DASH & HLS

In general, these streaming formats are complicated to implement by hand, so you would want to use them with a well-known and supported player or library, such as:

Multi-format players, with extra features:
- Google’s Shaka Player
  - Supports both DASH and HLS, plus tons of other features
- Bitmovin Player (Commercial)
Format specific
- Video-Dev: hls.js
- Dash Industry Forum: dash.js

There are also paid platforms out there that handle even more of the process, such as automatic transcoding of a single high-resolution input file into multiple bitrates and resolutions, and player embed code generation

Research and FAQ

These are notes that I jotted down while learning about MSE and trying to build out my example demos.

Important questions

MediaSource vs MediaStream
- Good answer: https://stackoverflow.com/questions/51843518/mediasource-vs-mediastream-in-javascript
Specs to know about
- w3c/media-source aka MSE (Media Source Extensions API)
- w3c/media-source
What can make up a MediaSource stream? What MIME types?
- https://w3c.github.io/media-source/byte-stream-format-registry.html
  - For video
    - video/webm (https://w3c.github.io/media-source/webm-byte-stream-format.html)
      - vorbis
      - opus
      - vp8
      - vp9
      - vp90...
    - video/mp4
    - video/mp2t
What does the MediaSource pipeline look like?
- https://w3c.github.io/media-source/pipeline_model.svg
How do you format the data that you feed into the buffer (e.g. through appendBuffer())?
- See: https://w3c.github.io/media-source/index.html#byte-stream-formats
How does WEBM fit into adaptive streaming?
- See: http://wiki.webmproject.org/adaptive-streaming. It is often used with MPEG-DASH.

Existing Tutorials and Examples

Issues I Ran Into

Uncaught DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source
- I ran into this error in Chromium, but not in Firefox. It can very often (and in my case too) be traced to mis-matched codecs (Firefox was more forgiving then Chromium in this instance)
- For example, when I saw this error, I ended up going to the Media tab of dev tools in Chrome, and then the player I was interested in and then Messages; I finally saw: Audio stream codec opus doesn't match SourceBuffer codecs.! Aha!
  - In this case, the buffer I was trying to append did correctly use the opus codec, and opus is a supported audio type to bundled with VP9. However, I forgot to declare it as part of the addSourceBuffer call
  - I had to change mediaSource.addSourceBuffer('video/webm; codecs="vp9"') to mediaSource.addSourceBuffer('video/webm; codecs="vp9,opus"')
- I think Firefox will automatically detect codecs from the data you append, whereas Chromium requires explicit declaration (just a guess)
- If you want to check if a mimetype is supported, you can query it with MediaSource.isTypeSupported(mimeTypeStr), like MediaSource.isTypeSupported('video/webm; codecs="vp9,opus"')
Gapless sequential playback (mode = sequence) does not work in Chrome (stalls), but works in Firefox just fine (for some files, see issue below this one)
- If you are using multi-track appends (e.g. separate files as opposed to chunks / stream), this is a complex and apparently buggy issue with Chromium (media-source/#190)
  - Chrome now even logs a message warning as such: Warning: using MSE 'sequence' AppendMode for a SourceBuffer with multiple tracks may cause loss of track synchronization. In some cases, buffered range gaps and playback stalls can occur. It is recommended to instead use 'segments' mode for a multitrack SourceBuffer.
- See my notes under “Segments vs Sequence”
In Firefox, multiple appends with separate files (or even chunks) seems extremely buggy depending on the input files - with certain input files, I’m seeing extremely frequent random stalls with zero logged errors. I’m wondering if it is super picky about conformance to codecs. Or it could be evicting buffer segments way faster than it should be.
- This issue seems highly related
- I don’t think it is just me; although this issue was closed, it doesn’t appear fixed in my version of Firefox (which is actually a lot newer than when that issue was posted and then marked as fixed).
- Same issue across stable, beta, and nightly build
- RESOLUTION: At least in my demos, switching over all the input files from VP9/Opus to VP8/Vorbis fixed the segments mode demo. It could be that the files I originally picked just happened to be malformed VP9, but it also seems suspect that it kept happening with so many different VP9 files…
- Also, should be noted that although Firefox does not have the awesome chrome://media-internals/ that Chromium browsers have (or the media DevTools tab), there is the Devtools Media Panel extension, which can be used with Firefox Nightly

Migrating and Redirecting Subdomains with Netlify

Joshua Tzucker — Sun, 22 Nov 2020 18:00:46 GMT

There is lots of information out there on migrating entire domains with Netlify, but less for migrating from one subdomain to another on the same host (main) domain. I wanted to put this post out there to confirm that that this is possible, and actually easy to do with Netlify Redirects.

The Old Way

Normally, If I wanted to migrate from one subdomain to another, with a site I’m hosting myself, the process might look something like:

Create new subdomain entry through CPanel
Add new DNS CNAME record for new subdomain, pointing to host
- Depending on your host, adding the subdomain in previous step might automatically do this
After DNS has propagated, implement 301 redirect from old subdomain to new subdomain
- This can be done in CPanel, with a wildcard redirect

With Netlify Redirects and Hosting

With Netlify as the host, this works a little differently.

I already had a CNAME entry pointing old-subdomain.example.com to Netlify. I added another one to point new-subdomain.example.com to the same Netlify host, but then had to sit and think for a second about the best way to move forward.

In order to use CPanel redirect settings for a subdomain, my host requires that the subdomain exist as a CPanel subdomain entity. This means it gets its own folder and CNAME entries
For redirecting the old subdomain, this didn’t really make sense; I don’t want it as a CPanel subdomain, because I’m not doing any hosting - it is still Netlify. Furthermore, CPanel wouldn’t even let me create it as a subdomain, since I still had a CNAME entry for it pointing to Netlify, and the CPanel workflow won’t allow the creation of a duplicate.
Theoretically, I could remove the CNAME for the old subdomain that points to Netlify, create a host folder, and then implement redirects to the new one, but that seems overly complicated…

Solution

A much cleaner solution emerged seconds later, and within minutes, I had everything live and working flawlessly. Here are the steps I ended up taking

IMPORTANT: Do not delete the old subdomain CNAME entry pointing to Netlify!

Create new CNAME entry, pointing new-subdomain.example.com to Netlify (Netlify Docs: Subdomains)
Go to Netlify -> Project -> Site Settings -> Domain Management, and add new subdomain as alias. Once DNS change from step 1 is working, you can set it as the primary domain.
✨ Use the Netlify special _redirects file to 301 redirect all pages from the old subdomain to the new one (Docs: Redirects) (see below for example). Push it live.

_redirects file:

# Redirect old subdomain to new one
https://old-subdomain.example.com/* https://new-subdomain.example.com/:splat 301!

And… that’s it 🤯! Seriously! No messing with CPanel subdomains or HTACCESS files! 😅

All the redirection is handled directly by Netlify, and as long as we don’t delete our old CNAME entry, visitors can still follow links to our old subdomain and get 301 redirected to the same page on the new one. There might be a noticeable dip in organic traffic, but it should only be momentary, as Google and other search engines respect 301 redirects and should very quickly recognize your new subdomain as legitimate.

Reminder

As with any domain migration, don’t forget to also update:

Analytics settings (Google Analytics, etc.)

Hard-coded internal links, and external links you have access to

Config files (e.g. siteUrl setting in gatsby-config.js)

Any where else the hardcoded URL might be used (email signatures, LinkedIn, etc.)

Using Binary Data with Front-End JavaScript and the Web

Joshua Tzucker — Sat, 21 Nov 2020 00:59:26 GMT

I’m making this post because, despite binary files being a huge part of what makes up both your local and online filesystem, I kept having difficulties finding explanations on how (and why) to deal with binary files on the web and in JavaScript. In addition, a lot has changed in the world of JavaScript over the past years, so a lot of information out there is rather dated.

Intro
Converting Binary Data to Strings
Using Binary Data as Source For Elements
Disclaimers
More Resources

Intro

First, some things that should be clarified before getting into the details:

Binary file / data: I’m using this to refer to pretty much any file or data that is not text-based. Examples: raw contents of image and video files, blobs, etc.
This post is mostly about front-end manipulation of binary data, not server-side (although some parts also apply to NodeJS)
Many examples used throughout the post require a somewhat modern web browser; e.g., it uses async functions

Converting Binary Data to Strings

JavaScript: Binary Data to Base64 (including Data URLs)

Note: If you are trying to convert from binary to Base64 in order to do something like display an image, you might not need to do this in the first place; browsers now support using raw blobs for media loading via Object URLs. I have instructions for this here.

In general, loading raw binary data via Object URL is recommended instead of using Base64 URLs

/**
 * Convert a raw binary blob into a Base64 String
 * @param {Blob} blob Raw binary blob
 * @param {boolean} [asDataUrl] Should full DataURI be returned
 * @returns {Promise<string>}
 */
const blobToBase64 = (blob, asDataUrl = false) => {
	return new Promise(res => {
		const reader = new FileReader();
		reader.onload = () => {
			/** @type {string} */
			const dataUrl = (reader.result);
			if (asDataUrl) {
				res(dataUrl);
			} else {
				// Remove MIME / dataUrl prefix
				res(dataUrl.replace(/data:\w+\/\w+;base64,/i, ''));
			}
		}
		reader.readAsDataURL(blob);
	});
}

async function runExample() {
	const binaryBlob = await (await fetch('https://www.wikipedia.org/portal/wikipedia.org/assets/img/Wikipedia-logo-v2.png')).blob();
	const base64Url = await blobToBase64(binaryBlob, true);
	document.querySelector('img').src = base64Url;
}
runExample();

Live Demo

See the Pen <a href='https://codepen.io/joshuatz/pen/zYBVowa'>zYBVowa</a> by Joshua T (<a href='https://codepen.io/joshuatz'>@joshuatz</a>) on <a href='https://codepen.io'>CodePen</a>.

JavaScript: Base64 to Binary Data Blob

First, I’ll point out that there are not a whole lot of good reasons to need to convert Base64 data to a binary blob in the browser to begin with:

If your data is in Base64, you can already use it for images, videos, etc. - by using Data URLs (aka Data URIs)
AFAIK, converting to a binary blob if you already have the data as a stored string is going to basically be double-allocating space for it

However, if you really need to do this, there are some common approaches. The newest, and easiest to use, is to use the fetch() web API, and its .blob() return method:

/**
 * Convert base64 string to blob
 * @param {string} base64
 * @param {string} [mimeType] Mime type used by data held in base64 string
 * @returns {Promise<Blob>}
 */
const base64ToBlob = (base64, mimeType = 'application/octet-stream') => {
	return fetch(`data:${mimeType};base64,${base64}`)
		.then(res => res.blob());
}

// Example:
async function runExample() {
	const imgStr = `Qk1aAAAAAAAAAEIAAAAoAAAABgAAAAYAAAABAAQAAAAAAAAAAADEDgAAxA4AAAMAAAADAAAAAAAA/wAA//8A2P//IAEgAAASAAABIAEAEgASACABIAAAEgAA`;
	const blob = await base64ToBlob(imgStr, 'image/bmp');
	document.querySelector('img').src = URL.createObjectURL(blob);
}
runExample();

Live Demo

See the Pen <a href='https://codepen.io/joshuatz/pen/OJXebjM'>Base64 to Blob</a> by Joshua T (<a href='https://codepen.io/joshuatz'>@joshuatz</a>) on <a href='https://codepen.io'>CodePen</a>.

You can also use fetch(resource).then(res => res.blob()) to convert any fetch-able object to a blob.

If you are targeting an older browser, you might not be able to use fetch(), but there are still other solutions you can use. I would recommend looking at this StackOverflow question and reading through the responses to find what works best for you: “Creating a BLOB from a Base64 string in JavaScript”

Converting Binary Files to Base64 Strings on your Local Computer / CLI

If you have access to standard *nix utilities, the easiest tool to use is the aptly-named base64 utility.

For example, if I had a JPG I wanted to get as a Base64 encoded string, I could use:

base64 --wrap=0 my-image.jpg > my-image-base64.txt
# You can also pipe to it
cat my-image.jpg | base64 --wrap=0

You’ll note that I always use --wrap=0; this is because the default behavior is to wrap (add line breaks) at 76 columns. This wastes space, and is completely unnecessary, especially if I’m just going to be pasting into a JS or HTML file.

On Windows, you can also get the string put right in your clipboard, with:

cat my-image.jpg | base64 --wrap=0 | clip

Converting Binary Files to Base64 - Online Tools

If you don’t have access to a terminal, or just prefer using a web browser, there are dozens of online file-to-base64 converters, such as this one by browserling.

As discussed above, base64 conversion can be done entirely in front-end JavaScript, so I wouldn’t trust any online tool that actually “uploads” your file anywhere.

Using Binary Data as Source For Elements

Loading Data via Object URLs

If you already have your data stored in a binary format, in memory, and need to load it into an element (for example an img element), there is an alternative to Base64 DataURLs that is much preferable: Object URLs.

These are essentially virtual URLs that point to raw data; this could be a blob of binary data stored in memory, or even a reference to a user’s local file from their OS. Because they are references to existing data location, they use less memory than Base64 strings (and you can even release the pointer after loading)

// Example: Loading an image blob
const blob = await (await fetch('https://www.wikipedia.org/portal/wikipedia.org/assets/img/Wikipedia-logo-v2.png')).blob();

const imageElem = document.querySelector('img');
// Good practice is to release once loaded
imageElem.onload = () => {
	URL.revokeObjectURL(imageElem.src);
}
imageElem.src = URL.createObjectURL(blob);

Just like Data URLs, Loading data through Object URLs works for a variety of media element, including <video> elements!

Loading Data via Base64 String Data URLs

Data URLs (formerly called Data URIs) are essentially just strings that are comprised of the data itself, as opposed to a pointer to where the data resides (how normal URLs or Object URLs work).

They follow a standardized syntax:

data:{mimeType};base64,{dataStr}

# Example
data:image/bmp;base64,Qk1aAAAAAAAAAEIAAAAoAAAABgAAAAYAAAABAAQAAAAAAAAAAADEDgAAxA4AAAMAAAADAAAAAAAA/wAA//8A2P//IAEgAAASAAABIAEAEgASACABIAAAEgAA

# If the data is *not* base64 encoded, you can leave off the `;base64` section
# Example:
data:text/plain,hello

They can be used in (AFAIK) pretty much any element with a src attribute, even including <iframe> and <script> tags!

<script src="data:text/plain,alert('hi!')"></script>
<!-- Base64: alert('hi from base64') -->
<script src="data:text/plain;base64,YWxlcnQoJ2hpIGZyb20gYmFzZTY0JykgDQo="></script>

If you are not using base64 encoding with Data URIs, you should be extra careful about characters that are not URL safe; some browsers might reject these. You can always use something like data:image/svg+xml;utf8,${encodeURIComponent(svgStr)}

Loading Data via Buffers

This is getting into the advanced part of data loading in JavaScript, but I’d like to point out that certain elements (HTMLMediaElement based, i.e. video and audio) now support loading data more directly, via streams, buffers, and the MediaSource interface. Collectively, most of these technologies fall under the “Media Source Extensions API” (aka MSE), which is an evolving specification for more advanced and dynamic media sources, such as append-able buffers.

The advantage to this approach is that it allows you to load in data in chunks, as opposed to through a single resource URL or blob. This makes it ideal for streaming applications, where video or audio is progressively loaded (and/or rendered).

Certain elements can even emit streams, which can be then captured and fed back into other media elements. A great example of this: you can stream data directly from <canvas> element to a <video> element on the same page (demo). Or stream from one video to another.

I’m not going to get too in-depth into this, as it is an advanced topic (and I have an entire separate blog post on it), but the general minimal steps required to load data into a (video) element with a dynamic MediaSource are:

Create a new mediaSource instance (new MediaSource())
Create an object URL that points to that source. Point the video to it
- Example: videoElem.src = URL.createObjectURL(mediaSource)
On the MediaSource instance, listen for the sourceopen even. Once it fires, you can begin preparing to load data into it
Before you start loading data, you need to create a buffer to hold it. You do so by creating a SourceBuffer attached to the source, with mediaSource.addSourceBuffer(mimeType)
- Example: const sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp9"');
Once you have the buffer, you can finally start loading in raw chunks of binary data
One way to load in a chunk is with ArrayBuffer(s), via the sourceBuffer.appendBuffer(ArrayBuffer) method
- If you have a blob instead of an ArrayBuffer, you can use myBlob.arrayBuffer() to get Promise<ArrayBuffer>
Eventually, you also need to deal with signaling that the end of data has been reached, and controlling the playback

You can find a full example of these steps put together on MDN’s page for SourceBuffer, and an even more advanced example in the MediaSource / MSE spec. ~~As I mentioned, I’m also planning to write and release a blog post on the topic soon.~~ I also just finished a blog post that goes into depth on this topic!

Disclaimers

I wrestled with the decision to include this section, as I don’t want to make any developers feel like I am trying to “shame” them, but I also feel that it is important to include it due to the level of abuse that Data URLs and Base64 often receive. This is an attempt to point out situations in which these alternative loading techniques are used, when maybe they shouldn’t be. Or ways in which they can used incorrectly. So, without further delay:

Base64 is not a form of encryption. It cannot be used to hide passwords, secrets, or prevent assets from being downloaded
Base64 is generally a less efficient way of storing binary data, and it is a non-zero amount of processing power required to convert Base64 strings into other formats
- This is (part of) why taking a 30 second video and storing it directly as a Base64 encoded string in the page and loading via Data URL, instead of just using a normal URL as the src, is very much a bad idea
Data URLs are not interchangeable with regular URLs, or Object URLs. There are caveats to how they work
- A huge drawback is this one: Data URLs are treated as an opaque origin as opposed to matching the origin of the page they are loaded into. (ref)
  - This means they do things like taint HTML <canvas> and affect CORS usage
- Another drawback is that there is actually limits to how long the Data URL string can be, and it is not that large, nor is it the same across browsers. In comparison, Objects URLs and blobs have enormous limits, that are governed more by the physical limitations of the computer being used.
Although MediaSource / MSE has been worked on since early 2013, it is a constantly evolving spec, and it can be difficult to find info on it. Not all browsers support it equally either.

More Resources

More About Me:

Firefox Backup Bookmarks Options and Automated Scripts

Joshua Tzucker — Tue, 17 Nov 2020 22:47:32 GMT

I’ve been a long-time Firefox user - for over ten years at this point. I’ve amassed thousands of bookmarks, some of which I have spent time organizing and tagging. As such, lately I was thinking I really should set up some sort of backup system, but I had trouble locating any commands or scripts that could accomplish this.

If you want to jump right to the solutions section, click here, otherwise read on.

Find Your Profile Folder

Pretty much all of your data in Firefox is stored in your profile folder. This is a special folder, whose location is unique to your computer due to its use of profile IDs and OS user directories.

This is the definitive documentation on how to find this folder location.

If you have the basic *nix utilities installed, you can use this one-liner to get the path on Windows:

tail "%APPDATA%\Mozilla\Firefox\profiles.ini" -n +2 | head -n 1 | sed -E 's/Default=//' | xargs -I % echo "%APPDATA%\Mozilla\Firefox\%" | sed -E 's/\//\\\/g'

# Example: "C:\Users\Joshua\AppData\Roaming\Mozilla\Firefox\Profiles\29adfe2f.default-release"

Research

First, let’s investigate how Firefox stores user bookmarks.

The best place to get this information is the Mozilla support page - “Where Firefox Stores User Data”. I’ll supplement this with some additional details below.

Some key things to note, which affect our task:

Bookmarks are contained in place.sqlite, along with download metadata and browsing history
- This makes extraction more difficult, as this is a special database, can you can’t easily copy & paste out of it, or view it, or easily import it into Firefox or other browsers
- This will be the most up-to-date stored version of your bookmarks, as the automated backup options (see below) either run periodically, or when the browser shuts down. In comparison, places.sqlite is updated in almost real-time as you edit bookmarks in Firefox.
There are some automated backup options, which essentially output a subset of places.sqlite, with just the bookmarks.
- /Profiles/{PROFILE_ID}/bookmarkbackups should contain daily backups of bookmarks
  - However, these are stored in a somewhat obscure .jsonlz4 file type, which is JSON compressed with a flavor of the LZ4 algorithm
- By changing the config value (in about:config) of browser.bookmarks.autoExportHTML to true, you can have Firefox automatically dump a backup of the bookmarks in HTML format, on browser close.
  - You can control the file location with config value browser.bookmarks.file
  - See this post for details

Converting the `.jsonlz4` Bookmarks File(s)

Note: You don’t need to convert the files if all you want them for is a backup that you can re-import into Firefox; Firefox understands this file format and will let you use it as an import, no conversion required.

The bookmarkbackups folder and its .jsonlz4 daily backup files are a nice existing backup to take advantage of, but how can we get the files in plain JSON, so we can do whatever we want with their contents (convert to CSV, cleanup, etc.).

Existing options (found via a quick Google search):

Everything listed in this thread
Online tool: Jefferson Scher: “BookBackReader”
Offline tools:
- C / Binary release: avih/dejsonlz4
- NodeJS / CLI: thrilleratplay/node-jsonlz4-decompress
- Python: TBlue Github Gist

Solutions

Manual Methods

For manual bookmark exporting, there are some nice built-in options. These are covered here, but to summarize:

Open the Bookmark Library (CTRL + SHIFT + B, or use Library Button in toolbar)
Open the Import and Backup menu
Use either of:
- Backup
  - this exports a UTF-8 JSON file, which is not compressed with lz4 - you can open and read this file easily
- Export Bookmarks to HTML
  - This exports a HTML file, which you can easily open with any standard web browser
  - The downside is that it is roughly 2x the size of the JSON export, and would be harder to parse if you wanted to automate the transformation of your bookmarks into another format

Also, if you don’t care about backing them up to a specific folder, than using the default automated backups (discussed above, here), might be enough for you.

Automated Methods

For putting together a command / script to grab a Bookmarks backup, I have three main options:

Hook into the internals of Firefox
- The best place would be the BookmarkJSONUtils.jsm utility - this has a native method for exporting to JSON - exportToFile
  - You can actually get to this in the browser, with resource://gre/modules/BookmarkJSONUtils.jsm
- Although this would probably be the most “to-spec” option, it would also take me a while to figure out how to do, especially given some limitations around scripting Firefox
  - For example, the headless mode of Firefox might let me eval() JS to extract the bookmarks, but you cannot concurrently use that and have an open instance of Firefox with the profile you want to use
Connect to the places.sqlite database and extract just the bookmarks, converting on-the-fly to JSON or HTML
- Complicated, and might have issues around concurrent db access
Just grab the daily backups (.jsonlz4 files) and copy to backup folder of choice
- This is, by far, the easiest option.
- I’ve implemented this below

Automated Methods - Batch Script

I’m not thrilled with it, but here is a batch file I put together to copy the most recent JSON file to my Dropbox. It requires that some standard *nix utilities are installed (these come default with git-bash), and unfortunately, you have to hard-code the backup folder(s). But, it works!

ff-bookmarks-backup.bat:

@REM # Hard-coded paths - EDIT ME
SET FF_DAILY_BACK_FOLDER="C:\Users\Joshua\AppData\Roaming\Mozilla\Firefox\Profiles\29adfe2f.default-release\bookmarkbackups"
SET OUTPUT_FOLDER="C:\Users\Joshua\Dropbox\Program Settings"

@REM # Remove quotes, as these are going to mess up paths later
SET FF_DAILY_BACK_FOLDER=%FF_DAILY_BACK_FOLDER:"=%
SET OUTPUT_FOLDER=%OUTPUT_FOLDER:"=%

@REM # Go to bookmarks folder
cd %FF_DAILY_BACK_FOLDER%

@REM # Workaround to capture variable that requires *nix utils
ls | tail -n 1 > backup_filename
for /f %%i in ('cat backup_filename') do set LATEST_BACKUP_FILEPATH=%%i
SET LATEST_BACKUP_FILEPATH=%FF_DAILY_BACK_FOLDER%\%LATEST_BACKUP_FILEPATH%
echo %LATEST_BACKUP_FILEPATH%

@REM # Copy file with rename
@REM # NOTE: This will overwrite, keeping a single file up to date with latest
cp "%LATEST_BACKUP_FILEPATH%" "%OUTPUT_FOLDER%\ff-bookmarks-latest.jsonlz4"

@REM # Cleanup
rm backup_filename

Automated Methods - A Better Solution

If I had a lot more free time, and wanted this badly enough, I would code this as a NodeJS script, and compile it to a portable binary using something like vercel/pkg. If this is something you are interested in doing so yourself, my advice would be to have the script:

Parse OS environmental variables and paths to find the Firefox user profile folder
Find the most recent file in the folder
Decompress it using thrilleratplay/node-jsonlz4-decompress
Copy the raw and de-decompress versions to the output directly, specified by the user through a CLI argument
1. Having the tool copy both versions to output could be made optional

Get and use local executables through NPM or Yarn

Joshua Tzucker — Tue, 10 Nov 2020 05:45:32 GMT

Here is a quick post on a common question; how do you get the path (and/or execute) a local executable (or cmd alias) that is installed locally via yarn or npm?

Getting the Binary Folder

Both Yarn and NPM have a command to get the local binary directory:

# NPM
npm bin

# Yarn
yarn bin

# Sample output: `C:/temp/my-project/node_modules/.bin

However, this command comes with a huge disclaimer; it returns the path where executables will be installed (if called from current working directory), not necessarily where they are already installed in a nearest subdirectory.

For example, with this folder structure:

project/
- README.md
- src/
  - node_modules/
  - package.json

… running npm bin in project root will echo project/node_modules/.bin, instead of the existing path of project/src/node_modules/.bin. Thus, it is always a good idea to cd to the directory with your project.json file before trying to grab the path, or prefixing.

In general, the binary folder should always be $PWD/node_modules/.bin, so you can also just hard-code the path in shell scripts and/or JS scripts.

Running a Local Binary

NPX

Many devs already know about this feature, so I’ll be brief on it; you can use npx {command} to target a package installed locally through package.json.

By default, npx will try to use a local install first, and if it can’t find it, it will download and execute a remote source. If you want to stop this, and have it just fail if the local copy does not exist, you can use the --no-install flag.

Variable Substitution

If for some reason you can’t use NPX, another option for shell scripting is to use variable substitution to capture the binary path and use it.

# Executing binary directly
$(npm bin)/local-binary
# Can use just like normal
$(npm bin)/local-binary --flag moreArgs

# Passing in package name with `yarn bin`
$(yarn bin my-package)
$(yarn bin my-package) --flag moreArgs

If you are on Windows, and you don’t have git-bash or anything like that installed, this should work in cmd:

for /f %%i in ('npm bin') do set NPM_BIN_PATH=%%i
%NPM_BIN_PATH%/my-package

With both bash and CMD, you can always just hard-code the path of the binary folder and use it that way too.

With NodeJS child_process

If you are using NodeJS’s child_process to execute binaries from JavaScript code (e.g. with .exec()), you might be wondering how to get it to see locally installed binaries from node_modules.

One option is to use a library, such as execa, which makes this easier.

The vanilla approach is to combine the system PATH with the local binary path, and pass that combined string as part of the env object in child_process method options:

/**
 * Example is using `supports-color` package, which is installed locally
 * We are going to extract the help text and log it
 */

// Assumes script lives in same directory as package.json
const LOCAL_BIN_PATH = require('path').normalize(`${__dirname}/node_modules/.bin`);

// Execute a local binary / command
const helpText = require('child_process').execSync(`supports-color --help`,{
	env: {
		...process.env,
		path: `${process.env.path};${LOCAL_BIN_PATH}`
	}
}).toString();
console.log(helpText);

Wrap-Up

I hope this helps someone out there! Do you know any other tricks for using local binaries? Feel free to share below in the comments!

Keep a Shell Open in NodeJS and Reuse for Multiple Commands

Joshua Tzucker — Sun, 08 Nov 2020 08:28:48 GMT

I’m making this post because this question has been bugging me, and I have had an unreasonably difficult time finding information on it. My question is simple:

Is there a way to spawn a shell in NodeJS, keep it open, and reuse it for multiple commands (perhaps even calling different binaries)?

Throughout this post, I’ll use this as the sample scenario:

Spawn a shell
Execute uname -a to get OS Info
Execute ls -a to list files in the current directory

Without Reuse

The standard approach is to not care about re-using a process or shell and letting NodeJS just spawn & kill them automatically. For example:

const childProcess = require('child_process');
const osInfo = childProcess.execSync('uname -a').toString();
const files = childProcess.execSync('ls -a').toString();
console.log({
	osInfo,
	files
});

If you wanted an async based approach, you could use something like:

const childProcess = require('child_process');

const osInfo  = await new Promise((res, rej) => {
	childProcess.exec('uname -a', (err, out) => res(out));
});
const files  = await new Promise((res, rej) => {
	childProcess.exec('ls -a', (err, out) => res(out));
});
console.log({
	osInfo,
	files
});

Spawning an Actual Shell Process

With most of the child_process commands you can always specify that a command should execute inside of a shell:

require('child_process').execSync('uname -a', {
	shell: true
});

In fact, you can even pass in whatever shell you want used, as a path:

// This will fail on Windows
require('child_process').execSync('uname -a', {
	shell: '/bin/sh'
});

However, the shell that is spawned by execSync, exec, etc., is sort of managed by the method, and not exposed directly. If your command is ls, even with shell: true, you are still piping commands to ls, not to the underlying shell. However… what if you spawn the shell as your command? That works!

const spawnedShell = require('child_process').spawn('/bin/sh');
// Capture stdout
spawnedShell.stdout.on('data', d => console.log(d.toString()));
// Write some input to it
spawnedShell.stdin.write('uname -a\n');
// We can add more commands, but make sure to terminate
spawnedShell.stdin.write('ls -a\n');
// End
spawnedShell.stdin.end();

This is pretty neat! However, it is not super portable (/bin/sh is certainly not going to work on Windows), and it is not really well-suited as a persistent shell that we can use in a nice async manner. Let’s refine it.

Reusing an Open Shell in NodeJS

The generic steps we can use to set up a long-lived shell to pipe data in and out of in NodeJS is:

Spawn the shell directly with child_process.spawn()
- Optimal solution would automatically use the correct shell based on OS
Add event listeners / hook into the subprocess streams
- At a minimum, we want to capture stdout, catch exit(s), and write to stdin
Allow results to be awaited

With this in mind, I put together a tiny wrapper around child_process.spawn, which launches the correct shell and exposes some utility methods. You can find it here, but keep in mind it is not optimized and comes with some caveats.

Performance

The main reason why I was looking into this in the first place was that I was wondering if there would be any performance benefits to keeping a shell open and reusing it, rather than launching an entirely new process for every command. I’m working on a project that tries to execute hundreds of commands, so any increase in speed is something I’m interested in.

Unfortunately, based on my simplistic test, reusing a shell actually is less performant than just allowing NodeJS to spawn an entirely new process for each command. Sample results (200 commands executed):

Method	Total Time (ms) (lower is better)	Ops per Sec
`spawn`	1892	105.7
`spawnWithShell`	2579	77.5
`exec`	2634	75.9
`persistentShellWithCapture` (**)	5008	39.9
`spawnSync`	5399	37.0
`persistentShell` (**)	5770	34.7
`execSync`	8982	22.3

** = persistentShell and persistentShellWithCapture are the wrappers I created around spawn, which spawn an instance of the default OS shell and pipe commands and data in and out.

This test was performed on a laptop running Windows 10 and Node v12.x

As you can see, child_process.spawn easily performed the best out of all the approaches. This makes a lot of sense, as pretty much all the other approaches build on-top of spawn, so by definition they are adding overhead. In fact, the idea that child_process methods are all extending from .spawn is called out in the very first paragraph of the official docs on child_process:

”[…] This capability is primarily provided by the child_process.spawn() function:”

At first, I was a little surprised at just how poorly my shell-reuse technique fared, but in hindsight, it makes sense. Even though the shell is reused, it still has to be launched (initially), and a communication layer maintained to pipe stdin/stdout. Furthermore, any command that is executed still requires the running of the process to support it; whether you run git log via a spawned shell or with Node’s spawn, both cases require the git binary to actually run, and in many cases, spawning and keeping open a shell wrapper around those commands is pretty much pure overhead.

Other Approaches? Libraries?

So, although you definitely can keep a shell open in NodeJS and pipe things in and out, in general it is not a great idea for both security and performance purposes.

So, back to the original reason why I was researching this; is there a different way to speed up slow command execution, especially in bulk? Well, I might have to get back to you, but I have a feeling that another avenue I need to explore more is native bindings.

For example, git exposes commands through its binary (which you can call through child_process or your CLI), but there are also packages that expose native bindings for Node, where Node talks directly to the low-level code of the program, rather than going through a CLI or spawning processes to execute the commands. In general, talking directly to lower-level (aka nearer to bare metal) code is going to be faster than passing messages through higher level interfaces.

A very interesting article I came across (of course, after I had already written most of this post), is by Github’s engineering team: “Integrating Git in Atom”. They switched from a native binding package (nodegit), to executing git commands via spawn (via child_process.spawn / execFile). They noted that there was a significant performance decrease caused by the overhead in process spawning per command, although they also noted some ways they helped mitigate the effect. This seems to confirm my point though, that native bindings are going to be more performant than spawning a new process for each command.

Links of Interest

These are links I came across while researching this topic, and that were helpful:

NodeJS source code
github.com/nodejs/help/issues/1183
Stack Overflow: Exec - Keep Shell Alive
Stack Overflow: Spawn vs Execute
FreeCodeCamp: NodeJS Child Processes - Everything You Need to Know

Adding Extra Attributes to Style and Script Tags in Wordpress

Joshua Tzucker — Sat, 24 Oct 2020 19:09:09 GMT

Intro

In WordPress, there are many ways to output a stylesheet <link> or JS <script> tag, but it might be unclear how to add arbitrary HTML attributes.

Through this guide, let’s pretend that our goal is to add the arbitrary attribute data-theme, and a value of light:

- <link rel="stylesheet" href="styles.css" />
+ <link rel="stylesheet" href="styles.css" data-theme="light" />

The easiest, but most unsafe way, is with the wp_head or wp_footer hooks. For example, if you wanted to simply output the tag in the <head> of the document, you could use something like this:

function add_my_tags() {
	?>
	<!-- We can output whatever we want here, including arbitrary attributes -->
	<link rel="stylesheet" href="styles.css" data-theme="light" />
	<?php
}
add_action('wp_head', 'add_my_tags');

However, this is not ideal, and goes against best practices; since you are not enqueuing the resources normally, there is no way for other plugins or theme files to know about it, and you might end up with duplicate scripts and styles getting loaded.

This can be especially problematic for JavaScript files, as they might be required to be loaded in a special order, or will error if loaded more than once.

Idea: Reusable and Shared Data

The safest way to add resource and scripts in WP is always through the global enqueue methods; wp_enqueue_style() and wp_enqueue_script(). However, these methods do not offer any way to add custom attributes to be echoed in the HTML.

What about the add_data() method, hmm? That looks pretty tempting…

Ah, no, that is very misleading. While add_data() does allow you add arbitrary properties to a enqueued object internally in WordPress (technically an instance of _WP_Dependency), only certain attributes are actually processed by WP and affect the echoed out HTML. For example, for style tags (<link>), WP will recognize:

rtl (link)
- suffix is combined with this (link)
conditional (link) (used to inject legacy conditional comments)
alt (link)
title (link)

This is also called out / mentioned in the actual exposed functions documentation:

However, this gives me an idea… what if we could use the power of add_data to store the attributes across the WP system, but then hook into the HTML generation filters to alter the output? We can; read-on for my solution.

My Solution

Although arbitrary properties added through add_data that WP does not know about do not affect output by default, that doesn’t mean we can’t use the underlying system to come up with our own solution, and make it work that way!

Here is what I came up with, which is flexible and reusable. All the code on this page can also be found in this Github Gist.

<?php
/**
 * @author Joshua Tzucker
 * @license MIT
 * @see https://joshuatz.com/posts/2020/adding-extra-attributes-to-style-and-script-tags-in-wordpress/
 */

/**
 * These should stay global, so you can access them wherever
 * you need to hack on an extra attribute
 * 
 * @example
 * ```php
 * global $ARB_ATTRIB_PREFIX;
 * wp_script_add_data($yourHandle,  $ARB_ATTRIB_PREFIX . 'your_key', 'your_val');
 * ```
 * 
 * You can customize this prefix, but be careful about avoiding collisions, and escape any characters that will break building the regex pattern
 */
$ARB_ATTRIB_PREFIX = 'arb_att_&#';
$ARB_ATTRIB_PATTERN = '/' . $ARB_ATTRIB_PREFIX . '(.+)/';

/**
 * Callback for WP to hit before echoing out an enqueued resource. This callback specifically checks for any key-value pairs that have been added through `add_data()` and are prefixed with a special value to indicate they should be injected into the final HTML
 * @param {string} $tag - Will be the full string of the tag (`<link>` or `<script>`)
 * @param {string} $handle - The handle that was specified for the resource when enqueuing it
 * @param {string} $src - the URI of the resource
 * @param {string|null} $media - if resources is style, should be the target media, else null
 * @param {boolean} $isStyle - If the resource is a stylesheet
 */
function scriptAndStyleTagAttributeAdder($tag, $handle, $src, $media, $isStyle){
    global $ARB_ATTRIB_PATTERN;
    $extraAttrs = array();
    $nodeName = '';

    // Get the WP_Dependency instance for this handle, and grab any extra fields
    if ($isStyle) {
        $nodeName = 'link';
        $extraAttrs = wp_styles()->registered[$handle]->extra;
        
    } else {
        $nodeName = 'script';
        $extraAttrs = wp_scripts()->registered[$handle]->extra;
    }

    // Check stored properties on WP resource instance against our pattern
    $attribsToAdd = array();
    foreach ($extraAttrs as $fullAttrKey => $attrVal) {
        $matches = array();
        preg_match($ARB_ATTRIB_PATTERN, $fullAttrKey, $matches);
        if (count($matches) > 1) {
            $attrKey = $matches[1];
            $attribsToAdd[$attrKey] = $attrVal;
        }
    }

    // Actually do the work of adding attributes to $tag
    if (count($attribsToAdd)) {
        $dom = new DOMDocument();
        @$dom->loadHTML($tag);
        /** @var {DOMElement[]} */
        $resourceTags = $dom->getElementsByTagName($nodeName);
        foreach ($resourceTags as $resourceTagNode) {
            foreach ($attribsToAdd as $attrKey => $attrVal) {
                $resourceTagNode->setAttribute($attrKey, $attrVal);
            }
        }
        $headStr = $dom->saveHTML($dom->getElementsByTagName('head')[0]);
        // Capture content between <head></head>. Kind of hackish, but should be faster than preg_match
        $content = substr($headStr, 7, (strlen($headStr) - 15));
        return $content;
    }

    return $tag;
}
add_filter('script_loader_tag',function($tag, $handle, $src){
    return scriptAndStyleTagAttributeAdder($tag, $handle, $src, null, false);
},10,4);
add_filter('style_loader_tag',function($tag, $handle, $src, $media){
    return scriptAndStyleTagAttributeAdder($tag, $handle, $src, $media, true);
},10,4);

And now you can easily add any arbitrary attributes, like so!:

<?php
// This is all it takes now to add an attribute pair that will get picked up by our filter and injected into the HTML tag
global $ARB_ATTRIB_PREFIX;

// Script
wp_script_add_data($handle, $ARB_ATTRIB_PREFIX . 'your_attrib_key', 'your_attrib_val');

// Stylesheet (identical syntax)
wp_style_add_data($handle, $ARB_ATTRIB_PREFIX . 'your_attrib_key', 'your_attrib_val');

Going back to our original example:

<?php
global $ARB_ATTRIB_PREFIX;
wp_enqueue_style('my-theme', get_stylesheet_uri());
wp_style_add_data('my-theme', $ARB_ATTRIB_PREFIX . 'data-theme', 'light');

And here is a more practical example; we can use this system to enforce Subresource Integrity (SRI) on a <script> tag, to make sure the contents does not change from the time we add it:

<?php
function addJqueryWithSRI(){
	global $ARB_ATTRIB_PREFIX;
	wp_enqueue_script('jquery-3', 'https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js');
	wp_script_add_data('jquery-3',  $ARB_ATTRIB_PREFIX . 'integrity', 'sha512-bLT0Qm9VnAYZDflyKcBaQ2gg0hSYNQrJ8RilYldYQ1FxQYoCLtUjuuRuZo+fjqhx/qtq/1itJ0C2ejDxltZVFg==');
	wp_script_add_data('jquery-3',  $ARB_ATTRIB_PREFIX . 'crossorigin', 'anonymous');
}
add_action('wp_enqueue_scripts','addJqueryWithSRI');

For those curious, this is not the first time I have written about hooking into the loader_tag filters - I had to use some similar code when figuring out how to add attributes like defer and async to tags, and wrote up my approach here. The research and coding I did for that was very helpful for this situation, and you should checkout that post if you are looking for more info on how WP generates the final HTML for stylesheets and scripts.

Why A Special Prefix?

You might be wondering why I’m using a unique string ($ARB_ATTRIB_PREFIX) to prefix any attribute key names we are adding, when later, I’m removing it via Regex. The answer is two-fold:

Collision avoidance with WP Core
- I want to be very sure that my function only adds attributes that were explicitly requested to be added; for example, the key conditional is already in use by Wordpress to allow for conditional comment inclusion (this is a legacy IE thing)
- Although I could filter out attribute keys that I know already belong to WP, that is not wise, since I would have to manually update it as WP Core changes
Collision avoidance with other plugins / themes
- I’m sure I’m not the only developer to ever think of hooking into add_data to store arbitrary parameters and then use them elsewhere. By forcing a unique prefix, I reduce the likelihood of a given key colliding with one added by a plugin or theme.

The sharing of $ARB_ATTRIB_PREFIX via global is not required; you could manually just copy and paste the unique prefix whenever you use add_data, but we all know how dangerous typos can be 😂

Room for Improvement and Alternatives

I’m sure there are a lot of ways that the above solution could be improved or altered; it was whipped up in an afternoon to answer a question posed to me. For example:

You could create a simple wrapper function around wp_style_add_data and wp_script_add_data, so you don’t have to always remember to grab and then use the global prefix
- I’ve included sample code for this in my gist.
You could bypass add_data and come up with your system for associating handles with metadata
Instead of using a unique prefix, you could use a global array (kind of like my $specialLoadHnds solution in this post)
There might be cleaner (and/or safer) ways to transform the $tag string and modify attributes; I went with PHP’s DOM Library (e.i. the DOMDocument class) based on this StackOverflow.
- Although RegEx as a solution for DOM manipulation is almost never a good idea, sometimes it is OK for situations like this one, and might be faster

The Best Solution

The best solution for this would be to have adding arbitrary attributes to script and style tags supported natively in WordPress methods to begin with!. This is not an unreasonable thing to ask of a CMS, nor is it a niche concern. What is frustrating is that is this has been proposed (multiple times), and there are unclosed tickets to add it (#22249, #33948) from 8 YEARS ago (2012)!

There have been entire new CMS frameworks, such as Gatsby, that have sprung to life and grown over just a fraction of that time period. Heck, the initial release of React was in 2013. I understand that the WordPress codebase is large and needs to cover a lot of legacy uses (old themes, plugins, etc.), but there is a point where this really should have seen more progress than it has, and that was several years ago IMHO.

The best scenario is that, any day now, WordPress core is updated with this feature released, and this whole article becomes obsolete (except for legacy WP installs) 😄

Make Ten Card Game with Svelte and TypeScript

Joshua Tzucker — Sun, 04 Oct 2020 09:02:32 GMT

How to Get It!

🃏 Online Game: make-ten-card-game.netlify.app

💾 Source Code: GitHub

What Is It?

I built this at the request of an elementary student tutor, who, due to the current pandemic, is running her sessions over Zoom. “Make Ten” is a simple card game, where you have to get rid of all the cards on the table by making sets that equal ten - for example, an 8 of spades, and a 2 of hearts. The cards are divided up into stacks, and once a stack is “cleared”, you can use its spot as a free place to move cards to in order to get them off the top of a stack.

Sorry, your browser doesn't support embedded videos.

This is meant for students that are learning the basics of addition, so that is about as complex as the game gets.

It also features typical game functionality like a built-in game timer, pause / resume / restart buttons, and fun animations. I built it as a SPA (single-page application), with Svelte and TypeScript.

Tools Used

Svelte (I learned it as I built this; it is my first Svelte project!)
TypeScript
JavaScript
HTML
CSS
IndexedDB (with localForage)
Netlify (automatic deployments from Github)

Development

This was my first Svelte project, so I was learning the framework as I built the application. Honestly, I had a great time learning Svelte and will very likely use it in future applications; it was a joy to use, and made building components feel very natural and easy. In fact, the entire game was built in about two days, despite having to learn Svelte as I worked!

The most complex parts of building the game were:

Managing the state of cards / stack
- I ended up using a multidimensional array, with identifiers for each card
- I had to be extra careful when moving the cards around; there are some particular nuances of how Svelte handles mutations
Animations
- There are some fancy animations in the game (cards being selected, moved between stacks, discarded)
- Svelte fortunately has a lot of built-in helpers to assist with complex animations; for example, crossfade and send / receive

Offline-First Database Options for Web Applications in 2020

Joshua Tzucker — Mon, 28 Sep 2020 04:12:08 GMT

Intro

For most people building web applications, the general idea is that the user is connected to your service a fair amount of the time; there might be periods of time where they go offline, but the goal is for a state of being “connected”. This assumption means that the most common methods for storing and retrieving user data for the UI are things like:

Standard AJAX requests
- Database agnostic; the client-side code does not even necessarily know what type of DB is on the other end
- Could be REST, GraphQL, etc.
Websocket session
- Similar to AJAX; database agnostic, requires active connection
Managed database services (such as Google Firestore and AWS DynamoDB)
- These typically support an “offline” mode, but the main functionality is for sync / online operations
- You don’t need to have a server

However, there is a gap left by all of the above approaches. What if we are developing something that is meant to be used offline, after the initial page load? Is there a way we can persist and query user data locally, without ever needing a server?

Apps that are designed to be fully functional without an internet connection are often categorized as “offline-first”. There is even a community around the topic.

Options

In 2020, the best overall option for local data persistence is going to be the IndexedDB browser API. The explanation for why this is the superior option (over localStorage, cookies, etc.) is long and technical, so I’ll defer to the experts (this is a good summary, from web.dev). However, using IndexedDB is a little tricky (it is a low-level API) and most developers (including myself) are going to prefer a wrapper library around it, which will make writing code to use it easier.

My research into the libraries supporting IndexedDB-backed local persistance and querying is the reason for this post. I was also interested in which libraries supported things like manual import / export shortcuts (so I don’t have to write a huge amount of code just so a user can download their own data), sync, NodeJS support, and legacy fallbacks.

Comparison Table

I’ve been using Google Sheets to record my findings - you can access it here (no login required). I’ve also copied and pasted the main table below:

Library	IndexedDB	localstorage (fallback)	NodeJS	Sync?	Native Export / Import (aka dump imports)?	Notes
Dexie.js	Yes	No, not really.	/ = You can polyfill	/ = adapter required	Yes (adapter)	Although legacy support might not be as nice as localForage, and sync not as nice as PouchDB, overall has great performance and is well-liked. Also in active production use with some major players.
PouchDB	Yes	Yes (adapter)	Yes (adapter)	Yes!	/ = maybe with some other libs	CouchDB interoperability and sync features are the main selling points, but is also a very well thought out product, that is well-liked and maintained. If you are not used to NoSQL / CouchDB, the syntax can be a little daunting.
RxDB	Yes	No	Yes	Yes	Yes	RxDB is built on top of PouchDB, and you can think of it as providing a reactive layer between your data and your UI, whereas normally you might need to manually code timed synchronization events. Since it is built on PouchDB, syntax is still NoSQL / documents. It comes with a large feature-set, beyond just the reactivity; schema migrations, ORM capabilities, and field encryption are all features not present in PouchDB, but come out-of-the-box with RxDB.
localForage	Yes	Yes	No	No	No	localForage has been around for a long time, and is stable and well-liked, but it also remains focused on a simple localStorage like API. For some, this might not be enough and you might want to evaluate some of the alternatives with a larger feature set.
nanoSQL	Yes	Yes	Yes	No	Yes	Seems like a very ambitious, but well thought out project, with good results. Extensibility and portability seem to be the key here, which is evident in the number of adapters, plugins, and customization options.
alaSQL	Yes	Yes	Yes	No	No	Looks really neat for large in-memory queries and offers some unique compatibilities (supports Excel `.xls` and `.xlsx`), so might be a good pick for BI applications. For lots of constant persistent syncing (e.g. IndexedDB read / writes), might not be the best pick since the IndexedDB connection is sort of manual.
LokiJS	/ = adapter required	Yes (default on Web)	Yes	No	It’s complicated / Depends	Focus is on in-memory manipulation.
lowdb	No	Yes	Yes	No	Kind of (since it is just JSON, you should be able to dump and import)	Focus is on fast, easy-to-use, and small footprint. Supports simple `.json` files as persisted database stores. Good for fast prototyping, but might be under-powered for larger projects.
WatermelonDB	/ = Using LokiJS Adapter	No	Yes	Yes.	No	Similar to RxDB, WatermelonDB is focused on data reactivity, tied to the React UI layer. Syntax uses a decent amount of ES6 decorators.
TypeORM	/ = Using sql.js + localForage adapters	Yes	Yes	Not really	No	ORM, not DB Layer
Orbit	Yes (std adapter)	Yes (std adapter)	Yes	Yes	No	This came from the Ember JS team and seems designed to solve a lot of complex problems. A lot of it is kind of over my head…
ydn-db	Yes	Yes	No	Yes	No	This does not seem in-use or maintained.
JsStore	Yes	No	No	No	No	Main focus seems to be on providing SQL syntax, but with IndexedDB backing.
turtleDB	Yes	No	No	Yes	No	Abandoned?
kinto.js	Yes	No	No	Yes	Yes	Looks like this was built by Mozilla employees and used for several large-scale Firefox products / features (see “Kinto at Mozilla”). kinto.js is also a part of a much larger eco-system - “Kinto-Storage”. For sync, it should be noted that this specifically requires a server running the Kinto server (which runs on Python + PostgreSQL)

Side Note - Managed Database SDKs

You might have noticed that some (very popular) managed / serverless databases were left out of the above table. Notably, Google Firestore and AWS DynamoDB. I’m always a little wary of recommending solutions like these, because they typically come with vendor lock-in, proprietary components, and cryptic pricing structures.

However… let’s be honest, many developers prefer these frameworks because they are well-supported, thoroughly documented, and easy to use. Some examples of these types of services and how they might (or might not) fit our needs:

AWS Amplify DataStore
- This might be the closest to what we are looking for, in a large-scale commercial offering, for several reasons:
  - Designed to align with the offline-first paradigm; local on-device database, that takes advantage of modern standards (e.g. IndexedDB on web), while also supporting syncing once online
  - Amplify DataStore appears to support a completely offline mode - no AWS account required
  - Supports sync (DataStore <-> AWS AppSync <-> Amazon DynamoDB)
  - Built-in conflict resolution
Google Cloud Firestore
- My opinion: This does not look like a good fit for “offline-first” applications, for several reasons:
  - Offline data with Firestore is meant to be used for caching or temporary network outages, but not as an offline-first local database
  - They openly admit that too much offline data slows down queries
  - It sounds like Google is not going to make efforts towards better offline support
Realm
- Realm is an entire platform for data usage in mobile apps. A lot of it seems pretty proprietary. They also do not (yet) have a web version.
- If you are looking for real-time sync with advanced conflict resolution strategies, this might be what you are looking for, but seems a little niche and outside of what I’m looking for

Side Note - NodeJS and Other Environments

Web is a very special environment, especially when it comes to databases. Part of the reason why things like IndexedDB are so important with web is because we don’t have access to the local filesystem with JavaScript; we can’t simply create a SQLite file and read/write to it. However, in NodeJS, as well as native environments like Android, iOS, and desktop applications, we are not subjected to the same restrictions.

However, several of the approaches and solutions above also work in non-web environments, such as NodeJS. For example, RxDB works with the web, as well as NodeJS, Electron, or even mobile (React-Native).

Wrap Up

What Should I Use?

You might have noticed that I did not declare a winner out of the options I came across in my research. That is because I don’t really think there is one clear best pick for all types of web apps. The general conclusion reached is:

No matter what, use something backed by the strongest standardized low-level API (that is IndexedDB for the forseeable future)
You should decide how much you care about the relationship between the UI and the data
- Do you want bi-directional data binding? Optimistic UI updates?
- If you care a lot about reactivity, you might want RxDB or WatermelonDB
Native export / import might not be worth that much as a feature, since you can always code a reusable method anyways, which can also handle versioning conflicts unique to your app

Adding an Emoji-Log Picker With Native VS Code Snippets

Joshua Tzucker — Tue, 22 Sep 2020 21:29:03 GMT

Intro

This is a quick guide on how to add an Emoji-Log picker macro to your VSCode setup, with a custom snippet that is just a few lines of code.

What is Emoji-Log?

If you haven’t heard of the Emoji-Log project / standard (by Ahmad Awais), you should go check it out. If you look at a lot of open-source repos, chances are you have actually come across its use in the wild at least once or twice. The TLDR is that it is a proposed standard / philosophy for writing the headline of commit messages, and using emoji prefixes to easily convey the type of change made. For example:

🐛 FIX: Broken redirect pattern

There is already a VSCode extension for Emoji-Log, but I wanted to share how I added an Emoji-Log picker / macro to my VSCode configuration, with just a few lines of code and the support of VSCode Snippets. It serves as a good example of how simple, yet capable, VSCode snippets can be.

VSCode Snippets

VSCode snippets are an incredibly powerful, yet underutilized part of VSCode. With enough manipulation, they can essentially act as text-expanders, macros, and formatters (and this isn’t my first promoting them).

Picking a predefined text choice and inserting it is an excellent use-case for snippets, and the impetus for this post.

Solution: Emoji-Log VSCode Snippet

Here is the snippet code:

{
	"Emoji Log Picker": {
		"prefix": "Emoji Log Picker",
		"description": "Pick an Emoji Log Emoji to use as a prefix",
		"body": "${1|📦 NEW,👌 IMPROVE,🐛 FIX,📖 DOC,🚀 RELEASE,🤖 TEST,‼️ BREAKING|}: ${2:IMPERATIVE_MESSAGE_GOES_HERE}"
	}
}

Click to see version that defaults to clipboard contents first

{
	"Emoji Log Picker": {
		"prefix": "Emoji Log Picker",
		"description": "Pick an Emoji Log Emoji to use as a prefix",
		"body": "${1|📦 NEW,👌 IMPROVE,🐛 FIX,📖 DOC,🚀 RELEASE,🤖 TEST,‼️ BREAKING|}: ${2:${CLIPBOARD:IMPERATIVE_MESSAGE_GOES_HERE}}"
	}
}

Adding the Snippet to VSCode

To add this to your VSCode environment:

Bring up the command palette (CTRL + SHIFT + P)
Start typing snippets, and then select Preferences: Configure User Snippets
If you already have a global snippets file, select it, otherwise create a new one with New Global Snippets file
In the snippets file that opens, paste the above snippet
You are done! Now, in almost anywhere in VSCode, you can press CTRL + SPACE to bring up snippets, and then start typing Emoji to bring up the log picker.

If you are looking for a more generic guide on VSCode snippets, this one looks good. And don’t forget the official docs!

Breaking Down How It Works

Since this snippet is pretty simple, it serves as a good example for learning how VSCode Snippets work.

First, let’s break down the main properties:

prefix: Serves as the main text string to be used for “triggering” the snippet; a better name for this might actually be keywords. This can be an array of strings, but a single string seems to work just was well (since VS uses sub-string matching)
description: Optional. Gets displayed by IntelliSense, in that little popup window as it shows snippets to pick from
body: The main thing that is inserted when the snippet is run
- 💡 TIP: The main way that snippets work is by inserting content, but through some tricky syntax, you can use them for formatting, replacement, etc.

💡 Another helpful optional field is scope. We could add that to our snippet with a value of "plaintext,markdown", if we only wanted our snippet to be suggested in those contexts. Without a scope field, the snippet will appear everywhere (unless the snippet file itself is scoped by language).

Next, let’s break down the body, the crucial part of the snippet. For the purposes of explanation, we can simply from this:

${1|📦 NEW,👌 IMPROVE,🐛 FIX,📖 DOC,🚀 RELEASE,🤖 TEST,‼️ BREAKING|}: ${2:IMPERATIVE_MESSAGE_GOES_HERE}

to this:

${ 1 | CHOICE_A,CHOICE_B | }: ${ 2: IMPERATIVE_MESSAGE_GOES_HERE }}
 |             |                 |                 |
 |-> Placeholder wrapper         |                 |
               |                 |                 |
               |-> Choice        |-> Tabstop       |-> Plain text

To break it down another way

${} are placeholder wrappers
The number prefixes ({$1}, ${2:}) inside the placeholders are tabstops.
- Unfortunately, they are mandatory with choice elements, which is why I have tab stop with the choice section, even though there should be nothing for the user to type in that spot
To offer choices, you need to use a special syntax with pipe wrappers:
- | CHOICE_A,CHOICE_B |
The : between placeholder wrappers is plain text that will be output
The benefits to using a placeholder wrapper in the second part instead of just using $2IMPERATIVE_MESSAGE_GOES_HERE are:
- When we tab to the second part, anything we type will replace the entire placeholder text
- The placeholder syntax is easier to read
- It makes it easier to upgrade our snippet later and expand with variables, nested placeholders, etc.

Adding the Clipboard

If we want the imperative message section to default to clipboard text, while still allowing the user to customize it, we can use the placeholder + variable syntax.

We simply replace this:

${2:IMPERATIVE_MESSAGE_GOES_HERE}

… with this:

${2:${CLIPBOARD:IMPERATIVE_MESSAGE_GOES_HERE}}

We now have a nested placeholder element, where it will fill with the value of the CLIPBOARD value, defaulting to IMPERATIVE_MESSAGE_GOES_HERE if the value does not exist. And since it is at a tabstop, it is easy for the user (aka you!) to customize the message.

Wrap Up

Even if you are not interested in using Emoji-Log, I hope you still learned something about VSCode snippets and maybe got some inspiration for your own workflow improvements!

Getting The Word Count of Multiple Markdown Files

Joshua Tzucker — Tue, 15 Sep 2020 16:57:45 GMT

Issue

I use Markdown on pretty much a daily basis. I enjoy its portability and wide acceptance, as well as its non-proprietary nature. However, certain literary features tend to be lackluster in the Markdown ecosystem, primarily because Markdown is most commonly used by developers, not writers (not to say that there is not overlap!). One of those features that it took me a little bit of time to suss out is how to calculate word counts for markdown files.

CLI / Shell Solutions 👩‍💻

Most of the below code snippets assume you are using a Unix-based OS, or an OS that has common Unix commands available (such as Windows, with Git-Bash installed, or coreutils).

We can build up a custom string of commands to get the results we want. Let’s break it down:

Getting word count (from plain text)
- If we want to accomplish this from the command line, it turns out there is a handy command we can use, aptly named wc (word count). We can use wc -w.
Getting word count (from markdown file)
- We first need to convert the markdown file into plain text, before passing to wc -w, to avoid inflating our count with MD syntax stuff
- We can use one of my favorite markdown processors, pandoc, for converting MD to plain text.
  - pandoc --strip-comments -t plain {markdown_file_path.md}
- Putting the above together with wc for word count:
  - pandoc --strip-comments -t plain {markdown_file_path.md} | wc -w
(BONUS) - What about getting a total word count across multiple files?
- For a bulk word count, we can use another command that will produce the list of filepaths, pass that into our previous command, and let wc sum it all up
- Example: ls | xargs pandoc --strip-comments -t plain | wc -w
- Example: find . -iname "*.md" | xargs pandoc --strip-comments -t plain | wc -w

Solution Summary

So, to summarize some quick options:

Word count for single file:
- pandoc --strip-comments -t plain {markdown_file_path.md}
Word count for multiple files:
- find . -iname "*.md" | xargs pandoc --strip-comments -t plain | wc -w

Accuracy 🎯

What makes something a “word”? With the above commands, anything separated by spaces is a word, including code, URLs, and other snippets that many might argue should not be included in the word count calculation.

The best way to address this, if you care to, is to use a Pandoc filter (either a program, or Lua script). I won’t go into details, but here are two resources that cover how to use this feature:

Torrecillas: “Meaningful Word Count for Markdown”
StackExchange / Superuser

Editor / GUI Solutions 🖱

Many Markdown editors support native word count reporting:

Nano (use ALT + D)
Typora
Mark Text
Notepad++

Oddly enough, VSCode does not natively offer a word-count feature, but there are extensions you can install (such as this one) which will provide that feature.

⚠ Warning: Many of the above editors use the same word-count rules as wc, which counts code and other non-prose content. See my notes under “Accuracy”

Selenium Webdriver - Disabling Chrome Logging Messages

Joshua Tzucker — Wed, 09 Sep 2020 06:50:16 GMT

This is a quick post, but took a little digging to come up with the solution featured.

The Issue

The issue, which you can find many users complaining about, is that when using Chrome as the browser with Selenium Webdriver and NodeJS, the default log preferences don’t seem to be respected and messages still appear in the console / stdout.

To start with, it seems at first as though it is impossible to get rid of this log message: “DevTools listening on ws://....” when first connecting.

To be clear, even if you explicitly tell Selenium that you want all logging off, you will still get this message. My first attempt tried to use this approach and failed:

const wdLogging = require('selenium-webdriver/lib/logging');
const loggingPrefs = new wdLogging.Preferences();
for (const logType in wdLogging.Type) {
	loggingPrefs.setLevel(logType, wdLogging.Level.OFF);
}

// I still got messages after creating a webdriver instance with these preferences

Solution

The solution involves the use of excludeSwitches - these are switches / flags that you want to exclude ChromeDriver from including by default. So, anything passed through excludeSwitches basically negates / removes a flag normally set as true by ChromeDriver.

Turns out, ChromeDriver has a default switch that enables logging - --enable-logging; you need to pass this to Selenium to turn it off. I found this via this S/O answer and the linked ChromeDriver Issue.

A concise example of adding the switch is this:

const { Options } = require('selenium-webdriver/chrome');
const chromeOptions = new Options();
chromeOptions.excludeSwitches('enable-logging');

But, for a full copy-and-paste example, here is working code:

// @ts-check
const path = require('path');
const { ServiceBuilder, Options: ChromeOptions } = require('selenium-webdriver/chrome');
const webdriver = require('selenium-webdriver');

async function main() {
	// I'm doing this because I hate adding things to my PATH 😅
	// Normally, webdriver will automatically grab the EXE from PATH if you have followed a generic setup guide
	const driverBinPath = path.normalize(`C:\\Users\\Joshua\\Downloads\\chromedriver_win32\\chromedriver.exe`);
	const serviceBuilder = new ServiceBuilder(driverBinPath);

	/**
	 * SOLUTION:
	 * By excluding the switch from defaults, this should disable the default logging
	 * @see https://bugs.chromium.org/p/chromedriver/issues/detail?id=2907#c3
	 */
	const chromeOptions = new ChromeOptions();
	chromeOptions.excludeSwitches('enable-logging');

	// Put everything together
	const driver = await new webdriver.Builder()
		.forBrowser('chrome')
		.setChromeService(serviceBuilder)
		.setChromeOptions(chromeOptions)
		.build();

	// Let's try something
	await driver.get(`https://example.com`);
	const title = await driver.getTitle();
	return title;
}

main().then(console.log);

You can also find the source code pasted into this Github Gist.

Other Languages

Since Selenium supports a lot of different programming languages, I’m not going to post the full code for each one, but the solution should still be the same, as long as Selenium is passing excludeSwitches through for that implementation.

For example, in Python, the main difference would be changing:

// NodeJS Code
const { Options } = require('selenium-webdriver/chrome');
const chromeOptions = new Options();
chromeOptions.excludeSwitches('enable-logging');

… to:

# Python Code
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])

Some More Background

I didn’t want to spend too much time digging into this, but my guess at why this behavior was introduced and past logging options no longer works is that the new flag probably overrides any granular settings, such as the old solution of --log-level-3.

It is also important to note that ChromeDriver is a completely separate project from Selenium Webdriver; part of why it took me a bit to find the solution is because I kept searching Selenium’s source code for text that was too specific to ChromeDriver - ChromeDriver itself actually lives in the Google chromium mono-repo.

There might be other solutions out there, but this one worked for me! Feel free to let me know if you find differently!

Fixing JSX types between Preact and React Libraries

Joshua Tzucker — Sun, 06 Sep 2020 13:25:26 GMT

The Issue

I have a small project that I’m starting, and I decided to try out Preact for the first time, using their TypeScript starter template. Everything installed (relatively) smoothly, and it seemed like everything was fine… until I tried to add a third-party component library. Then everything exploded.

To be specific, I tried to add Chakra UI, and immediately got this error (“cannot be used as a JSX component”) when trying to add any component:

'___' cannot be used as a JSX component.
  Its return type 'ReactElement<any, any> | null' is not a valid JSX element.
	Type 'ReactElement<any, any>' is not assignable to type 'Element'.ts(2786)

Trying to chase down this incompatible return type (“not a valid JSX element”) led me down a long trail, but the basics seemed to be that Preact components, such as FunctionalComponent (used by the template’s default App.tsx file) expect preact.VNode elements to make up children / returned elements / etc. This makes sense; preact.VNode is part of Preact’s VDOM implementation, which is distinctly different from React’s. It’s also worth noting where this comes from; in Preact’s JSX system, Element (aka JSX.Element) extends preact.VNode<any>.

What confused me about this error was two things:

Why is this happening? According to this page, preact-compat is included by default in the newer versions of Preact. And I’ve followed the one step listed in the docs - setting skipLibCheck to true.
How do I fix this? Is there a way in TypeScript to override namespaces and module definitions, to the extent that it could fix third party imported code?

The Solutions

The first issue I found (react-colorful/#39) was very close to mine, but not yet solved. However, after some more digging, I finally ended up on preact #2222, #2150, and probably the most relevant, PR #2329.

🚧 Once PR #2329 is merged and released, I believe that this issue will no longer exist, and you won’t have to do any manual patching to get third party React libraries to work.

To summarize, the main issue seems to be that, although Preact is “shipping” a lot of types needed for React compatibility, they aren’t necessarily getting automatically “pulled in” for TypeScript to understand that a third-party lib’s use of JSX Element should be aliased / pointed to Preact’s implementation.

Option A - Use Path Mapping

TypeScript’s config has an incredibly powerful option that allows import paths to be remapped to destinations of our choice; the paths option.

For our case, we can use this to remap the resolution of react and react-dom to the preact-compat layer, which exposes Preact’s JSX implementation, among other Preact types.

Here is the change to tsconfig.json (thanks to this comment):

{
	"compilerOptions": {
		// Needed in most cases
		"skipLibCheck": true,

		// You need a value set for baseUrl
		"baseUrl": ".",

		// This is the key - module resolution path remapping
		"paths": {
			"react": ["node_modules/preact/compat/"],
			"react-dom": ["node_modules/preact/compat/"]
		}
	}
}

⚠ Warning: If you are using a separate bundler / compiler (outside of TSC), you might also need to inform it of the path aliasing. For example, if you use Parcel.

If the types exported from preact/compat don’t cover it for you, and you need to provide more React type shims, another variant of the above solution is to create your own compatibility declaration file (react/index.d.ts), and use the paths option to map to it. This approach is used in the issue #2150 thread.

Option B - Manually Override Types

I couldn’t get this working 100% (at least not without turning on skipLibCheck), but I believe it might be possible to “monkey-patch” the definitions for the React module in-place, by using TypeScript’s module augmentation and declaration merging features.

This was my (bad) attempt, which was close to working, but violated a lot of TS rules:

import * as React from 'react';

declare module 'react' {
	namespace React {
		type FC<P = {}> = preact.FunctionComponent<P>;
		// ...
	}
	export = React;
	export as namespace React;
}

However, I have a feeling this is the wrong approach; namespace merging is a complicated topic with TypeScript, and it looks like the best solution for global module augmentation is to follow the pattern of:

Creating a folder with the module name
Creating an index file in that folder (/{moduleName}/index.d.ts)
Telling TSC about the file (if necessary)

Again, these steps are outlined in this comment on #2150, and partially in the TS Docs. If you can find better documentation / best practices, please let me know; this is a part of TS that I always feel I could use a better grasp of.

Getting Files In a Project Based on Gitignore Exclusions

Joshua Tzucker — Wed, 02 Sep 2020 16:32:03 GMT

Given that so many projects use git and its .gitignore file for specifying which files should be tracked as part of the repo’s source code, it is very tempting to reuse the .gitignore file as input into various devops tools and automated scripts. How can we do this? 🤔

With Git Itself

If we have access to git and its command line interface, there are several ways that we can natively get a list of files based on the gitignore patterns.

One-by-one: check-ignore
- You can check whether a specific file is excluded, by using the git-check-ignore command.
- Example: git check-ignore {myFilePath}
Bulk: ls-files
- You can use the git-ls-files command to query git for lots of info related to its file tracking
- Example: git ls-files --full-name to get full paths of files that are not ignored
- Limitation: Slightly complicated to capture everything you need:
  - Tracked files only: standard git ls-files works
  - Untracked only, but respect .gitignore: Need to use git ls-files --others --exclude-standard
  - BOTH tracked and untracked, and respect ignores: git ls-files --others --exclude-standard --cached
Bulk: ls-tree
- You can use the git-ls-tree command for tree object based file ls operations
- Example: git ls-tree --full-tree -r --name-only HEAD - get all file paths, from root dir (credit)
- Example: cd subdir && git ls-tree -r --name-only --full-name HEAD to get full filepaths from a subdirectory
  - Note that we drop --full-tree and add --full-name to make sure we get files only for the subdirectory, but with absolute paths
- Limitation: Requires a reference to a git object (e.g. HEAD, SHA, etc.). Therefore, fatally fails on a brand new repo (no commit history).

Best Git Command?

Based on the above research, the best command to get an exhaustive file list of included files, when you care about respecting .gitignore but don’t care about tracking status, is:

git ls-files --full-name --others --cached --exclude-standard
#                |--> Get full file name (abs path)
#                            |--> Include *untracked*
#                                     |--> Include *tracked*
#                                                   |---> Apply `.gitignore` rules

If you want the same as above, but want the filenames of those files specifically excluded (the inverse of above), add the --ignored flag.

With Libraries

Using the Git commands above have the benefit of being portable and requiring only access to git and a terminal. However, they also come with caveats, complexity, and the limitations that come with shell scripting (😬).

If you have access to NodeJS, there are a lot of libraries out there that will make retrieving file lists a lot less of a headache 😅

Modules / API only
- NPM: ignore
- Glob expanders that respect .gitignore or allow ignore patterns to be used
  - NPM: globby
CLI
- NPM: globby-cli (this is a wrapper around globby, above)
  - Example: npx globby-cli "images/**" --gitignore
  - Make sure you include the --gitignore flag, since the default for respecting it is false.
- NPM: globstar

NPM Package: static-build-cache CLI and Module

Joshua Tzucker — Fri, 28 Aug 2020 05:34:29 GMT

How to Get It!

📦 NPM Package: static-build-cache

💾 Source Code: GitHub

What Is It?

A tool for projects that have a build step that produces static website content; it uses Git to check for committed changes and avoids redundant rebuilds if nothing has actually changed. Speeds up build & serve type deployments.

For more details, make sure to checkout the repository and README.

How Does It Work?

The easiest way to use it is to add it as a dependency and then call it from the command line, or package.scripts, with the exposed command: static-build-cache. You can also import it and use it as a JS module.

It is designed to be able to run with a zero-config input, but has a bunch of optional parameters (such as the serve command to execute) which can be specified.

Tools Used

TypeScript
Commander (CLI parsing)
Git
Bash / Cmd scripting
Ava (tests)
NYC (coverage)
Github Actions (automated test runs)

How to Fix Android Messages Having No Notification Sound

Joshua Tzucker — Wed, 26 Aug 2020 04:22:15 GMT

This guide is specifically for Google’s “Messages” app (com.google.android.apps.messaging) - this is the default / stock messages app that comes with many new Android phones, including the newest Pixel 4A model. However, some of the advice about generic notifications settings is applicable to all apps and Android in general. Especially for apps that have their own notification controls.

The Issue

The problem is pretty straightforward.

After getting a brand new phone (Pixel 4a, with Android 10), incoming text messages / SMS no longer triggered an audible notification sound. Even after tweaking global notifications settings and setting ringer volume to maximum, no text messages could trigger an audio alert. There would be a visual notification shown, but no sound.

Searching across the internet, it is clear this is not an isolated incident. At first, I was going to blame myself, but when the same exact thing happened to some one else I knew, I realized there is a clear issue at hand.

The Fix

After tweaking every setting I could find related to notifications, I finally found the solution in a menu buried several layers deep. Turns out that Messages has its own notifications settings that override global settings, and for some reason, the notification sound was set to none! Setting it to any other sound option fixed it.

There are a few ways to get to this buried setting.

Open the “Messages” App
Tap the three dots in the upper right to open the menu, then select Settings
Tap the Notifications menu option
Tap the Incoming messages menu option
Make sure the setting on this page is set to “Alerting” and not “Silent”. Now look towards the bottom of the screen and tap the Advanced label to expand the advanced sub menu
In advanced sub menu, look for the Sound option. If it is set to None, then this is definitely your issue!
Tap the Sound option and select one of your ringtones / notification; this will be the sound that plays when you get a new message. Make sure to hit save!
You are done! Try to test it by having someone send you a text message. Or use a site like this one to test it.

Alternatively, you can get to the setting by starting a slightly different way:

Long press the “Messages” app icon in your app drawer or home screen, and tap the App Info option or icon
Tap the Notifications option
You can now start on step 4 of the above section.

If you want to see these steps in action, below is a screen recording I made:

You can also find the video above here.

If this fix did not solve your issue, feel free to explore the “Further troubleshooting” section below.

The Culprit / Theories

Since I observed this issue happening to two different users who were both switching to new phones (with Android 10), my guess is that it has to do with Google’s protocol for device migration and setting sync (related: backup overview). My theory is something like this:

User is on an older version of Android. They have their messaging app notifications set to either default notification sound, a system specific sound, or a custom sound
User gets a brand new Android phone and / or upgrades to Android 10+
The Android setup process tries to transfer the old setting, but the previous setting is not an option on the new phone. This could be because the ringtone file did not get transferred over, the previous user used a system level notification sound which was deprecated in Android 10, or some other reason.
Rather than default to a specific notification sound, the system defaults to null / None.

If I could talk to a Google engineer, I would point out that this feels like a UX process that could be improved. I would argue that the majority of users probably want some sound set as the default, even if they can’t keep their old setting.

Further troubleshooting

If the above fix did not solve your issue, here are some further things you can try (in combination with making sure the above has been tried):

Make sure there is a default notification sound set:
- Search: “default notification sound”
- Path: Settings -> Apps & Notifications -> Notifications -> Advanced -> Default notification sound
Make sure “Do Not Disturb” mode (aka DND) is OFF
- Search “do not disturb” OR “dnd”
- Path: Settings -> Sound -> Do Not Disturb
- You can also toggle DND through the Android pull down shade, and depending on your settings, you might also have DND configured to turn on automatically based on schedule and/or other triggers
- If it is ON, you should see a persistent icon in the status bar that looks something like this:
Make sure your Ring volume is not set to zero / vibrate only
- It used to be that “ringtone” volume was separate from “notifications” volume, but starting with Android 10 (?), these have been combined into one setting: “Ring volume”. If it is set to zero or vibrate only, then no notifications will emit sound (unless there is an app-specific override?)
  - If it is set to vibrate only, you should see this icon in your status bar:
  - If it is set to zero and vibrate is off, you should see this icon in your status bar:
- Warning: Using the physical rocker buttons on your phone to adjust volume no longer adjusts the ring volume (starting with Android Pie?) - it adjusts the Media volume. I agree with those dissenting that this is a silly change for Android to make.
Make sure that you don’t have individual per-contact / conversation notifications set to silent
- See below section

Per Contact Messages Notification Settings

Android actually lets you set the messaging notifications setting as granular as per-contact / conversation. So, for example, contact “Joshua” that texts you too many jokes that don’t need immediate attention can be set to “silent”, whereas “Boss” is set to “priority”, since you never want to miss a text from them.

Unfortunately, this is also a way that your notifications settings can get screwed up. There are a few ways to check for this, and remedy it.

If you want to check to see if any contacts have a customized messaging notification setting, navigate to Settings -> Apps & Notifications -> Conversations:

This screen shows you any text conversations / contacts that have a special non-default notification setting applied (or if you even opened up that panel in the past)
There seems to be a glitch with this settings page, at least on my device:
- All entries show as Default settings within the list, even if they are actually set to Silent or something else, which does show up if you click through to change the setting

To fix this, and restore contacts to the default setting, you can:

Use the above route ( Settings -> Apps & Notifications -> Conversations) and change each entry in the list to Default
Within individual messaging conversations within the Messages app, you can access the notification setting for that conversation by clicking the three dots in the upper right, then Details, and finally Notifications
Bulk: In my research, the only bulk approach that I came across was by clearing the data for the messaging app. According to multiple comments, this will reset all the per-conversation notification settings, but not delete any messages.
- I am slightly reluctant to recommend this, as it is something I have never tried. If you were to try this, I would strongly recommend backing up all your messages first (like in this guide).

Video Overview

This video does an excellent job of covering an exhaustive list of reasons why incoming messages might not be triggering audio alerts:

It is also uploaded here, and here.

Video PiP (Picture-in-Picture) Enabler Bookmarklet

Joshua Tzucker — Sat, 22 Aug 2020 04:24:30 GMT

Vid Popper Bookmarklet

This tool is pretty simple. It is a bookmarklet (bookmark that executes a tiny bit of JavaScript when clicked) that removes any HTML attributes or JavaScript properties that might try and prevent the use of PiP, and then automatically pops out the first playing video on the page, or if on Hulu, the main video player.

This enables PiP mode on Hulu, on both Firefox and Chrome.

This tool will override the disabled elements in both Firefox and Chrome, but will only auto-pop the videos in Chrome, since Firefox does not yet support requestPictureInPicture(). See this page for how to manually pop videos in Firefox.

If you are still unable to use PiP, make sure you have the newest version of your browser, that your browser supports PiP, and that it is not disabled behind a feature-flag. See details for Firefox, and details for Chrome.

Usage:

You can find the source code here.

Or, more simply, drag this bookmark to your toolbar and click it when on a site you want to use PiP on: Vid Popper

Background

I am a huge fan of browsers offering PiP (picture-in-picture) for video playback. I often watch videos while I tinker on the computer, picking shows that are light in plot and I can kind of tune in & out of while I do things, and PiP is perfect for this. It lets me shrink a video down as small as possible, so I can put the majority of my focus on the task at hand, but also keep the video on top, as I move windows around.

I recently noticed that for Hulu, despite PiP being available and working on Firefox, PiP options were curiously absent on Chrome. I figured Firefox probably just had more of the spec implemented and didn’t think much of it, until it started to bother me and I decided to poke around the webpage.

I quickly noticed that Hulu had actually added some overrides to their video elements to purposely disable PiP support - namely an HTML attribute (disablePictureInPicture) as well as JavaScript properties. The reason why it was working in Firefox but not Chrome, is that the overrides are part of the spec that is still “up in the air” so speak (more accurately, marked as “experimental”), and Chrome is currently honoring them, while Firefox is not.

Testing Git Outside Source - Thank You Temp Directories!

Joshua Tzucker — Thu, 20 Aug 2020 06:07:58 GMT

The Problem

I use Git on a daily basis, as my default version control system and integration with Github. I also am acquainted with writing tests, and how to integrate those with Git - e.g., which test files should be gitignore’d vs tracked. However, I just encountered a new wrinkle, which is how to deal with tests that rely on Git itself, or rather, the absence of Git. This pretty much only comes up when building tools for the developer toolchain / devops, but that is exactly what I’m doing at the moment.

I’ve actually previously created a Git based tool and test suites for it; however, those tests used the easy trick of nested git repositories. If you have project which is a git project, and you create project/git-test and then run git init inside git-test, you end up with a separate git repository inside your actual source code’s one, complete with separate tracking (there are issues however, if you forget to gitignore the child repository).

The complication is when you need to test your tool against a directory that is not part of a git-initialized repository. For example, if your test makes sure that your code throws the appropriate error when ran against a non-git project, where git log will fail with fatal: not a git repository, you need such a directory to exist to test against. Where do you create that test directory?

If you create it inside your source code, like src/__tests__/tmp, it will inherit the git tracking of src.
If you try to go up a level from your source code (e.g. src/../tmp) and create it there, you can’t be sure that won’t lead to a directory collision, or that your code hasn’t ended up nested in another git repository.
Creating it at the highest level (e.g. /) seems like a bad idea, and one might argue it doesn’t respect the User’s filesystem

The Solution: OS Designated Temporary Paths

I found the solution to this issue in two places, at roughly the same time. The first place was in checking out how an existing tool, gitchangelog, creates temporary directories with Python, and the second place was in the NodeJS fs docs, when I was trying to look up some methods.

If you are more seasoned with {{devops, IT sysadmin, etc.}} than I am, you might have already identified the solution long ago; OS temporary directories.

Essentially, over time a standard practice has emerged for operating systems to expose a temporary directory, which is periodically cleaned, and guaranteed to exist on boot.

Standard Library Wrappers and Implementations

Using temporary directories is such a common practice that many “standard libraries” have methods built-in to get the OS temp dir path, as well as create temporary folders directly.

Here are some examples:

NodeJS
- os.tmpdir() to get system temp dir (docs, implementation)
  - Also see sindresorhus/temp-dir, for version that always returns real path, even if system returns symlink
- fs.mkdtemp() to make temporary directory (docs)
Python - tempfile (docs)
- tempfile.gettempdir() to get system temp dir (docs, implementation)
  - Wrapper around _get_default_tempdir() to get system temp dir (private, implementation)
  - OS default strings are from _candidate_tempdir_list() (implementation)
- tempfile.mkdtemp() to make temporary directory (docs, implementation)
QT5
- QDir::tempPath() to get system temp dir (docs)
- QTemporaryDir to make temporary directory (docs)

Note that using temporary directories goes beyond just finding the right directory; the sub-directories that you create also need to have unique names that won’t collide - many of the methods above automatically do this for you.

Usage Considerations

Using temporary directories comes with some important caveats, the most important being that your data is unlikely to be persisted for very long. Some OSes will periodically clear temporary storage, or do so on events like reboots, or users might even choose to manually clear their own temporary folder when their HDD is nearing capacity.

For my use-case, creating a non-git-initialized directory with a few files for testing, this is perfect. In fact, best practice would be to clear / remove this test directory after each run, so the fact that it might not be persisted past a reboot is a moot concern anyways.

How to Fix Create-React-App Projects Stalling on Glitch

Joshua Tzucker — Sat, 15 Aug 2020 19:09:40 GMT

I recently ran into a perplexing issue with some of my React based projects on Glitch. They would stall in the startup stage (package.json -> scripts -> start), and eventually the live view would simply display an error in the form of:

failed to start application on PROJECT_NAME.glitch.me

This is most likely because your project has a code error.
Check your project logs, fix the error and try again.

Normally, I would agree with the error message that in all probability, the issue is probably with my code. However, this was happening with old repos, that had worked before, and the final straw was that it happened 100% of the time with a brand new blank React project scaffolded with Create-React-App.

Click here to jump right to the solution. Read on for full details.

The Issue

A quick search to see if anyone else was experiencing this issue brought me to two posts on the Glitch forums:

Both of those threads point to the post “create-react-app and express together on Glitch” as being the best answer to this issue.

Although the post mentioned above does indeed contain a solution, complete with working Glitch project that you can clone / remix, I personally find the solution reached to be rather complicated and overall something that founds like a substantial workaround. Not to mention, it starts to become pretty far removed from the default CRA starter…

However, after some digging, I actually found a far simpler solution. It looks like the crux of the issue might have less to do with conflicting ports, and more to do with CRA’s (and React-Scripts) reliance on having an interactive shell available (tty + open stdin).

There are tons of posts that indicate this could be the issue, including the following:

CRA GH: Issue #8688 (“Fails to start in Docker”)
S/O: React App exiting in docker container with exit code 0

I can’t find Glitch’s Docker config to see if they have TTY turned off, but it certainly would line up with the results I’m seeing. And the fact that this appears to be a newer change to CRA, which would explain why repos using older versions of react-scripts still work.

The Fix

Thankfully, someone on the thread for issue #8688 identified an easier solution than rolling back CRA scripts or changing the Docker config (which would be impossible on Glitch). In this comment, they mention using CI=true to set the environmental variable CI to be true.

To clarify, CI stands for “continuous integration”, and in the context of React, this is important for automated testing. Setting CI=true is meant to be used when running tests, for a proper setup. However, it also makes sense that it would work for the tty issue, since setting CI=true likely also forces React into a non-interactive mode that does not require an interactive shell, since many CI pipelines also lack an interactive shell for executing automated tests (reasonable).

The Fix - Quick Package.json Edit

If you need to set CI=true for Glitch, that could be as easy as changing your start entry from react-scripts start to CI=true react-scripts start. However, that is going to disable the interactive mode of CRA everwhere, not just on Glitch!

If you want to only apply the fix only on Glitch, there is another trick up my sleeve for you!

You can take advantage of the fact that Glitch sets certain environmental variables within it’s own system. For example, this will work right now to only apply the CI fix on Glitch:

{
	"scripts": {
		"start": "echo $PROJECT_REMIX_CHAIN | grep -E \".+\" && CI=true react-scripts start || react-scripts start"
	}
}

An even easier fix is to use a tool I built that takes care of checking the environment for you - “detect-is-on-glitch”. With that tool, the entire command above could be shortened to:

{
	"scripts": {
		"start": "isglitch && CI=true react-scripts start || react-scripts start"
	}
}

isglitch is another way to call detect-is-on-glitch

Plus, my tool scans multiple environmental values, works across multiple OSes, and I’ll update it if Glitch makes changes that stop certain variables from being exposed.

WARNING: the CI=true syntax will fail on Windows - see section below for details.

Cross-Env Considerations

WARNING: The CI=true is the Unix / bash syntax for setting environmental variables.

If you want a way to set environmental values across all operating systems, you will need to add cross-env as a dependency, and then change CI=true to cross-env CI=true.

In addition, for reading environmental variables in a cross-env nature, you can use a quick hack of node -p "process.env.VARIABLE_TO_CHECK". If you want to ensure you receive either the value itself or an empty string (but not undefined), use node -p "process.env.VARIABLE_TO_CHECK || ''".

Using glitch.json

If you are reluctant to edit your package.json to accommodate Glitch, an alternative solution can be to use glitch.json to tell Glitch to use the special start command. However, this takes some extra configuring; you would have to find the global path of NPM (or pnpm for Glitch, likely), setup a shell script to call react-scripts with CI=true, and then point glitch.json at the shell script.

If you really want to go down that route, make sure you review my Glitch guide, and in particular, the Troubleshooting section.

Demo

I put together a quick demo / starter repo, generated from Create-React-App, and configured with my fix for Glitch. Source code is on GitHub, and the live Glitch is here.

Conflicting Ports

Even if the above does not work for you, and you are convinced it is an issue with conflicting ports, you don’t have to roll your own proxy code like they did in that linked post. There is actually a built-in feature in react-scripts that will take care of it for you; all you have to do is declare the proxy destination in package.json -> proxy field. This will proxy all unknown (e.g. non-static files and routes) to the defined proxy.

For example, to proxy all non-static requests to port 3001, use:

{
	"proxy": "http://localhost:3001"
}

You can find the docs on this feature here, and Dave Ceddia has an excellent writeup on using it with CRA + Express.

NOTE: You can also use environmental variables to change the port dev server listens on (with PORT=#) and the host it is bound to (with HOST=___). Refer to the advanced configuration docs for details.

NPM Package: detect-is-on-glitch CLI, module, and script

Joshua Tzucker — Fri, 14 Aug 2020 05:59:45 GMT

How to Get It!

📦 NPM Package: detect-is-on-glitch

💾 Source Code: GitHub

The package is lightweight; works fast even via npx detect-is-on-glitch.

Intro

What is Glitch?

For those not aware, Glitch is an online code editor, with a focus on collaboration, sharing, and live editing. You can edit and run static websites, as well as full-stack projects that depend on build tools, NodeJS scripts, and other bits and pieces.

What is this tool?

detect-is-on-glitch is exactly what it sounds like; a tool to detect whether or not your code is running on Glitch, as opposed to your local PC, build server, or anywhere else.

The idea for this is that you might want your app to behave differently on Glitch versus anywhere else. Maybe you want to serve a slimmed down version of your app for Glitch’s live hosting, or display special console instructions to those remixing your project versus regular users. This tool can let you do that.

How Does It Work

This will probably the least complex NodeJS package I ever create. You can check out the source code yourself, but to sum up; it checks process.env and the hosted domain for any traces that Glitch normally leaves.

Right now, this works for all the samples I have tried, but might need to be updated in the future if Glitch introduces new changes.

You can call it via CLI (alias is isglitch), NodeJS, or even front-end code.

Tools Used

NodeJS
JavaScript
TypeScript
Glitch
🧠—> Research, to find what Glitch exposed as unique properties

Sample

If you are still trying to think of how this could be used, here is a quick sample: running a different “start” command on Glitch versus anywhere else

When Glitch detects a code change, or your project has “gone to sleep” due to inactivity, it will run your start script. You can read more about Glitch’s deterministic task setup in my Glitch cheatsheet.

Using this tool, you can have a different command execute (node server-light.js) for Glitch

package.json:

{
	"scripts": {
		"start": "npx detect-is-on-glitch && node server-light.js || node sever.js"
	}
}

How to Delete Files Based on a Pattern with SHX and ShellJS

Joshua Tzucker — Thu, 23 Jul 2020 23:13:37 GMT

Intro to SHX

One of my personal goals with NPM package development is to get better at cross-OS support; I don’t want to distribute code that only works in one specific environment. As part of my effort in working towards this, I discovered the shx CLI package, which is really mostly a wrapper around shelljs. Both of these tools let you write commands that use standard unix built-ins, like cat, but will work across any OS (Win / Mac / Linux).

For example, shx cat index.js

Using SHX to Delete Files Based on a Pattern

Problem:

If you are used to writing your commands without concern for portability, you might write a command to delete files based on a pattern like this:

# Delete all .md files in /dist
find dist -name '*.md' -delete

However, this command will fail with shx:

shx find dist -name '*.md' -delete

# ERROR: find: no such file or directory: -name

This is because the arguments to find are not currently supported by shx - you can find this tracked as issue #177.

Solution:

Since shx supports automatic glob expansion to standard commands, there is no reason why you can’t write the delete command as this:

shx rm 'dist/**/*.md'

Best Solution

There is a secondary problem here, which exists with many commands, which is that the command might exit with a non-zero code, preventing chained commands (ala &&) from running. This is especially problematic for NPM packages and the scripts section.

For example, if our package.json has this:

{
	"scripts": {
		"clean": "shx rm 'dist/**/*.md' && yarn build"
	}
}

… and there are no .md files inside dist, then the first part of the clean command will exit with:

error Command failed with exit code 1

… and yarn build will not execute.

There is any easy workaround though! Just use the & instead of the && operator, which lets the second command execute regardless of the success of the first.

You can also force the command to return exit 0, by piping and then returning true:

{
	"scripts": {
		"clean": "shx rm 'dist/**/*.md' || shx true && yarn build"
	}
}

Tip: You can also use shx --silent to suppress the displaying of the error message.

Using Google Forms for Quick Website Feedback and Bug Reports

Joshua Tzucker — Tue, 30 Jun 2020 08:19:50 GMT

Intro

When building a new website or application (SPA, PWA, etc.), one of the things you think about before launching is a way for users to leave feedback and/or report issues. For large-scale releases and long-term setups, you really should use a professional feedback and bug reporting system; something like Usersnap, or a custom solution built in-house.

However, if you are building a demo, MVP, or personal project, a great temporary, or even long-term solution, can be using Google Forms with Google Sheets.

Google Forms for User Feedback and Bug Reports

The simple overview of how Google Forms works is that:

You use the Google Forms builder to visually build out a form to your liking. It is super easy to use; drag-and-drop fields, easily add validation logic, etc.
Once you have the form to your liking, you can pick how you want to add it to your site:
- Have it open in a new tab: Click Send, then the link icon, then copy the URL. Now you can paste that URL into the href of a standard link element, router system, etc.
- Embed it directly in your site: Click Send, then the embed icon, then copy the iframe embed code. Now you can paste that code into a static HTML file, React component, etc.
Once someone submits the form, their response will be saved with the form, and visible to the form creator inside Google Drive.
- Optionally, you can also connect the form to a Google Sheet file, so each response automatically creates a new row in a spreadsheet
- You can also set up email alerts, and advanced automations (see below sections)

Simple Integration

As covered above, the most simple approach to integration is either dropping a link to the form in the code of your site/app, or embedding directly via the iframe option. Both work with almost any website or app, including most WYSIWYG builders like Wix or Squarespace.

Advanced Integration

One of the lesser known features of Google Forms, and what makes it an actual suitable solution for small use-cases, is that it supports pre-filling / auto-filling form fields with URL query string parameters. This means that you can do things like:

Autofill a username field based on logged-in info
Autofill a where did this happen? field with the current webpage URL
Map existing form elements on your page to the Google Form and pre-fill the values
Etc.

I recommend following the tips from these three pages to get started on using URL pre-filling:

If you want to see a live example, I use the prefill trick to auto-fill the URL field in my feedback form on cheatsheets.joshuatz.com. For example, go to this page, hit the question mark icon in the upper right, then the Feedback button, and then notice how the form, which opens in a new tab, has the URL field pre-filled with the URL of the page that you were just on! The feedback button and its corresponding URL is actually generated and rendered in React.

Limitations

There are a lot of limitations with Google Forms, which make it unsuitable for many professional and/or large-scale setups. Google Forms was not built for bug reporting or large-scale user feedback collection, so most of these limitations are understandable given the intended user-base. Here is a sampling of some of the major limitations:

Lack of native hidden fields
- This is a big limitation, because this is often how user feedback forms collect automated information (such as user-agent string) without cluttering the form for the user or allowing for accidental field editing
Lack of official integrations
- There are lots of ways that you can jury-rig connections between Google Forms and other systems, often by using something like Zapier, but these require manual setup and maintenance, and are liable to break if Google changes things
- In comparison, many other user feedback and bug reporting SaaS platforms have native integrations with other systems like Jira or PagerDuty
Lack of plug-n-play visual integration
- The two integration options, a standard web link or iframe embed, do not always visually blend well with an existing site. It might require a bunch of custom coding to incorporate nicely.
- In comparison, many feedback SaaS products offer copy-and-paste embed codes that seamlessly blend with your site or app, including advanced UI widgets like floating invite buttons and scroll animations

Google Forms Automation

One perk of using Google Forms is that it is integrated nicely into the whole Google Apps Script (GAS) ecosystem. In addition to having its own service layer, if you are piping responses from Forms into Sheets, you could also use the Google Sheet scripting service to build your own integrations or automations.

Google Apps Script is a highly extensible and powerful scripting platform that is tightly integrated with many of Google’s services. Connecting Forms to GAS unlocks an almost endless number of customized possibilities.

Twitter Iframe Embed Generator

Joshua Tzucker — Tue, 16 Jun 2020 02:04:25 GMT

Intro

Occasionally I have wanted to embed a specific Tweet from Twitter into a post on my website, and have been irked by the fact that the default embed method requires me to add a third-party Javascript embed to my site, which could potentially be used in malicious ways. I almost always prefer to use IFrames to embed third party content, as they provide a superior level of “sandboxing” and content isolation.

I realized that building an Iframe embed generator for Tweets should be fairly easy, and this tool is the result of that goal. It allows you to plug in the URL of a tweet, and get an IFrame embed code, with configurable options.

The Tool

The tool is fairly simple: Plug in the URL of a tweet, hit the “generate” button, and get back IFrame embed code that you can paste into your own website, or forums that accept IFrame embeds. Unlike other approaches (e.g. Twitframe), my tool simply wraps the Twitter embed code in an IFrame wrapper, using some IFrame generation tricks.

I built it using vanilla JS - no frameworks. I’ve used React, Vue, and other frameworks on other projects, but wanted to get some practice coding things from scratch on this, especially since I wanted a fast build.

Learnings / Tools Used

JSONP still is useful!
- It has been a while since I have had to use the JSONP workaround method, but it proved invaluable on this project, since Twitter does not provide the right headers for CORS, but does support JSONP responses.
Twitter’s OEmbed API
My previous research on different iframe generation methods.
Materialize CSS
Fairly vanilla JavaScript
SRI

Why Not Use SRI?

S**ubr**esource I**ntegrity (SRI) (MDN Docs) is a security feature baked into several modern browsers, that provides a way to verify (and enforce) that the external resource received from a remote source exactly matches what was requested.

You can use it by including a “hash digest” when you add <script> and other resource tags to your website, as an integrity attribute. In practical terms, this means if you add a script tag that loads fake-external-website.com/script.js, but you hash and add the hash as an integrity key when adding it, you can be guaranteed that you will always get the exact same script.js each time. This is an important security feature, because you can audit scripts as you add them, and if four months after adding them, the vendor suddenly adds a keylogger, the script will get blocked because the hash will no longer match the original hash.

This all sounds great, so why not use it with Twitter? Well, you can! But there are some downsides, which I’ll cover first:

Every time Twitter changes their code (in https://platform.twitter.com/widgets.js), it will break all embeds that rely on it
- Since the hashes have to exactly match, even them adding a single semicolon will cause the script to start getting blocked
- Each time they make an update, you would have to audit the change, then compute a new hash digest and update your embed code
SRI is not the same as sandboxing
- With SRI, the only guarantee is that the external code has not changed; not that it is safe to use. Auditing the code is up to you, as a developer, and if you miss something, there could be catastrophic results (embedded keyloggers, PII stealers, etc.)
- In comparison, IFrames provide a high level of sandboxing by default
SRI does not provide isolated styles
- Another benefit of Iframes are that they prevent conflicting CSS rules, by isolating the content inside the iframe from the content outside
SRI is not supported by all browsers
- Adoption is growing though, and already has about ~95% global support

If you don’t mind all of those downsides, here is how you would go about enforcing SRI on a Twitter embed.

Compute the hash digest of the Twitter embed JS file, with a given hash function. Once you have it, prefix the hash with the function type and a hyphen (e.g. sha256-)
- At the time of writing this, for Twitter’s widgets.js, the string would be: sha384-SR82MxVe9kFn41L4YvfLllMtR3HJ/jl5ubYdBBHjbTUBG31k2Zo4ZTuZXFxeP3Gd
- You can use online tools to get the hash, like srihash.org, or the OpenSSL CLI, with cat FILENAME.js | openssl dgst -sha384 -binary | openssl base64 -A
In the Twitter embed snippet, find the widget.js <script>, and add the integrity attribute with the string you generated above
- Example: <script async src="https://platform.twitter.com/widgets.js" integrity="sha384-SR82MxVe9kFn41L4YvfLllMtR3HJ/jl5ubYdBBHjbTUBG31k2Zo4ZTuZXFxeP3Gd" charset="utf-8"></script>
Because Twitter does not return the right CORS headers, and SRI checking requires a value for the crossorigin attribute, you also need to add crossorigin="anonymous" as an attribute key-pair.
- Update example: <script async src="https://platform.twitter.com/widgets.js" integrity="sha384-SR82MxVe9kFn41L4YvfLllMtR3HJ/jl5ubYdBBHjbTUBG31k2Zo4ZTuZXFxeP3Gd" crossorigin="anonymous" charset="utf-8"></script>

Why Not Twitframe?

Twitframe is a cool project that has the same end goal as mine; replace Twitter’s third party JS embeds with IFrame wrappers. However, Twitframe uses their own servers to proxy the Twitter request; this means that, at the end of the day, using their service still requires a level of trust of a third party. Since the content is now being served / proxied through Twitframe, you now have to trust that they will simply mirror the content from Twitter, and not alter it in any way.

Since the IFrame protects you from JS exploits, the worst case scenario using Twitframe is unwanted visual content on your site. For example, they could get hacked and someone could force it to response to every embed request with obscene content or clickbait malvertising.

My approach does not alter Twitter’s response or embed code in any way; it is truly just a “dumb” wrapper / sandbox around it. The actual network request for tweet content goes directly from your visitor’s computer to Twitter, and is not proxied through any third-party servers.

Downsides / Caveats

Limited IE support
- IE only supports Data URIs (the default IFrame method my tool uses) for images and resources, not HTML / frames (see caniuse for details).
- I could code a workaround for this, but it would likely lose cross-origin protections (example).
No Responsive Design:
- The biggest drawback from this method is that the IFrames are not responsive (by default). Meaning, you must specify the height beforehand, and if the tweet ends up being larger (longer) than the height you set it at, there will either be a scrollbar in the embed to see the rest of the content, or, if you turned off overflow, it will simply be hidden.
- I could design a way to make these embeds responsive (likely using postMessage()), but it would take a bunch of extra JavaScript and come with further caveats
Blocking by user or extension
- Potentially, a user could have their browser set to block IFrames by default. However, this is extremely rare, and even ad-blockers don’t usually interfere with IFrames

Elliptical Street View Controller - Arduino USB Adapter

Joshua Tzucker — Wed, 10 Jun 2020 00:48:19 GMT

This is actually a rather old project of mine that I meant to post about a long time ago, but didn’t get around to it until now. I believe I originally built this around October 2013, but my original notes on this build are pretty sparse.

Idea

The idea for this was pretty simple; I wanted my elliptical machine to control Google StreetView running on a computer, to where walking forward on the machine would translate into moving forward in the street view webpage. I also wanted to be able to easily adjust the POV of the Street View with a hand-held controller, so I could change direction without needing to step off the machine and go to the computer.

Even with the little bit of knowledge I had of electronics and programming at the time (I hadn’t really started learning to code much at this point in my life yet), I knew this should be very easy to accomplish with a microcontroller, a sensing circuit, and a little bit of code.

Execution

Demo Video

You can also find the video here.

Hardware

This project is comprised of just a handful of components:

Arduino Leonardo (Microprocessor) (brain and keyboard emulator)
Automobile Power Mirror Control Switch (got in parts auction)
Resistors, capacitors, etc.
Shift Register (drives LED counter display)
7 Segment LED Display (counter)
Red and green LEDs
3.5mm mono jacks (to connect to the elliptical)
Radioshack enclosure
Push buttons (for adjusting step-to-keyboard press ratio)

My favorite part of the build was integrating the recycled automotive power mirror switch. Even though I built this 7 years ago or so, I can still remember sitting down and reverse engineering the pin mappings on it, since I couldn’t find a wiring diagram and really wanted to integrate it. There is something really fun about reusing scrap components in a way that is far outside their original intended use, especially as inputs (I feel like Foone would be proud of me).

Here is my assembled adapter box, connected to the original elliptical display unit via 3.5mm cables, and to the computer via USB:

Software

The software side of things is fairly simple. A tiny routine, written in C++, runs on the Arduino. It listens for button presses, as well as polls an analog sensor tied to the elliptical output in order to sense steps.

The way that my specific elliptical (an old “Guthy-Renker Powertrain”) talks to its LCD readout / console display is with a single wire, and a variable resistance. During a few tiny parts of each step, the resistance of the wire falls to almost zero (closed circuit) and allows current to flow, and during the rest of the step, the resistance approaches infinity (open circuit) allowing no current to flow. I use this by reading the change in voltage (as a proxy for resistance, V=IR) with the Arduino’s analog input pin. A certain number of voltage changes equals a certain number of steps, which I translate into a keyboard press to move along the Street View image.

Since the Arduino Leonardo has built-in support for HID emulation, sending keypresses to the computer over USB is easy as including Keyboard.h and using something like Keyboard.press(KEY_UP_ARROW);.

There is extra logic for adjusting how many steps equal one keypress, as well as handling the shift register for driving the LED counter.

Gamification as Motivation

For me personally, a big part of why I’ve struggled with staying motivated to exercise is the lack of an immediate goal. To clarify, my long-term goal has always been to lose weight and stay in shape, but each time I exercise, I never see any part of that goal reflected in the immediate result of what I just did.

For a certain percentage of the population, the release of endorphins, or other physical stimuli and feedback loops, is the immediate goal that provides motivation to exercise. For me, this has never been the case; I have never gotten that positive experience that so many others describe, and exercise has always been something that taxes my body in an uncomfortable way.

To solve my motivation problem while building this project, I decided to implement a game for myself. The rules were simple:

I had to pick a real route, one that I would be interested in taking in real life
- Completing this route in Street View would be the long term goal
Each time I exercised, the following rules applied:
- I had to resume street view from the same place where I stopped my previous exercise session
- As much as possible, I tried to keep freeways off limits, as that felt like cheating, since you would not be able to jog / bike on those in real life
- Fun Rule: In order to end an exercise session, I had to make it (or at least try to make it) to a place that would be a resting place in real life. For example, hotels, motels, camp grounds, and rest stops would all be acceptable spots to end a session at
  - This was something that I felt added a lot to the gamification aspect and addressed my motivation issue; it gave me a short-term, immediate, and achievable goal to aim for each session. I also felt that it pushed me a lot harder than a simple time limit, since a distance goal requires a specific amount of movement, whereas a time limit does not dictate that.
Progress along the route was tracked with all of the following:
- Spreadsheet (start address, end address, and all exercise stats)
- Custom google map to visually show progress along route
- Calories also tracked in dieting app

Plans for the Future

Ideally, I would love to reboot this project sometime in the future. If I was going to go for my dream project, it would be to focus on the software side; I think you could easily turn a variant of this idea into a profitable SaaS product.

For example, instead of requiring specific exercise equipment or sensors for tracking input, you could use the accelerometer of a phone to track movement. The input could be fed into a street view program, VR games, etc. This would allow for extreme flexibility; you could theoretically design a piece of software that would work with any stationary exercise equipment (treadmill, elliptical, bike, rower), and have multiple mini-games with progress tracking. Throw in the requirement of a monthly subscription, and you have a nice monthly revenue generator.

You could also greatly expand on the gamification aspect; track your progress on multiple routes, gain XP, unlock achievements and “powerups”, etc.

This topic (VR exercise) fascinates me a great deal, and is something I think needs to be talked about more. Unless we make some big changes to public transit and focus on increasing pedestrian areas while reducing roads (which we should be doing), it will continue to get harder and harder to find open areas to jog / bike / stroll. Especially areas that are not crowded with people. As such, VR exercise might eventually become a more frequent alternative to the real thing.

Since this interests me so much, I have collected (and continue to collect) related projects and resources, as a shared Google doc. Feel free to check it out, and if you think something is worth adding, drop a comment below, or email me (see top menu bar for contact info).

Finding a Cheap Webcam in 2020, and Alternative Solutions

Joshua Tzucker — Sun, 07 Jun 2020 17:00:49 GMT

Due to COVID-19 and the related shift to remote work, there has been a shortage of webcams for the past couple months, especially those that are both name brand, and in the lower price range (under $100). For whatever reason, this shortage doesn’t appear to have an end in sight, but in the meantime, I thought I would share some workarounds and alternatives that I found, while trying to come up with a solution for myself to use for Zoom or other software.

Off Brand Webcams on eBay

For some reason, even the knock-off webcams on Amazon are still way overpriced - cameras that would normally go for $10 are selling at $50+. However, on eBay, the prices are closer to what they should be, and you can get an unbranded 720P or 1080P webcam for as little as $15!

Here are some filtered search results that show what I mean.

Warning: I would recommend only buying from sellers that have a decent amount of positive feedback, to avoid being scammed. That being said, eBay / PayPal generally sides on the buyers side, so you are protected no matter what.

Most of these are completely unbranded, or an unrecognized brand that is just slapped on. This usually also means that you will probably receive your webcam with:

A rather plain cardboard box, with little to no branding
No model number, or something generic, like A870
With either no manual, or something barely readable and not very helpful
With no customer support contacts, other than the eBay seller that you purchased it from
No dedicated control software
No warranty

However, most of these negatives don’t actually matter much. As for actually using the webcam, the majority of these unbranded models are completely plug-n-play, so there is not even any software to install anyways.

Now, I know what you are thinking (“you get what you pay for!”), but, at least from the videos I’ve seen, the image quality actually isn’t as bad as you might think for the price, and definitely passable for video conferencing.

Btw, I have a pretty low opinion of name-brand webcams; I think Logitech and similar brands have not really done much to improve quality over the years, especially in comparison to smartphone cameras, so I feel like it doesn’t take much to match their quality.

Smartphones as Webcams

The easiest solution, if you have a smartphone, is to simply use its built in camera and join your meetings on your phone instead of on your computer, using the mobile app version of your meeting software (Zoom, Teams, Skype, etc). If you need to be able to share your computer screen, check if your meeting software allows multiple concurrent sign-ins, and if not, see if you could join on one device as a logged-in user, and on the other, as a guest.

If you want to use your smartphone as a “true webcam”, where it is providing a camera feed into your laptop or desktop computer, that is a possibility, although it takes some work. No matter what, you are probably going to need to buy a piece of software either:

Android: DroidCam
iOS: NDI HX Camera

For a more in-depth page, check out this guide from Gizmodo. However, I think you are much better off just using the meeting app directly on your phone, rather than trying to route video from your phone through your computer (and Lifehacker came to the same conclusion as a I did).

Action Cameras as Webcams

This surprised me, but I came across a recommendation on Twitter to checkout action cameras (e.g. GoPro and clones) as an alternative to dedicated webcams, since many models support a “webcam mode”, where you can plug it in to your computer via USB and pull the camera feed. I’m glad I checked it out, because this ended up being the best option for me. With webcam prices being inflated as they are, you can buy a great action camera for the same price, or less, as a dedicated webcam.

I put together a comparison table between several of the lower-priced options. ~~I ended up going with the Victure AC700, which although pricier, has much better image quality than a lot of the competition.~~***

This option is also great for those planning on returning to work soon, or for whatever other reason only need a webcam temporarily. After you are done using it as a webcam, you now have a fully functional action camera that you can use!

*** = I actually have been having a lot of issues with the Victure AC700 and using it as a webcam. Certain applications work fine with (for example, Windows Camera), but with Zoom and some other apps, I have to fight against the image constantly going dark, or dropping out entirely. I can no longer recommend it as a webcam replacement 😢

Security Cameras as Webcams

Some security cameras can pull double duty as a webcam.

A great example is the Wyze Cam. It costs around $25, supports 1080P, and due to the Corona Virus, Wyze released a free firmware upgrade that lets you use it as a webcam.

Old Devices as Dedicated Video Conferencing Devices

This is also a great time to think about putting some new life into old devices you might have floating around the house. Outdated tablet? Old backup smartphones? Ancient laptop? It doesn’t take much to run most meeting software, and even if that is all the device can run, you could always keep it as a dedicated conference device.

DSLRs / High-End Cameras as Webcams

If you have a nice “professional” camera, like a DSLR, there are ways you can use it as a webcam and really upgrade your meeting image quality.

For certain cameras, there might be an existing free way that you can use it as webcam. For example, if you own a Canon DLSR, you should check out this article from Lifehacker on the release of Canon’s EOS Webcam Utility.

For pretty much any camera that offers HDMI out, you can always use a “capture card” to capture the video output on your computer. In fact, this is what many professional video streamers use in part of their setup. Unfortunately, these are usually pretty pricey ($50-200 range).

Elgato Cam Link 4K (normally ~$130, sold out right now)
BlueAVS 1080P Capture Card (~$50)

A common issue with using a DSLR as a webcam that you should be aware of is overheating; many cameras will automatically shut off after a set amount of time (such as 30 minutes) to avoid overheating.

Export LinkedIn Profiles and Connections to VCard or Sync

Joshua Tzucker — Fri, 05 Jun 2020 17:46:28 GMT

Intro

The vCard file format (often used with the extension .vcf, for Virtual Contact File) is a standardized way of storing contact information, such as name and telephone number, in a text based file. Since it is standardized, vCard files generated by one application, such as Google Contacts, can easily be read by another, such as Microsoft Outlook.

This article discusses the various ways that you can export profiles from LinkedIn as a vCard file, so you can add them to Outlook, Google Contacts, iOS, etc. It also discusses alternative ways to sync LI contacts / connections between different platforms.

Alternative Methods

Before I get into ways to generate vCard exports, I first want to cover some alternatives that are less technical and complex for most users, but will work if your goal is simply to sync contacts from LinkedIn to an external contact book.

Alternative Method - Native Sync

One less complex option that is available to LinkedIn users is to use the native sync feature. LinkedIn offers a built-in sync feature that synchronizes your LinkedIn connections with different external services; right now only the following are supported (accessible via your “Manage Sync Options” page):

Google (aka Google Contacts)
Microsoft Outlook (both Work and Personal)
Mobile Devices (through official LinkedIn App)

To my knowledge, the Outlook integration is the only option that allows for bi-directional sync, where LinkedIn connections will get exported from LI to your contacts, whereas the others only support importing from your contacts to LI.

The other major downside here is that you are giving LI a lot of your private information when you setup a sync, and many people might not be comfortable with that level of trust.

Bidirectional sync on mobile used to be supported, but has been deprecated.

Alternative Method - Third Party Services

Although there is a limited number of native contact sync integrations built-in to LinkedIn, there is a growing number of third-party services (usually paid) that have their own integrations built with LinkedIn’s API, or the new Sales Navigator system.

Warning: I have not tried, nor do I endorse, any of the following. Use at your own risk.

Zoominfo’s ReachOut Chrome Extension
SalesQL Chrome Extension
List of Official SNAP (Sales Navigator) Partners with CRM Support
- Example: Zoho CRM - LinkedIn Integration

Alternative Method - Scrapers

I really would rather not discuss this much, as I can think of very few good reasons to engage in this type of behavior, and hundreds of bad reasons to do so. Plus, scraping PII data off LI is, in general, against their TOS, and is a great way to get your account banned or limited.

There are enough bad things in life; please don’t scrape and spam from LinkedIn results.

Generating Exports and Converting To vCard

If you don’t want to use a synchronization service, and want to manually pull contacts from LinkedIn into a different program, your options are somewhat limited, especially since LinkedIn is restrictive in allowing PII data to flow out of their platform.

CSV Export from LinkedIn

If you want to bulk export all your LinkedIn connections contact data, you can do so, via the Request Archive feature of LinkedIn’s Privacy controls.

You can follow this step-by-step guide on how to request the CSV file, and then, once you have it, you can do any of the following:

Import into another program that accepts CSV inputs (Google Contacts, Outlook, etc.)
Use option above, and then export back out of the program in a different format (such as .vcf)
Use a converter to bulk convert the CSV rows into a different format such as .vcf
- Since this is a niche request, these tools often come at a price

Warning: Most of your contacts will likely show up in your export without an email address, even if you can see their email when logged into LinkedIn; this is due to a privacy settings specifically for downloading contact data. See the footnote in this LI answer page for details, and this article that explains the policy change and why so many people have email export disabled.

My Tool - Exporter Chrome Extension

I previously built a Chrome Browser Extension (linkedin-to-jsonresume) to export any given LinkedIn profile to the JSON Resume format; this is a schema for representing Resume information (work history, education, etc.) with JSON. Recently, I was asked if I could add vCard exporting to it, which is what actually prompted me to write this post, as I first wanted to research existing solutions before I spent any time writing my own.

I ended up adding basic vCard / VCF exporting support to the extension, which landed in my v1.1.0 release.

Please note that my extension is primarily focused on exporting an individual profile, and does not support bulk exports.

VR PSA - Check Your Cable Setup When Using Extensions

Joshua Tzucker — Wed, 03 Jun 2020 09:39:07 GMT

Discovering the Issue:
The Issue:
The Fix - Ways to Properly Extend VR USB Cables
TLDR
Extra Troubleshooting
Extra Reading

Disclaimer: This post contains several Amazon affiliate links.

I’m writing this post because I really wish someone had warned me about this when I first bought my VR headset (a WMR Samsung Odyssey+), and if they had, it would have saved me a lot grief. For a while now, I’ve been struggling to overcome certain issues with the headset, such as unreliable tracking and random glitches.

I think I finally just found the culprit, an underpowered cable setup, and this post was written as a guide to explain the issue, as well as solutions for properly extending VR cables without introducing issues.

Discovering the Issue:

These random glitches and tracking issues had been getting worse and worse recently, and at a certain point the headset felt unusable.

At that point, while troubleshooting everything I could possibly think of, I tried plugging the headset directly into the computer (no extension cables), and BAM - it started working a million times better. No tracking issues, no lost boundaries, and no glitches.

The Issue:

Like many other VR headsets, the Samsung Odyssey’s cable is really too short (4 meters / 13.1 feet) for comfortable room-scale play, so shortly after I bought it, I also bought and started using a 10 foot USB 3.0 extension cable. At the time, I didn’t think much about adding it to my setup - I knew at least that it was important to use 3.0 (or, more likely, a 3.1 variant) instead of 2.0, since VR needs the higher data speed and throughput, and this cable satisfied that need, so I thought I was good to go. Heck, the cable even advertises itself as being “for … VR Headset(s)”.

However, it’s not that simple. See, like many people, I was unaware that the recommended maximum length for a run of USB 3.0/3.1 cable is only about 3 meters (9.8 feet)(!), depending on cable thickness. The cable from the VR headset already exceeds that length, so adding an extension cable puts it way over the recommended limit, and explains why removing the extension cable suddenly fixed all my issues.

The Fix - Ways to Properly Extend VR USB Cables

Playing with the headset directly plugged into the computer without extension cables obviously works the best, but it is not practical for room-scale games, or even most standing games for that matter. So how can we extend the cable without exceeding the limits? It depends.

Best overall solution: HTV Vive Link Box (Amazon product page, Official Product Page)
- Pros:
  - Multi-purpose: repeats both USB and HDMI signals, as well as converts DisplayPort to HDMI
  - High quality: You can check out the internals in this teardown. For example, the USB hub chip is a Microchip USB5744, which supports full USB 3.1 Gen 1 speeds.
  - Great for mounting on ceiling, as single repeater for both HDMI and USB cable
  - Built-in Bluetooth, in case your computer doesn’t already have it
- Cons:
  - Pricey
  - Requires a separate power adapter (see below)
  - Requires a separate male-to-male USB cable
Powered USB 3.0 Hub (Amazon search results)
- Pros:
  - Relatively inexpensive, ~$10-20 for most
  - You might already have one laying around
- Cons:
  - Won’t repeat / extend HDMI connection
  - Requires an electrical outlet
“Active Extension Cable” (Amazon search results)
- Pros:
  - Some are self-powered (over the same USB connection it is extending)
  - Less clutter (single cable, instead of a connecting box)
- Cons:
  - Tends to have way more issues than alternatives that are externally powered, especially with VR headsets - see my explanation as to why, below
    - Externally-powered ones, like this one, tend to work better, but require an extra power cable, so you might as well just get a powered hub
  - Self-powered models only boosts signal, not voltage
  - Can have limited or no support for backwards compatibility (USB 2.0) or variation in spec
  - Often limited in disclosing the full specifications of their product

Vive Link Power Adapters

As great as the Vive Link Box is, a major drawback is that it does not come with a power adapter, and finding a suitable one to use with it actually took me a while.

To save you the same hassle, I’ll warn you that you will likely not be able to find an original AC adapter for sale (at least not “new”), and the requirements are as follows:

12V (DC)
1.5A (minimum)
Center Positive Polarity
Barrel Jack Size: 3.5mm x 1.35mm

With these requirements in mind, I ended up getting this adapter, and so far it has been working great. The most unique thing about the adapter is the barrel jack size; it is much easier to find 5.5 x 2.1mm 12V adapters, and then purchase a size adapter, like this 2 pack.

Thank you to these Reddit threads for providing details: this one, and especially this one (power adapter answers)

Issue With Self-Powered “Active Extenders”

There are a bunch of cables on Amazon that don’t require external power adapter, unlike USB hubs or the Vive Link Box, but still call themselves things like “Signal Booster”, “Active Extender Cable”, and “Active Repeater Cable”. Although these aren’t “snake oil” products, the way they work can have some negative consequences for VR applications.

Most of these cables include some sort of repeater IC (Integrated Circuit) chip, built into the cable itself. Usually this is noticeable, because one end of the cable will be very bulky to accommodate the IC circuit. To power these circuits, the cable basically siphons current from it’s own wiring - the connection it is extending.

This is often referred to as being “bus powered”, since the power is drawn from the computer’s USB bus, rather than an external power source.

For many devices, these cables work fine. However, VR devices tend to have very tight requirements and tolerances, since their main concern is latency; delivering an image with an added 10ms of delay (that’s just 0.01 of a second) can be the difference between a great VR experience, and one that makes you feel like you are going to throw up from motion sickness.

To get back to the issue, there are two major problems with these types of cables:

Those repeater chips (often called “redrivers”) often only need a small amount of power to be siphoned off. For example, the NXP PTN36241B, which is used in the CableCreation 16.4 Foot Active Extender, only requires about 100ma of current while active, although the circuit integrating the chip probably draws a little over that. 100ma is not much, but the VR headset itself also draws a significant amount of power, to run the displays and assorted circuits, and it can be the case that there simply is not 100ma to spare, especially if you have an older computer with a low maximum supply current for USB.
- This is an issue for the Samsung Oyssey+ in particular, because of how much current the device draws. The Odyssey and Odyssey+ draw about 900ma, which is right at the border of what the USB spec says a port is required to provide.
The “bus powered” repeater chips boost the data signal strength, but do not usually boost the voltage, since realistically this would require an external power supply (to handle the current required by the boost circuit itself). This can be a big problem, because voltage will naturally drop as a cable extends in length, and many VR headsets are not tolerant of lower voltage supplies.
- For example, in a 16 foot run of USB cable, without a voltage boost, the voltage drop could easily exceed 0.5v, which could be outside the USB specifications, and likely to to cause issues with a VR headset
- Normally, your motherboard and USB port can adjust to compensate for a voltage drop, but combined with the issue above, it might not be able to with the load of both the repeater and headset running

These two limitations of self-powered repeaters, especially when combined, are often deal-breakers for VR headsets. To this point, note the small FAQ section the product page linked above for the CableCreation active extender:

Q: Does this extension work Samsung Odyssey VR?

A: Yes, but because of the extended length […] you need to connect a powered USB hub or adapter for external power supply.

So, if you buy a self-powered repeater cable, you might also have to buy a powered USB hub to plug it into, which kind of defeats the purpose…

Sensors VS Headsets

You might be wondering why on a lot of Amazon reviews for various extenders / repeaters / cables, you see a common pattern where there are positive reviews saying “this worked great for ____ VR sensor!”, but also negative reviews of “this didn’t work at all with ___ VR headset”.

The reasons for this are pretty simple:

Sensors usually require a lot less total throughput, compared with a headset (which also often has built-in sensors itself)
Sensors require much less power than headsets, since they aren’t driving bright displays combined with audio, processors, and more.
Sensors often have greater tolerances for different types of hosts (for example, the Rift Sensor can work over USB 2.0 in addition to 3.0, whereas most headsets require 3.0)

In general, a standalone sensor just requires a lot less “stuff” to function, in comparison to a headset, which is built with a bunch of sensors, plus dozens (if not hundreds) of other components.

Samsung Odyssey and Odyssey+ Peculiarities

I’ve touched on this above, in various sections, but I think part of why the Samsung Odyssey+ in particular seems to have so many users complaining about issues with extending USB length is because the device was built to basically be right at the edge of all “maximums”, out of the box. For example:

Various manuals and webpages make it sound like the device absolutely needs a full 900ma to function. That is literally the maximum required by the USB 3.0 spec.
- USB ports are free to supply more than that amount, but are not required to, and it is up to the PC manufacturer.
Various sources also make it sound like the Odyssey is not tolerant of voltage drops; e.g. it needs the USB supply voltage to be as close to 5V as possible
- This is especially problematic since the Odyssey+ comes with a 4 meter (13.1 feet) USB cable, which is already long enough to be in the danger zone (or right at the edge) for causing voltage drops. It helps that the USB cable is thicker than most.

Disclaimer: I’m not an Electrical Engineer, so most of this is based on quick research, not prior knowledge.

These Reddit threads (1, 2, 3, 4) are good examples of what I mean by this being a common issue for Odyssey users (who could probably benefit from my advice on this page).

Why Is HDMI Less of An Issue?

You might be wondering why, so far, I have written pretty much exclusively around how extending USB is an issue, and doing it incorrectly leads to issues with VR headsets. Why not HDMI as well?

I’m not on expert on these things, but I think I can simplify without being incorrect, in saying that extending the HDMI cable with VR headsets is generally not as much of an issue as USB because of the following two factors:

HDMI has a longer maximum length than USB to start with
- There is not a specific maximum length put forward in a standardized HDMI spec, but generally, 25-50 feet is considered the maximum, without a repeater.
- In comparison, a standard thickness (24-28 awg) USB 3.0 cable should not exceed somewhere around 10 feet.
HDMI is being used for less things than USB, on a VR HMD
- With a standard VR headset, HDMI carries the video feed, and maybe the audio feed. This data (mostly) only has to flow in one direction, and there is some tolerance built-in
- USB, on the other hand, is carrying the power supply, sync data, microphone feeds, and sensor data. Sensor data could be multiple bi-directional feeds, encompassing live cameras, accelerometers, gyroscopes, compasses, and more. Many of these pieces of data have to be delivered with minimum latency, as any delay will increase the risk of motion sickness.

Having said all that, extending the HDMI cable could still be an issue for you, either over longer lengths or with thinner cables, which is why I still recommend the Vive Link Box as the best extender solution, since it extends both the USB and HDMI connection in one device.

TLDR

Here is the TLDR of this post; if your VR headset frequently has tracking issues, loses boundaries, or other random glitches, take the time to check your cable setup and make sure you are providing enough power for the length of cable used.

List of symptoms that could appear with cable length issues:

Frequent “lost boundary” error message and/or room starts spinning
Controllers lose tracking and start floating randomly
- Tends to happen with fast games, like Beat Saber
“phantom” controller input; random key presses and trigger pulls
- This one also seems to be caused by sunlight and/or reflective surfaces around the play area
Entire headset randomly restarts
- Especially when cable is moved
Headset takes a long time to recognize environment and/or sync with controllers
Random audio “hiss” or “static” noise

Best ways to extend cable length without issues:

Use a powered repeater box (powered USB 3.0 hub, or “Vive Link Box”)
Use cables marked as “Active Extension” and/or “Active Repeater”, especially those that require external power sources

Extra Troubleshooting

Make sure your USB cable is plugged into a 3.0/3.1 port. Usually, these are marked with a “SS” logo, for SuperSpeed.
- If your PC has any USB ports labeled “high power” or something like that, switch your VR headset over to that port
Make sure you are using an HDMI connection to match the requirements of your HMD, usually 2.0
Check that controllers are charged up
Check for firmware updates
Try re-pairing controllers
Make sure your play area is well-lit, but not with sunlight, and is free of reflective surfaces

Extra Reading

If you are nerdy like I am (🤓), you might share my interest in these resources that I came across while writing this:

Intel / HP / TI / Microsoft / Etc. : Universal Serial Bus 3.0 Specification (This is the full USB 3.0 specification document)
USB org / Intel: Power Delivery Design Issues for Hi-Speed USB on Motherboards
NTI (Network Technologies Incorporated): An Introduction to USB Extenders
USB org: Logo Guidelines (found when searching for port identifiers)
Techquickie: When Does Cable Length Matter (queued up to USB part)
Microsoft - Windows Mixed Reality: Enthusiast Guide - Troubleshooting Windows Mixed Reality

VSCode Intellisense Autocomplete for Webpack Config Files

Joshua Tzucker — Fri, 15 May 2020 02:27:36 GMT

Click here to jump right to the recommended solution.

Warning: I will use the terms “intellisense”, “autocomplete”, and “hints”, somewhat interchangeably throughout this post. They are not technically equivalent, but you should be able to get the gist of what I mean.

Introduction

One of my favorite parts of how the developer experience has improved is how much better IDEs have gotten at providing “hints”, or warnings about “silly” mistakes. Intellisense for config files is something I deeply appreciate, as I don’t care to memorize hundreds of different schemas in my head, especially when the IDE can do it for me, as well as validate my input.

There are many config files that have intellisense support built-in when edited in VSCode, such as .babelrc, package.json, and jsconfig.json, plus hundreds more are contributed by extensions and modules.

Problem:

You might have noticed that, by default when you are building a webpack.config.js file, VSCode does not provide intellisense / hints as you type. Given the somewhat notorious complexity of webpack, this is something I was somewhat surprised to find, and doubly surprised to find very little information out there on how to enable. Thus, the reason for making this post.

Solution(s):

Solution A: Use TypeScript Definition

Using TypeScript definitions for webpack config intellisense was actually previously suggested on multiple related Webpack issues (VSCode #24270, VSCode #24657, and TS #13333), but for a while, was not working. Now, thanks to improvements in VSCode’s handling of TS-Powered JSDoc comments (very impressive), it works just fine.

To use, simply prefix where you are declaring the config object with a JSDoc type annotation, like so:

/** @type {import('webpack').Configuration} */
module.exports = {
    mode: 'development',
    // ...
}

Here it is in action:

The interface declaration comes from @types/webpack.

If you run into issues, make sure you have already added Webpack as a dependency, and as a last resort, you can add @types/webpack as a devDependency.

Solution B: Use JSON Config and Published Schema File

According to this issue, Webpack already supports passing in a .json config file instead of JS. However, creating a webpack.config.json still won’t get Intellisense to kick in automatically, so you have to go one more step - tell VSCode what schema to use.

I don’t want to go into too much detail on this, but JSON Schema is a very powerful mechanism, and is well-supported in VSCode. I have a cheatsheet section devoted to it, here.

After adding webpack as a dependency, in webpack.config.json:

{
    "$schema": "./node_modules/webpack/schemas/WebpackOptions.json",
    "mode": "development", // ...
}

This method is not recommended in comparison to method A, because you lose the ability to do anything outside of the JSON spec - e.g., you can’t use comments, you can’t use variables and globals (no __dirname!), and many other restrictions. Plus, now that method A works, there is no real benefit to this approach, and I’m mainly including it as a demonstration of how VSCode supports and resolves JSON Schema.

Wrap-Up

Hope this helps someone! Feel free to let me know in the comments if I missed anything!

More About Me:

Building a Perfectly-Sized Dynamic Cover Image with Cloudinary

Joshua Tzucker — Wed, 13 May 2020 08:30:54 GMT

For those unfamiliar with Cloudinary, it is a cloud service that is focused on image serving, with unique features like transformations, responsive generation, and a speedy CDN for delivery. I have no affiliation with Cloudinary, other than being a fan of their service.

Introduction

One of the big changes in web development over the years has been a shift from “fixed” designs to “responsive”. For example, where we used to define layouts or the display of assets with fixed units, we now try to use percentages or other ratio based approaches to adapt to any display size. This is something to be celebrated, and Cloudinary has an entire feature set built around this, but today I want to talk about when the opposite approach is needed.

We often have “cover images”, “hero images”, and/or images that we need for social media sharing sites, and these frequently have specific desired pixel sizes or aspect ratios. For example:

Dev.to cover image: 1000 x 420 (see Editor Guide)
Twitter: (depends) 16:9 (1.77:1), 2:1, 1200x628 (1.91:1)
Facebook feed: 1.91:1

Finding images that perfectly fit these aspect ratios is often difficult and time-consuming, and manually cropping and resizing existing images to fit, equally so.

What if there was a better way to generate the perfectly sized hero image, with any image as the input?…

Our Goal

So, here is our example goal. We want to produce a 1000 x 420 pixel (2.38:1) cover image for Dev.to, but do so automatically, with a method that accepts any image as the input. If the input aspect ratio does not match the desired output, then the input will be centered, and a blurred background based on the same input shown behind it.

Sample Input:

Sample goal output:

The cover image for this very post was created with the method outlined below! But, you’ll see that later.

Building It With Cloudinary

I’m going to walk through how to build this with Cloudinary image transformations, using URL parameters alone. This makes it very easy to use anywhere, with any backend or frontend, since all we need is a URL.

Base Image: https://res.cloudinary.com/demo/image/upload/sample

When you see Bold and Italic text in the URL, that indicates what was added in the step.

Steps:

Start with the “Base Image”: https://res.cloudinary.com/demo/image/upload/sample
- Added: sample
Use image to fill exact desired size
- https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420/sample
- Added: /c_fill,w_1000,h_420/
Since this is actually going to be the background, apply a blur effect and reduce opacity
- https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/sample
- Added: e_blur:1500,o_75
- Blur is on a scale from 1 to 2000
- Opacity is on a scale from 0 to 100 (100 being completely opaque)
Overlay the same image, but limit its size to the “canvas”
- https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/l_sample,c_limit,w_1000,h_420/sample
- Added: /l_sample,c_limit,w_1000,h_420/
- Notice how l_{uploadedId} is used to generate an image overlay layer.
Add a nice blurred border to our overlay
- https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/l_sample,c_limit,w_1000,h_420,e_outline:outer:2:100/sample
- Added: e_outline:outer:2:100
- You might notice that the border now pushes the overlay outside the canvas…
Prevent “overflow” caused by new border
- Adding a border to the overlay actually made it larger than the boundaries of the background (base image). We have two options to remedy this:
- Option A: Re-crop: Since adding a border to the overlay made it larger than the boundaries of the background, we can simply re-crop the output
  - https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/l_sample,c_limit,w_1000,h_420,e_outline:outer:2:100/c_crop,w_1000,h_420/sample
  - Added: /c_crop,w_1000,h_420/
  - If we wanted to re-crop just the overlay, we could use fl_layer_apply to prevent crop action from applying to the entire chain
- Option B: Use the no_overflow_flag
  - This is a super helpful flag to remember! From the docs:
    - Prevents Cloudinary from extending the image canvas beyond the original dimensions when overlaying text and other images.
  - https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/l_sample,c_limit,w_1000,h_420,e_outline:outer:2:100,fl_no_overflow/sample
  - Added: fl_no_overflow

Final output, using option B:

Extras:

There is still more we can do with this!

Add compression
- To save some bytes, we can tell Cloudinary to serve as a compressed JPG with a quality of 80%
- https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/l_sample,c_limit,w_1000,h_420,e_outline:outer:2:100/c_crop,w_1000,h_420,q_80/sample
- For our sample image, this gives us a saving of approximately 50%!!!
Add overlay text
- We can add overlay text, such as the title of the post that our cover image is for
- https://res.cloudinary.com/demo/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/l_sample,c_limit,w_1000,h_420,e_outline:outer:2:100,fl_no_overflow/l_text:Roboto_100_normal_stroke_:Hero%20Image,co_rgb:FFFFFF,bo_8px_solid_black,g_south,y_30/sample
- stroke, and border (bo_) are used to provide an outline around the text

Wrap-Up

This might have seemed like a bunch of work, but don’t forget; now that we have the transformation chain built, we can reuse it across an unlimited number of images, simply replacing {uploadedId} with our desired input:

const myCloudId = 'acb123';

// pretend `myImages` is an array of hundreds of image IDs
myImages.forEach(uploadedId => {
    const heroImage = `https://res.cloudinary.com/${myCloudId}/image/upload/c_fill,w_1000,h_420,e_blur:1500,o_75/l_${uploadedId},c_limit,w_1000,h_420,e_outline:outer:2:100,fl_no_overflow/${uploadedId}`;
    saveImage(heroImage);
});

In fact, here it is in action, working its magic across any input I give it (large GIF, give a second to load…):

Nice, right!?

We could also…

Save this as a Named Transformation
- Once we do this, reusing it becomes as easy as /image/upload/{transformationName}/{uploadedId}
Implement this via Cloudinary API
Integrate into any backend (WordPress, Gatsby, etc.) for dynamic meta image tags to serve to Twitter, FB / OpenGraph, etc.
… and many more things!

If this was interesting to you, I have a few other Cloudinary related projects that might be worth checking out:

Cloudinary Cheatsheet - WIP quick cheatsheet

Desktop Cloud Transform - Transform local images with Cloudinary, with easy drag-and-drop GUI

Cloudinary WYSIWYG - Visually build Cloudinary transformations with interactive canvas

More About Me:

Scripting My Morning Wake-Up Alarm and Lights with Android and Kasa

Joshua Tzucker — Wed, 06 May 2020 03:56:20 GMT

TLDR: This post covers how (and why) I setup an Android automation flow that automatically turns on lights in my room whenever my morning alarm goes off, regardless of which alarm app I use!

Intro

I have something to admit. I’m one of those people that has struggled with my sleep schedule my entire life. It constantly shifts, and when I force myself to wake up at a certain time with an alarm clock, both my mind and body fight against the process. Don’t get me wrong (especially potential employers 😀) - I can always force myself to get up when I need to, but it has not been easy.

As such, it should be no surprise that I also have tried many different types of alarms, and have settled on “challenge” alarm apps. These are alarm apps that force you to complete certain tasks before the alarm will shut off, such as scanning a barcode, solving math problems, or shaking your phone.

Can I Get Some Light in Here?

One of the challenges that my alarm clock app (Sleep as Android) has is called “Let there be light”; in order to complete the task, you must expose your phone to a high level of light for a minimum amount of time (adjustable).

I sleep with blackout curtains because I’m light-sensitive, and this is great for falling asleep, but makes it too easy to sleep-in late in the day. Thus, this challenge has been great for forcing me to bring some light into my pitch-black environment in the mornings.

…but, I was wondering; can I take this a step further? Can I have my lights automatically turn on whenever my alarm goes off?

Automating Power Outlets with Android

I already had a TP-Link Kasa Wi-Fi Plug that I wasn’t doing much with, so I set out upon figuring out how to automate it based on my Android alarm settings.

The Kasa software does support built-in scheduling, but you have to manually enter times for it to turn off and on; I wanted an automatic solution that would hook into the system-level Android alarm and would work regardless of what alarm app I was using, and no matter how many times I changed my alarm time.

Google Apps Script Wrapper

My first step was to create a “wrapper” script / API around the unofficial Kasa API. I wanted to do this, because using the Kasa API requires a multi-step token exchange, and I wanted to be able to control my switch with a single network request (which I knew would make the automation piece easier down the line).

I used Google Apps Script for this, since it is a free and very easy to use JavaScript serverless environment; similar to AWS Lambda, but without the complicated pricing or web console. Its ease of use makes it great for one-off hobby projects like this one.

I won’t go too much into the details of the script here, but the basics are that it:

Caches my credentials and token to avoid extra API requests
Refreshes the auth token when it expires
Simplifies using the Kasa API by wrapping and formatting requests for it
Published as a “web app”, so I have an endpoint I can POST to

I’ve posted my script here, as a Github gist.

Integrating with Android via Automagic

Since my Google Apps Script gives me a single endpoint I can turn my lights on with a single request, all I need to automate my flow in Android is an app that can A) listen for the alarm trigger, and B) make a POST request with a payload. Luckily, I already have a go-to automation app that meets both those requirements. Meet Automagic.

The Automagic flow is super simple:

Trigger = Next Alarm
- Fires when alarm goes off
Make sure I’m at home (connected to home WiFi SSID)
- If false, stop flow
Fire POST request to my Google Apps Script endpoint
- JSON payload says to turn my bedroom light on
Capture and log response from script, just in case something failed

And here is what that flow look like in the Automagic UI:

The actual POST request that triggers the light turning on is very simple as well (thanks to the Google Apps Script wrapper). Here is an example:

{
     "pass": "REDACTED",
     "deviceId": "REDACTED",
     "action": "on"
}

Android Internals - Intent Broadcasts

Because I was curious about how Automagic might have built that trigger behind-the-scenes, and how you might approach this with other Android apps, I did some digging into Android Intents. The TLDR of an Android Intent, is that it is a request and/or message that is broadcasted from one app to another (or the general system). The Android OS itself broadcasts many of these messages, which makes it easy to automate things like this.

For example, if I wanted to have my light come on 10 minutes before my alarm goes off, I could use something like this:

Listen for android.app.action.NEXT_ALARM_CLOCK_CHANGED
Use getNextAlarmClock()) to determine the time of the next alarm, then subtract 10 minutes
Use the calculated time to set a trigger for the POST action

You can read more about Intents here: Docs - Intents and Intent Filters

No-Code Options

I thought it important to note that this kind of automation is very close to being a prime candidate for a “no-code” approach, where WYSIWYG tools are used to build the automation as opposed to coding.

Most of what I put together could probably be replicated with a combination of some of the following:

IFTTT (one of the few official integrations for Kasa)
Zapier
… etc.

Get ALT Keyboard Shortcuts Working Quickly With JavaScript

Joshua Tzucker — Tue, 28 Apr 2020 23:18:09 GMT

I was trying to use a web app today, and ran into an interesting issue. It uses ALT + Left Arrow Key as a shortcut for a very handy operation, one that I would like to use frequently. However, at least on Windows, that keyboard shortcut gets stolen by the browser, and navigates me back a page. How do I work around this?

Well, there are multiple solutions. This post is meant as a quick notes session as I explore them, so for more depth solutions, please do your own research.

For the Designer of the Web App

If I was the creator of the web app, there would be a very simple solution. You can simply use the event.preventDefault() call inside an eventListener callback, to prevent the default browser action. For example, if I wanted to prevent the backward page navigation and do something else, I could use this code:

document.addEventListener('keydown',(evt) => {
	if (evt.keyCode === 37 && evt.altKey) {
		// Stop browser action
		evt.preventDefault();
		// Do something here
		console.log('ALT + Left Arrow Pressed');
		// ...
	}
});

You can actually this type of code to override any browser keyboard shortcut - for example, if you wanted to override F5 to have it do something else instead of refreshing the current page.

The only thing you really cannot override is Operating-System level keyboard hooks, such as CTRL+ALT+DEL on Windows.

For more details on event.preventDefault(), see the MDN doc page.

For myself, as a user of the app

Since I’m not the creator the app, if I wanted to fix this myself I would either need to make a pull-request to the codebase (assuming it is open source), or implement a local workaround. I’m going to explore the second option now, since it will also be faster for a temporary workaround.

Injecting Code

As a user of the web app, although I can’t touch the source code, I can inject whatever JavaScript I want into the page through the developer’s console, browser extension, bookmarklet, or other method, and there is not much the app can do to prevent that.

Assuming that the app I’m using is a SPA (Single Page Application), then a simple bookmarklet should do the job, since the JS environment is not reset between actions (as opposed to a regular web site, where actions usually cause a full page refresh). I’ll write a little JavaScript that calls preventDefault() on all alt key combinations, and wrap it up in a bookmarklet. In fact, here is that code and bookmarklet:

Code:

document.addEventListener('keydown', (evt) => {
	if (evt.altKey) {
		evt.preventDefault();
	}
});

Bookmarklet - Drag the below link to your bookmarks: ALT preventDefault

Exceptions:

As already mentioned, there are exceptions to the ability to prevent keyboard presses from triggering browser or OS level events (for example, CTRL + ALT + WIN on Windows).

Another issue is that the implementation of keyboard events and their propagation is some what left up to the browser. For example, my above code will successfully block ALT keyboard combos on both Chrome and Firefox, but for a single ALT keypress, it will only block the menu from opening on Chrome, and not on Firefox. However, this might be a bug rather than a specific choice - see this bug tracker.

React Github Contribution Graph Mock Tool

Joshua Tzucker — Tue, 14 Apr 2020 23:03:34 GMT

Project Links:

What is this?

This is a tool, purely for fun, to put text and different graphics into a mocked-up Github Contributions Graph (see this if you don't know what that means). This was a random idea that I had, and wanted to see if it was possible to build from scratch, with minimal component libraries. It ended up working better than I had expected; just take a look below!

Tech Stack and Learning:

Most of what I used to build this was pretty vanilla React stuff:

JavaScript
ReactJS
Materialize CSS (for styling)
Canvas API

The most challenging part, and also the most interesting, was turning the text input into a point-based visual representation. Mostly, this meant learning how to use the Canvas API (JavaScript & HTML) and processing the raw data it returns.

I also tried to use React Hooks more on this project than I previously have (I'm used to TypeScript Class or JS ES6 Class based components), which required some additional research and self-educating.

What is a Github Contribution Graph?

For those unfamiliar with the site Github, it is hard to describe the full depth of what they offer, but at its core, Github is a service that developers can use to store their code, as well as track changes and collaborate with other developers.

As a Github user, you get your own profile page (for example, github.com/joshuatz), which you can customize with some links and text. One of the fun features of Github's profile page is an automated widget, commonly referred to as the Github Contribution Graph. It shows how frequently you have been pushing code through Github, with different intensities of green to signify more activity on a given day. Here is an example:

This type of data representation is also often referred to as a heatmap.

Remote Tutoring Software - Screen Sharing and Video Call Showdown

Joshua Tzucker — Sat, 04 Apr 2020 19:35:12 GMT

Intro
Different Types of Software
- Note on Applicability
Deciding
Research Scratchpad
- Search Terms I Used:
- Lists I found helpful:
Recurring Issues
- Click-and-drag on Mobile
- Mobile PiP Support
Final Note:

Intro

Due to recent events (Covid-19), my mom, a bit of a technophobe, has suddenly taken an interest in using technology as part of her tutoring business. She tutors children (K-12, with a focus on elementary), so her requirements are different from college professors that might only need basic video conferencing tech to lecture for an hour at a time.

As she asked me questions about what she could use, I started to realize that a lot of her software needs are actually going unmet by the industry, and there actually exists a strange gap in the market for quality educational collaborative software. Since others might find it interesting or helpful, especially if they have similar needs, I thought I would document my research here, and outline what is available and what is not, as well as what I thought the best option was.

Different Types of Software

First, a discussion of what types of software are out there that could be used in her tutoring business. These categories are also applicable to dozens of other industries:

Type	Description	Examples
Video Conferencing	Software where the focus is on video conferencing, video chat, etc. Might also include features such as screen-sharing, remote control, or other built-ins, but these are often are not full-featured.	Zoom, Skype
Collaboration Platform	These are usually marketed directly towards businesses, and meant to serve as a one-stop shop for collaborating with peers. Common features are text chat channels / instant messaging, video chat, project planning, calendars, and enterprise software integrations. These are usually overkill for smaller businesses, and especially for tutoring.	Microsoft Teams, Zoho Cliq
Screen Sharing	Software that lets you share what you see on your screen with another user, or vice-versa. This is rarely ever its own category of software, and usually just an included feature in either Video Conferencing (see above) or Remote Control (see below) software.	Screenleap, Slack Screenshare
Remote Control / Remote Desktop / VNC	Software where the focus is on letting one user control another user’s computer. Does not usually allow for more than two concurrent users at a time (not meant for conferencing)	TeamViewer, GoToMyPC, Windows RDP, Chrome Remote Desktop
Cobrowsing	A way for multiple users (2+, but often limited to just 2) to synchronize browsing of the same web page and/or web browser. There are variations in how this is done, and some cobrowsing software is geared towards technical support or live customer assistance chat (e.g. web commerce), not educational or personal use.	Samesurf, Surfly, Acquire, Upscope, Vectera
Shared Whiteboarding	A shared digital whiteboard or notebook where multiple users can edit content and have changes reflected live in all participants’ views. Ideally the content that is created during a session can be persisted (saved), and edit history preserved. This feature is baked into many video-conferencing solutions, such as Zoom and Teams.	Twiddla, Miro, LiveBoard, Microsoft Whiteboard, OneNote, Google Drive (to an extent…)
LMS (Learning Management System)	LMS software is an extremely broad category of product offering, so features vary greatly from one software platform to another. For example, some LMS programs offer built-in video conferencing and meeting tools, while others focus only on content and webpages (which makes them basically just an expensive educational focused CMS…) Most LMS are focused on building a “virtual classroom”, with core features centered around content, moderation, and support for a large number of participants. LMS are often purchased not by the educator, but by the institution (college, high school, etc.). These factors combined often make it a bad choice for tutoring software.	Open edX, Google Classroom, Blackboard, Canvas, Electa Live
Virtual Assistance	Usually this software is sold to tech support teams / call centers, and not geared towards education use. This type of software allows an “operator” or “agent” to provide virtual assistance to a “client” or “customer”, in order to walk through a specific problem. Common features are co-browsing (“let me control the webpage you are on to show you how to use X feature”), live chat, remote desktop control, and multi-agent / ticketing system support. These are often missing collaborative features, like webcam sharing, whiteboards, or allowing more than two participants at a time.	Zoho Assist, ConnectWise, AnyDesk, BeyondTrust, LogMeIn Rescue, TeamViewer
Sales, Sales Funnel Assistance	There is a specific niche of collaboration software that is for salespeople pitching to potential clients / customers. Core features might include slide decks, video conferencing, co-browsing, email integration, etc. Features that are often left out are annotation, whiteboard, and other “creative” collaboration tools, since the goal is to sell something existing, not brainstorm a solution.	DemoDesk, CrankWheel

Note on Applicability

If you are looking at the above table and feeling lost, I can offer a few pointers. In general, each of the software categories above is marketed towards a certain industry and/or type of occupation, but that doesn’t mean it is only useful for that industry. For example, Remote Control software is always heavily marketed to tech support and IT professionals, but it has a million uses beyond that field. So, focus on what the software does, not how it is marketed.

Another example: DemoDesk, which markets itself for sales and customer support, was remarkably close to what I was looking for, and I have a feeling a lot of educators might actually find it useful.

And hey, marketing departments? Take note; you don’t always know your target demographics as well as you think you do.

Deciding

Requirements

Here are the main features that we consider required in our search:

Video chat that can stay up throughout session, even while switching to remote desktop control
- Tutor can see student, student can see tutor, entire time
Support for remote desktop control, where host computer can be controlled by participant
- Student can control tutor’s screen, in order to share browser / educational desktop apps
Mobile / Tablet / Touchscreen Support: There must be touchscreen support, and that includes the requirement that remote desktop control support “click-and-drag”, just like how you can with a mouse
- Necessary because lots of elementary school activities require drawing or writing, instead of typing
- This is where most offerings fell short, especially on click-and-drag support - see recurring issues section.

These would be nice bonus features, but not required:

Native browser support (no client to install)
Built-in whiteboard and annotation support

The basic experience I was looking to find was one where the tutor would feel like the student was sitting next to them, sharing the mouse and keyboard of their own computer.

Coincidentally, this set of requirements is almost identical to what most programmers are looking for in remote “pair programming” solutions!

Based on the above requirements, the software category that best fit was screen sharing, in places where it overlaps with Remote Control and Video Conferencing software. This combination also overlaps with Remote Pair Programming software, which has similar requirements. Knowing this, I researched (and actually tried out!) a multitude of options, and compiled the results into a comparison table.

Click here to view the raw Google Sheet document in a new tab.

Embedded:

Side Note: Web RTC

Technical: Web RTC is a set of web APIs and standards that enables and streamlines the process of capturing and streaming multimedia (including video, such as from a webcam) data streams, in an efficient and robust manner. It allows for zero-install setups, since these are native browser APIs (albeit still changing).

Web RTC, in my view, is an incredibly strong indicator of how on top of modern technology a given video chat / screen sharing program is.

In fact, to my knowledge, the three pieces of software that I’m going to recommend for my use-case all rely on Web RTC to deliver their streams. Jitsi Meet, one of three, and an amazing piece of open-source video conferencing technology, even published a post on how they can out-perform Zoom with Web RTC.

Summary: Web RTC is awesome, and usually means there is no software for your user (student, client, etc.) to install or setup.

Close Fits & Honorable Mentions

BrainCert - LMS
- Looks like a really cool LMS, but no mouse sharing / remote control is a deal-breaker for our use-case
BigBlueButton - Education Focused Web Conferencing Solution
- BigBlueButton (aka BBB) is an extremely impressive piece of software, even more so since it is both open-source and free. It has been put to the test as well; it is the default video conferencing solution baked into the Canvas LMS.
- BBB combines video conferencing, interactive shared whiteboards, and screen-sharing, with a focus on educational features (such as polling and special presenter controls)
- One major drawback to BBB is that it is only self-hosted, meaning that you need to have access to a (powerful) dedicated server you can run it on. As a nerd and open-source supporter, I’m a big fan of having a self-hosted option, but… I would not expect the average teacher or tutor to know how to provision a Linux server, interact with bash scripts, and setup port forwarding…
Join.Me
- Superior to Zoom in that it supports “no-install” use; guests can jump into a meeting and get access to all features from their web browser, without installing any software. This includes the use of remote control and share screen.
- UI is very intuitive and pleasant to use
- Mobile client (app) does not support remote cursor control at all
  - You can get around this by opening the meeting link in Chrome for Android, in an incognito window, with “Desktop site” emulation setting checked.
    - However, this has the same “click-and-drag” issue as Zoom :(
DemoDesk
- This was pretty close to being exactly what we needed in so many ways:
  - No bulky software to install
  - Co-browsing support
  - Mobile touch screens support click-and-drag
  - Built-in video conferencing
  - The “playbooks” feature is a neat way that a tutor could preload content that is going to be worked on during the session
- Pretty much the only reason why this was not a suitable fit is because remote control / mouse sharing only works for co-browsing webpages or uploaded documents - it does not work for sharing your desktop or OS applications
- I still think this is a very unique and powerful web-app that might be the perfect for other use-cases.

Our Picks

Runner Up:

Screen.so
- WOW! So close to being exactly what I was looking for!
- Similar to USE Together, but adds video chat and annotations
- Blockers:
  - Buggy. This is understandable, as I was testing it just days after it was released to the public
  - Odd touchscreen to mouse mapping: trackpad mode
    - Clicking at point 0,0 on your screen does not place the host mouse at 0,0 - but dragging your finger from 0,0 to 100,0, will move the host mouse 100 units to the right (scaled) - just like a trackpad
    - To click-and-drag, you first long-press, then start moving your finger
    - Maybe in the future they will support multiple mapping modes and let you switch between? That would be nice!
- If the blockers are ever addressed, I will probably recommend to my mom that she instantly switch to using Screen.

🎉 Finalist 🎉

The final pick is actually a combination of two pieces of software together:

Piece A: Video Conferencing - One of…

Skype (yes, it’s still around)
- Most important feature: picture-in-picture support on both mobile and desktop!!!
  - As already mentioned, this is crucial so that tutor and student can keep seeing each other even as other applications are taking focus
- Supported on many different devices, been around forever
- Negative: Cannot set password on ad-hoc “meet now” meetings that don’t require participants to login
< ==== Or… ====>
Jitsi (aka Jitsi Meet)
- Open-Source!!! - github.com/jitsi
- Powers 8x8 video conferencing
- Jitsi has a lot of features that other video conferencing apps are missing:
  - picture-in-picture support on both mobile and desktop!!!
  - Optional self-hosting / server control
  - No clients to install (native browser support)

Piece B: Remote Desktop Sharing:

Remote desktop sharing: USE Together
- I found it to have the smoothest cursor sharing experience - very little lag
- Unfortunately, basically the only feature offered is cursor sharing; there is no video chat built in, or annotation support
  - This is why I have to combine it with a video chat app, so that the tutor can see the student (and vice-versa) while sharing control of the computer
  - There is built-in audio chat, so if you don’t need video, you could get by with just this application

Using these two pieces of software together feels very close to what I was hoping for; the experience of simply sharing the keyboard and mouse with the other person, while seeing them the whole time.

Research Scratchpad

Search Terms I Used:

Collaborative web browser
synchronized web browsing
live webpage sharing
lms with rdp
video conferencing with shared cursor
tablet share mouse control video conference
android share mouse control video conference
web rtc rdp
android video chat “SYSTEM_ALERT_WINDOW” OR “pip” OR “picture in picture”
remote pair programming with video chat
remote desktop with video chat
pair programming “remote” “video chat” OR “video call” OR “webcam”

Lists I found helpful:

Process.st - The 11 Best Screen Sharing Apps
FreeConferenceCall - Best Remote Desktop Screen Sharing
Wikipedia:
- Comparison of Remote Desktop Software
- Comparison of Web Conference Software
https://www.capterra.com/screen-sharing-software/
https://www.g2.com/categories/screen-sharing
Using “alternative to ScreenHero” as a base
- ask HN (A huge amount of recommendations have completely shut down 😭)
- Product Hunt
- A guide to remote pair programming tools (they actually have a very similar set of criteria to us!)
- alternativeto
Computer World: 22 Free Screen Sharing Apps for Work at Home Collaboration

Recurring Issues

Click-and-drag on Mobile

A constant recurring issue I ran into when looking for suitable software was that many pieces of software that allow remote control don’t allow click-and-drag from a mobile touchscreen (particularly Android). This is a deal-breaker for educational uses, since many interactive games and applications (such as SmartBoard software) require that the student click and drag things across the screen.

I wrote up a summary of my research and conclusion as to why this feature is so often omitted - you can find it here. The short answer is “because it adds complexity” 🤷‍♂️.

Mobile PiP Support

For those not familiar, Picture in Picture, aka PiP, is a feature supported in both the iOS and Android operating systems that allows an application to continue displaying a video in small box overlaid on the screen, while you, the user, navigates to and uses a completely different application. Here is the demo from Google’s dev page on the topic, which just happens to be for a video calling app:

Here is the perplexing thing; despite PiP support on mobile having been a thing for a while already (Android: August 2017, iOS: ~2016 ), barely any video conferencing apps support it as of 2020.

The most damning current example I can give is Zoom; the company has a market cap of +$40 billion, and over 1,900 employees, and somehow hasn’t gotten around to supporting PiP. For example, on Android if you have the Zoom app open, and then switch to a different app (the web browser, for example), you can keep talking to and hearing the person on the other end, but you lose video.

The lack of PiP support in video conferencing apps is particularly annoying for our particular use-case, because we needed to combine a video chat app with another app for remote control, so PiP (or something like it, such as split-screen or floating windows) was a necessity.

And it’s not like supporting PiP in a video call app is impossible; it is the sample demo video provided by Google, and already implemented in these Android apps:

Whatsapp
FB messenger
Skype
Google Duo
Netflix
YouTube
Mixer
Etc…

As another, very positive example, the open-source video conferencing app Jitsi Meet supports Android PiP flawlessly.

Technical side note for Android: Some apps might use the SYSTEM_ALERT_WINDOW (aka “draw over other apps permission”) and generate floating windows instead of using the actual PiP API. Or they might not support PiP, but support split-screen. Any of these are acceptable to me, although PiP is definitely the preference.

Final Note:

I hope writing all this up helps others in their search for software! If you think there is a piece of software that I overlooked or missed, please leave a comment and let me know!

Prettier Function Parenthesis Spacing - Opinionated is Key

Joshua Tzucker — Fri, 03 Apr 2020 19:20:10 GMT

Intro

As I progress as a developer, I’ve tried to embrace code formatting tools more and more frequently. That hasn’t stopped my frustrations with them however, and I wanted to provide a short example from a very recent frustrating day that reminded me of their limitations.

The Issue

I had an existing codebase that was poorly formatted (🙃), so I was planning on using ESLint + Prettier (common combo) to clean it up, and enforce some standards going forward. Upon running Prettier, I noticed that it kept adding spaces between the function keyword and the argument parenthesis. For example (demo):

// Before ESLint + Prettier
const myFunc = function() {
    //...
}

// After formatting
const myFunc = function () {
    //...
}

For some reason it was bothering me, and, at least for this particular repo, I was hoping to be able to preserve my style of having no space before the parenthesis.

However, when I tried to go and actually change the formatting default, I ran into issues.

ESLint Rules

A quick search of my issue made it seem like this was because of a very specific ESLint rule: space-before-function-paren. Knowing this, I tried every variation I could think of to turn the rule off.

Examples:

// .eslintrc.json

// Standard approach
{
    "rules": {
        "space-before-function-paren": ["error", "never"]
    }
}

// Applying via plugin rule
{
    "rules": {
        "prettier": {
            "space-before-function-paren": ["error", "never"]
        }
    }
}

I also tried variations with “off”, etc.

Nothing seemed to work. This is because I rarely think about Prettier, and basically forgot the main premise of its functionality.

Prettier - An Opinionated Formatter

Here is a refresher on Prettier (or introduction, depending on your experience):

I’m going to paste two really important bullet points from the home page of Prettier, under the “What is Prettier?” section (emphasis mine):

An opinionated code formatter

Has few options

Basically, the main point of Prettier is to stop developers from wasting precious time fretting or arguing about formatting styles, hand-editing code to meet style requirements, and messing with complicated config steps. Using Prettier should be a quick set-and-forget step, and this is both its strength and its weakness.

The weakness is that there are very few ways to modify Prettiers’ options, and this rule is just one of many that you cannot override 🔒.

Space After Function - Prettier 2.0

I think part of why this issue might have stuck out to me is because, even though I’ve only used Prettier a handful of times in the past, I recognized that this was new behavior that I was not used to. Sure enough, after a bunch of digging, I found the exact discussions and PRs that brought this in as the new default style in Prettier.

This change was merged and released in the major 2.0 release, very recently in March 2020. You can find confirmation of this in the official release notes, here, or in the main merge PR, #3903.

🔎: For those interested, you can also find lively discussion on the decision on #1139, #3847, #3845.

Solution

Solution A: Don’t use Prettier:

This is harsh advice, but if you are the kind of person that cares strongly about this type of thing, enough to research how to override it, Prettier might not be for you. Again, the whole point of Prettier is to make those kinds of decisions for you, and they are strongly against exposing overrides as options, as that allows for more time wasted on manual configuration and bickering over how to set it up.

Plus, there are other auto-formatting options. For example, ESLint + AirBnB is a great common combination, where you get a bunch of common sense formatting defaults, but you can also configure the setup exactly how you like it using ESLint configs. In fact, just like how Prettier can auto-fix issues, so can ESLint.

Solution B: Force an Old Version of Prettier

Since this change was introduced with version 2.0 of Prettier, if you force your IDE to use an older version of Prettier, prior to the v2.0 release, you shouldn’t see this behavior. There are multiple ways to do this, but the easiest and most recommended way is to “pin” the version in your local package.json file.

For more details, assuming VSCode as your environment, refer to this documentation for “Prettier Resolution” in prettier-vscode.

Warning: Solution B is not recommended as a long term solution, since you will be missing out future (and post-2.0) performance improvements, added features, and security patches.

Solution C: Let ESLint override Prettier

Part of the reason why my attempts at overriding Prettier’s default with ESLint’s rule (space-before-function-paren) did not work, is because I was following Prettier’s recommended setup for ESLint, which includes telling ESLint to disable rules that conflict with Prettier. I’m guessing one of those “conflicting” rules is the space in function declarations (particularly anonymous function declarations).

This brings me to my second solution; let ESLint override Prettier.

This is pretty convoluted, and I’m not even 100% sure it would work, but I figured I would outline the steps anyways

First, you would need to stop telling ESLint to drop rules that conflict / override Prettier. For example, going from this (in .eslintrc.json):

{
    "extends": ["airbnb-base","plugin:prettier/recommended"],
    "rules": {
        //...
    }
}

… to this:

{
    "extends": ["airbnb-base"],
    "rules": {
        //...
    }
}

Next, you would need to setup your formatting workflow to call Prettier first (e.g., prettier --write .), to apply it’s opinionated styles, and then call ESLint’s formatter (e.g., eslint --fix). You could do this with chained command line commands, or, if you are using VSCode, via extensions and some custom tasks or something like that.

Solution B is not recommended, and is mostly included as an additional way to illustrate how ESLint and Prettier interact

This is the recommended ESLint extension for VSCode, which includes a built-in formatter

Wrap Up

Hope this helps someone! Feel free to leave a comment if you found an additional workaround or have any feedback to share!

Click and Drag Mobile Touchscreen Mouse Mapping

Joshua Tzucker — Mon, 30 Mar 2020 10:26:16 GMT

Intro
Research
Hackish Workaround (Zoom)

Intro

While trying to help someone find the perfect remote desktop / remote control / VNC / etc. app, I realized that many Android and iOS apps are missing a key feature when it comes to controlling a desktop mouse with your phone… click-and-drag!

To be clear, a click-and-drag operation would be when you left click your mouse, and then, while still holding down the left mouse button, you move your cursor to a new position, and once there, release it.

At first, this might not seem like a huge deal; there are many things you can do on a computer without needing to click-and-drag. However, when it comes to collaborative and creative tools, this becomes a nightmare as pretty much everything requires click-and-drag support! Annotating, drawing, moving virtual objects, rearranging slides, etc…

The more I thought about this, and came across more and more “big league” apps ( > million downloads) without click-and-drag, I started to doubt my own assumptions about mobile development. Is there a good reason why this feature is so often omitted from remote control apps? Is there an OS-level API blocker? Permissions issues?

I decided to do some digging…

Research

Initial Thoughts

My guess as to why this is so often omitted, is because mapping a touchscreen to a mouse input is slightly complicated by the fact that there are multiple ways to do it, and there is no perfect mapping system. There is no perfect mapping, because a mouse doesn’t act like a touchscreen. With a mouse, you can move from point A to point B without holding down any buttons. But to move from point A to point B on a touchscreen, without jumping in between, you have to hold down your finger to the screen while moving. How should this be interpreted?

If we map “user holds finger to screen and moves from A to B” to the mouse simply moving from A to B, then we cannot, at the same time, interpret it as a click and drag operation.
- This might be called “touch mouse” mode
- This is why many apps require a secondary indicator action from the user (long press, multi-finger tap, etc) as the start of a click-and-drag operation. But an equal, or greater amount of apps, seem to not bother spending the time to implement this at all, and just omit the click-and-drag ability altogether (looking at you, Zoom)
If we map “user holds finger to screen and moves from A to B” to the mouse clicking and dragging, without requiring that the user long press first or something like that, than we lose the ability for the user to move the mouse in a continuous path, without dragging at the same time.

Real-World Examples:

Apps Missing Click-and-Drag Support

Here are some examples of apps not supporting click-and-drag on mobile for remote mouse control:

https://crankwheel.com/introducing-crankwheel-remote-control/#limitations
- A viewer on a mobile device has slightly more limited control than a viewer on a desktop or laptop computer. They can press to simulate a left mouse-button click anywhere on the screen share, and they can perform keyboard entry by first bringing up the keyboard using a button provided in their viewing interface. Dragging and dropping, or dragging scrollbars, is not supported, nor are keyboard shortcuts such as Ctrl+B since there is no way of generating them on most mobile devices.
Zoom: there is no control to start click-and-drag
Many don’t allow for remote mouse control from a mobile touchscreen, period!
- GoToMeeting
- Microsoft Teams
- Blue Jeans
- …and many more (😢)

Different Mapping Options

Here are some examples of developers using some different mapping options:

Github thread on the topic: https://github.com/novnc/noVNC/issues/1267
RealVNC’s unique mapping model
- Double-tap to start click-and-drag
- Even middle-click is supported (three finger tap)
Android-VNC-Viewer allows for the user to switch between mapping modes!
- This is very unique; I wish every remote control app offered this!
- This is also a good way to show how complex mapping is
Google Chrome Remote Desktop: Android Police writeup
- Click and drag is also a bit confusing at first. Most apps use double-tap and drag, but not Chrome Remote Desktop. With this app you long-press until a ripple effect appears around the cursor. At that point you can begin dragging where you want.
TeamViewer: On Android, uses double-tap to start click and drag
Splashtop: Tablet and Mobile Phone Mappings

Web Complexity

An added “wrinkle” in developing a touchscreen-to-mouse mapping is that if your app is a webpage / wrapper around a web app, then you are going to be receiving both mouse and touch events, and you need to add logic to differentiate between and map appropriately.

For example, you might get all of the below:

pointerover -> pointerenter->pointerdown->touchstart->pointerup->pointerout->pointerleave->touchend
mouseover->mousemove->mousedown->mouseup->click

Related article (although dated): html5rocks.com - Touch And Mouse

Conclusion

It looks like, although this is a common issue, there are plenty of exceptions that make it clear that this is not an OS limitation of Android or iOS, but has more to do with how each individual app developer decided to implement the handling of touch events.

For example, Android-VNC-Viewer, one of the earlier VNC clients for Android, has a great breakdown in their wiki on different mapping modes that they make available to the user, which includes click-and-drag support.

So, the TLDR? Most likely this feature is omitted simply because of the added complexity. Eye-roll please.

Hackish Workaround (Zoom)

For Zoom, I built a quick workaround kludge, that intercepts when the Zoom Android client sends a right click (which happens on a “long press”) and turns it into the start of a click-and-drag operation. You can find it here: github.com/joshuatz/right-click-and-drag

Right Click and Drag - AutoHotkey Program

Joshua Tzucker — Thu, 26 Mar 2020 06:02:00 GMT

Utility program, built with AutoHotkey, to use the right mouse button to start a click and drag operation. Useful for certain remote control / VNC clients.

Using Windows Task Scheduler to Automate NodeJS Scripts

Joshua Tzucker — Thu, 20 Feb 2020 17:33:44 GMT

This is a post about using Windows Task Scheduler to automate the execution of NodeJS scripts, and other NPM / Yarn based tasks. If you don’t use Windows, this post is probably not for you (but feel free to read anyways 🤷‍♀️)

Why? 🤔

Tons of reasons! Maybe you are trying to emulate a production environment that has NodeJS scripts as scheduled CRON tasks. Or, for your own productivity or fun you want to script things to happen based on Window events.

For example, you could write a NodeJS script that talks to your project tracker of choice via API and stops any running timers when you lock your computer to take a break.

Why not just use `crontab` under WSL?

Good question! If you have WSL (Windows Subsystem for Linux) installed, and you only want to trigger actions based on time, then you should totally give crontab under WSL a shot!

Although there used to be issues with it (in past versions, WSL used to kill background tasks when you closed the console), I just gave it a shot, and had success. If there is interest, I might make a separate post about how to setup crontab under WSL.

However, Task Scheduler still has value as a separate tool, since more than just time can be used as a trigger; you can execute tasks based on computer unlocks, power events, and more. You can’t do that with crontab.

How? 🤓

Steps:

Find where the binary / application you need to run is stored
- You can use where npm or where yarn from the command line to find the path
  - Example: My yarn path is C:\Program Files (x86)\Yarn\bin\yarn.cmd
Open Task Scheduler (search in programs, or WIN+R, taskschd.msc)
Start the task creation process by clicking on “Create Basic Task” or “Create Task” in the sidebar
Pick a trigger
- “On a Schedule” (like CRON)
- “At log on”
- Etc.
Add your Action: Action -> Start a Program
- “Program/script”:
  - Here is where you plug in the path to the application you found in step 1
- “Add arguments” - You should put whatever you would put after npm or yarn normally.
  - If normally execute npm run myScheduledTask, you would want arguments to be run myScheduledTask
- If you are calling a scripts entry in a package.json file, you need to tell scheduler to run this where your package.json file is located.
  - If using Yarn, you can pass the working directory through args, with cwd.
  - Otherwise, use the start in (optional) field to specify the directory

If you are concerned about keeping track of the results of what ran, you can also capture the result of anything spit out to the console by using >> task_log.txt or something like that.

👩‍🍳 - You can combine actions and triggers

One nice feature of task scheduler that I did not notice immediately is that it does not have to be a 1:1 mapping of task-trigger-action.

For example, you can group ten different actions under a single task with a shared trigger.

✨ - You can use Git Bash for more advanced scripting

Instead of targeting NPM, Yarn, or Windows CMD, if you have Git Bash (comes packaged with Git for Windows), you can it as the target “Program/Script”, and then execute a more advanced command that uses some bash tools. For example, a sample task that performs some backups for a project might look like:

“Program/Script”: C:\Program Files\Git\git-bash.exe
“Add Arguments”: cd C:/projects/my-proj && node prep-dirs.js && npm run backup >> backup_log.txt

💥 - `%1 is not a valid Win32 application`

If you see this error, you have probably selected the wrong application as the Program/Script to execute. For example, using /yarn instead of yarn.cmd will result in this error.

⚙ - Stop the Cmd window from popping up

If the black windows Command prompt window keeps popping up whenever your task runs, you need to change one of the basic settings:

Change security options to: Run whether user is logged on or not
- 🔐You will probably also want to check Do not store password

There is no harm in having it show up; but it might get annoying if your task is scheduled to run frequently.

⏰ - How to schedule more frequently than every 5 minutes

You might have already noticed that the smallest interval that shows up in the repeat task every duration picker, under trigger settings, is 5 minutes. Uh-oh!

Actually, this is an easy fix - you can actually type a custom interval in that box! So if you wanted an entry that was equivalent to a CRON of * * * * * (every minute), just type in the box 1 minute and set for a duration of to Indefinitely.

Here is what that looks like:

Comparison with CRON

Since this is likely to come up in the comments (I can already hear the annoying response; ”why don’t you use a real operating system? lol!”) - yes, Task Scheduler is not a perfect replacement for CRON on windows. But it is not really meant to be, and this post is not advocating as such either.

Plus, you can use crontab under WSL now (see my note under “why?”).

Wrap Up

I hope this was helpful! This is a bit different from what I normally write about, but felt compelled to publish it as I had trouble finding existing resources on the topic.

Gatsby - Wrapping Markdown in HTML Divs with Custom Classes

Joshua Tzucker — Tue, 18 Feb 2020 18:49:35 GMT

Intro

For reasons I won’t go into, I wanted to build a Gatsby plugin which wraps elements / blocks of my choice in a div with a custom class name applied. I’m using Markdown as the source material for my Gatsby project, so here is the basics of how I wanted it to work - this…

<!-- CUSTOM_CLASS_WRAPPER="special-heading-style" -->
# Hello World

…would get transformed to this…

<div class="special-heading-style">
    <h1>Hello World</h1>
</div>

I also wanted it to work with Markdown components that generate multiple HTML elements. For example, this…

<!-- CUSTOM_CLASS_WRAPPER="special-list-style" -->
 - My list
	 - Sub Item

…would become

<div class="special-list-style">
    <ul>
        <li>
            <p>My List</p>
            <ul>
                <li><p>Sub Item</p></li>
            </ul>
        </li>
    </ul>
</div>

I thought this would be simple. After all, this is actually a tiny modification, and Gatsby is designed around extensibility and plugins. But things weren’t as simple as they seemed…

Background on Markdown Conversion

For the uninitiated, the majority of Gatsby projects that take Markdown as the source content, such as the go-to starter blog, under the hood use a plugin called gatsby-transformer-remark to convert the markdown to HTML. This, in turn, can accept hundreds of other plugins that extend it and modify how it handles the markdown conversion.

When you are writing a plugin, you basically are writing a callback function that gatsby-transformer-remark will call with, among other arguments, a Markdown AST (more on this later), which is a representation of what was extracted out of the markdown source. Your plugin can modify that content, by adding, deleting, or changing nodes in the AST.

The Problem

Going back to what we want to accomplish, it seems like writing our plugin should be straightforward. Receive the AST, look for our special HTML comment, and if found, wrap the first adjacent sibling in a new HTML node. Otherwise, do nothing. Here is a rough attempt:

const visit = require('unist-util-visit');

const plugin = (args, options) => {
	const markdownAST = args.markdownAST;
	const commentClassPatt = /^[ ]*<!--\s*ADD_CLASS="([^\r\n>"]*)"\s*-->[ ]*$/;
	
	visit(markdownAST, `html`, (node, index, parent) => {
		commentClassMatch = node.value.match(commentClassPatt);
		if (commentClassMatch) {
			const className = commentClassMatch[1];
			const siblings = parent.children || [];
			const nextSibling = siblings[index + 1];
			
			if (nextSibling) {
				visit(nextSibling, (node, index, parent) => {
					const wrapperNode = {
						type: 'html',
						value: `<div class="${className}"></div>`
					};
					wrapperNode.children = [
						{
							...node
						}
					];
					node = wrapperNode;
				});
			}
		}
	});
	return markdownAST;
}

module.exports = plugin

So, once again, our steps look like:

Take the Markdown AST spit out by the main transformer, and look for our special magic HTML comment
If found, check if it has an immediate adjacent sibling
If sibling is found, wrap with a <div>, with the desired class name
Replace the sibling with the wrapper, with itself as child

If I add breakpoints and logging output, I can see that my code is finding the right triggers and modifying nodes in place. However, this does not work! Why? Well, it comes down to a misunderstanding of how Gatsby, the Markdown AST, and HTML all fit together.

Finding the Solution

The missing piece of understanding here is twofold.

In a Markdown AST, you can have nested markdown (AST) nodes, but you can’t really have an HTML node that contains nested markdown
- You can see this yourself in the handy AST Explorer tool - sample
Using a Markdown node within an HTML node requires conversion first

So, re-evaluating things… if the AST node we want to wrap in a custom div…

…is markdown
- We first need to convert it to HTML, then wrap
…is HTML
- We could wrap it in place

BUT, either way, we must replace the node we want to wrap with a single HTML node, with the value as a string.

Markdown to HTML?

So far, so good. But how do we actually convert a snippet of markdown to HTML in Gatsby? This is where there is a bit of a gap in documentation, as most plugins are generating HTML from internal logic, rather than from markdown.

Option A) Use `remark` / `unified` library from scratch

If you peek inside the gatsby-transformer-remark plugin, you will see that Gatsby actually uses remark for markdown to HTML transformation, and that in turn is heavily powered by the unified suite of tools - there is a reason Gatsby is a gold sponsor of unified :)

There is nothing stopping you, as a plugin developer, from using these libraries directly. In fact, I would probably recommend it for situations like these; that way if some of the Gatsby internal APIs change, you are not forced to update.

Option B) Use methods from `gatsby-transformer-remark`

Rather than rolling our own code for converting a bit of markdown to HTML, shouldn’t we be able to just use transformer-remark itself? After all, that is the main point of the plugin; converting markdown to HTML.

At first glance, it seems like we are in luck; there is a method exposed in the plugin arguments - options.compiler.generateHTML, and if we look at the source for it, it says it takes a markdown node and returns html. PERFECT! Let’s try passing in a markdown node from visit() that we want to convert to HTML and then wrap…

UNHANDLED REJECTION Cannot read property ‘contentDigest’ of undefined

… now is the time when angry words are muttered.

OK, what now? Well, it turns out that we have mixed up some terminology. There are really two types of markdown nodes.

First, there is a markdown AST node - which is a node of the markdown AST generated from the markdown file
- This is what we just passed to generateHTML()
But, then there is a Gatsby markdown node, which is a special node specific to the Gatsby ecosystem, and has more to do with GraphQL and Gatsby internal than ASTs
- That is what the function is expecting
- You can see more about that data structure here

Further digging: Can we create a node to pass to `generateHTML()`?

Aha! Yes we can!

You can manually construct Gatsby nodes by hand - either by constructing the raw object by hand, or using actions.createNode - however, you need to have the string value of the node (raw markdown string) stored as node.internal.content, because that is what the method uses as input to remark.parse().

So, how do we get the raw markdown string back out of the MDAST node? We can use unified and their module remark-stringify:

- unified().user(remarkStringify).stringify(node)

Let’s say we have a MDAST node that we want to convert to a Gatsby MD node - here is how we could do so:

const unified = require('unified');
const remarkStringify = require('remark-stringify');

const plugin = (args, options) => {
	const { createContentDigest, createNodeId } = args
	// Assume we have MDAST `node` from unist-util-visit, or something similar
	const rawMarkdown = unified().use(remarkStringify).stringify(node);
	const gatsbyMdNode = {
		id: createNodeId(rawMarkdown),
		children: [],
		parent: null,
		fields: {},
		internal: {
			content: rawMarkdown,
			contentDigest: createContentDigest(rawMarkdown),
			owner: "gatsby-transformer-remark",
			type: "MarkdownRemark",
		},
	}
}

For the above, you could also use the actions.createNode utility method (and probably should…)

Now, we can pass that Gatsby MD node to the generateHTML function without the contentDigest error.

New Problem

The HTML generator introduces a new problem to be aware of… it is an async function! How are we supposed to this inside a Gatsby plugin for transform-remark?

Luckily, we have we have a great blog post by Huy Nguyen that discusses this exact issue.

Based on that post, we know that all we really have to do is have our plugin return a promise to Gatsby. There are a a bunch of ways you can do this; wrap the entire body in a promise and return it, write it as an async function, return Promise.all(), etc.

I like the async / await syntax, so I just made my entire plugin function an async function.

If you are looking for some resources are writing an async plugin, here are some helpful links I’ve found:

Other Issues:

Replacing an AST node in-place
- According to everything I could find, assigning my new node to the reference to the old one should replace it; however, I could not seem to get this to work
  - Replacing it in theparent.children array also seemed troublesome
  - The best working method I found was to leave the reference in place, but override the props with Object.assign(existingNode, replacementNode)
Irregular Gatsby types
- Be warned: it can be hard to find up-to-date type definitions for all the internals of Gatsby
- It helps to look through Gatsby’s source code, and existing TypeScript plugins

Solution - Final Working Version

After all that fuss and troubleshooting, the final code is actually pretty simple and short:

const visit = require("unist-util-visit");
const unified = require("unified");
const mdStringify = require("remark-stringify");

const plugin = async (args, options) => {
  const { createContentDigest, markdownAST, createNodeId } = args;
  const markToHtml = args.compiler.generateHTML;
  const transformPromises = [];

  const wrapMdAstNode = async (node, index, parent, className) => {
    let generatedNode = {};
    const wrap = htmlContent => {
      return {
        type: "html",
        value: `<div class="${className}">${htmlContent}</div>`,
        children: undefined
      };
    };
    // Process
    if (node.type === "html") {
      generatedNode = wrap(node.value);
    } else {
      const rawMarkdown = unified()
        .use(mdStringify)
        .stringify(node);
      const dummyNode = {
        id: createNodeId(rawMarkdown),
        children: [],
        parent: null,
        fields: {},
        internal: {
          content: rawMarkdown,
          contentDigest: createContentDigest(rawMarkdown),
          owner: "gatsby-transformer-remark",
          type: "MarkdownRemark"
        }
      };
      const html = await markToHtml(dummyNode);
      generatedNode = wrap(html);
    }
    Object.assign(node, generatedNode);
    return node;
  };

  const commentClassPatt = /^[ ]*<!--\s*ADD_CLASS="([^\r\n>"]*)"\s*-->[ ]*$/;

  visit(markdownAST, `html`, (node, index, parent) => {
    const commentClassMatch =
      typeof node.value === "string" && node.value.match(commentClassPatt);

    if (commentClassMatch) {
      const className = commentClassMatch[1];
      const siblings = parent.children || [];
      const nextSibling = siblings[index + 1];

      if (nextSibling) {
        transformPromises.push(
          wrapMdAstNode(nextSibling, index, parent, className)
        );
      }
    }
  });

  await Promise.all(transformPromises);
  return markdownAST;
};

module.exports = plugin;

5 Underrated Built-In VSCode Features

Joshua Tzucker — Thu, 02 Jan 2020 01:59:00 GMT

Intro

First, to get this out the way, this post is not a list of “top extensions you need to install”. Although those lists are helpful (albeit “clickbaity”), and I love that VSCode has an extremely “healthy” ecosystem of extensions and plugins, I noticed very few posts about the amazing stuff that is built-in to VSCode from the get-go, and that is what I want to talk about.

These are features of VSCode that are baked into the editor (no extensions or tooling to install), but I personally don’t think are getting enough attention and could use a PSA.

#1: JavaScript Type Safety

Although the rest of the list is not really ordered, this definitely deserves a #1 spot, by a large margin. I’m not going to shove TypeScript or type-safe JS (what?!) down your throat, but I’d like to just point out how VSCode has a feature that allows you to get some of the benefits of type-safety, without needing to switch to TS files, transpiling / compiling, or requiring that other devs use TypeScript.

This is a wonderful half-way solution for those looking to introduce some type-safety into their JS code immediately, with minimal changes and setup.

I don’t want to go too in-depth on this, as there are multiple guides on this out there, but I’ll cover some “quickstart” basics:

Enabling JS Type Checking

There are multiple ways to enable type checking in JS files, but the easiest if you are looking to just try it out, is to add // @ts-check to the top of your file.

See the “Type Checking JavaScript” section of VSCode’s “Working with JavaScript” docs for more details.

What does this get you?

If you don’t want to do any extra work (such as annotating types or setting up configs), simply adding // @ts-check is a way to instantly get TypeScript powered type checking in VSCode.

For example, checkout this screenshot comparing a JS file without and with TS-checking on:

In the above example, the developer has forgotten that querySelectorAll does not return an array, it returns something special - a NodeList - which does not have the built in map() method that arrays have. With type checking on, the fatal error triggered by the use of map is caught in VSCode and flagged.

For those wondering, an easy way to convert from a NodeList to an array is to use Array.from: Array.from(document.querySelectorAll('a[href]'))

Annotating Types with JSDoc and Advanced Usage

The “power-user” part of this feature is the ability to use JSDoc comments to tell VSCode’s TypeScript powered intellisense system about advanced types that are within your JS code. This doc - “Type Checking JavaScript Files” goes into depth on it, but for brevity, I’ll just paste a cool example:

For even more advanced usage, there are some great resources out there, starting with my VSCode cheatsheet which has a whole section devoted to JSDoc powered type checking and intellisense 🙂. It also has a list of links to other helpful related resources.

#2: Making things easier for contributors

Spending time to streamline the process for other people to contribute code to your codebase might feel like a chore, but in the long-run it makes things better for everyone. The easier and more “mistake-proof” the process is for devs to contribute, the less likely it is that you will have to field duplicate questions, reject PRs for violating guidelines, and deal with unnecessary back and forth.

What you might be surprised to learn is that VSCode has some built-in features that help with this goal.

Recommended Extensions

One of them I actually learned about when I poked through Dev.to’s own source code, as it was the first time I had seen it used - “Recommended Extensions”. The way it works is that you add a special file - .vscode/extensions.json - to your project root, with a list of extensions that you want all contributors to be using (for example, Prettier), and VSCode will pick up on that and popup a recommendation to Devs to install them when they clone and open your repo.

You can find more details on how to use this in VSCode’s workspace recommended extensions documentation.

Workspace tools and settings

Built-in to VSCode are multiple places where you can override user settings and craft per project configurations, both of which can make it easier for contributors to work in your codebase with VSCode.

For example, if you manage a mono-repo with distinct parts, you might want to take a look at “multi-root workspaces”.

For overriding settings, such as default indent and spacing options, you can commit .vscode/settings.json to the root of your repo with the overrides, but be respectful of your contributors and don’t override anything that you want them to be able to change.

#3: User-Defined Snippets

Although VSCode does not yet support advanced macros (without the use of an extension that is), did you know that it does already support advanced snippets? At first glance, these might seem like they can only insert basics blocks of text, but you can actually do some advanced things with them.

For example, here is a snippet that I quickly whipped up to convert a markdown style bullet list into Github styled checkbox list:

{
	"MD List to GH Checkbox": {
		"scope": "markdown",
		"prefix": "list-to-checkbox",
		"body": [
			"${TM_SELECTED_TEXT/^([\t ]*-) ?(.*)$/$1 [ ] $2/gm}"
		],
		"description": "Transform MD list to MD checkbox list"
	}
}

Here it is working in action:

The main docs page for creating custom snippets is here.

#4: Reordering Imports Automatically

First, I would like to thank Ryan Chenkie, for this tweet that brought this to my attention:

For Windows users, that key combo is SHIFT + ALT + O, by default.

As a reply to that tweet pointed out, you can also have VSCode automatically do this, by configuring it in your settings.json file, like so:

{
	"editor.codeActionsOnSave": {
		"source.organizeImports": true
	}
}

#5: Ability to build a custom extension for yourself or your company

One of the best things about VSCode is how extensible it is, and how easy it is to build and test new extensions. Often when this is discussed, the focus is on really flashy and/or impressive extensions, but I want to remind everyone that extensions that solve a singular niche problem can be just as important, especially within certain organizations.

Within the discussion of “tooling”, you should think if there is a custom extension that could be built for your company (or just yourself) that could improve your developer workflow, help convert legacy code, and/or interop between processes that are specific to your work. It might be easier than you think to build one, and as a bonus, is sure to impress others.

Wrap Up

I hope you found at least one of these items helpful! Thank you to the VSCode team and open-source contributors for building a revolutionary code editor!

Using Google Cloud Functions Permanent URL to Proxy Ngrok Requests

Joshua Tzucker — Thu, 28 Nov 2019 20:12:37 GMT

Background - Why might you need a reverse proxy?

Web development tooling has come a long way, and most frameworks and SDKs have made previewing your app as you work a breeze; often it is as simple as running something like npm start from your CLI. But an unexpected hurdle that many web developers hit, and can seem much more complicated, is “how do I use this outside of my local network?”. Things like screen-sharing, screen recording, and VSCode’s Live Share can solve this issue if all you need to do is show your localhost demo to your remote coworker. But if you need to actually expose your code to the outside world, those tools fall short.

For example, a very common scenario is trying to test a third-party service (Slack, Github, Twilio, etc.) that talks to your codebase, maybe through a webhook. In this scenario, you can’t run the third party code locally (e.g. running all of Github’s platform on your computer), and if you put something like “localhost:3001/webhook” as the webhook, that third party is going to be unable to reach the code running inside your local network (for good reason!).

How do we solve this?

Reverse-Proxies

The common solution to this issue is a reverse proxy combined with ssh tunneling. You have a public URL, which points to a public server, which uses a reverse proxy to hand-off the request to a different port and/or server, which in turn gets proxied / tunneled via a SSH tunnel between the server and our local development computer. This is much easier to show visually, so here is a simplistic diagram showing this:

There is a lot that goes into this solution. You need to run a public server, set up SSH support on it, have a domain name, DNS entries, SSL certificates; the list goes on and on. It’s not all that complicated (here is a simple guide), but it is also not free or quick.

Quick, Simple, and *Free - Ngrok

Given the complexity of setting up a full reverse-proxy through SSH pipeline, services have sprung up to simplify this for developers.

One of the most popular and easiest to use is Ngrok. With a single command, you can instantly get a public URL that you can use to access your localhost from anywhere - no public server or complicated setup necessary!

For example, to relay HTTP requests from outside your home network to your localhost on port 3001, you could use ngrok http 3001 and be up and running in seconds!

Ngrok basically takes the place of both the SSH tunnel setup (with its CLI client), and the public server with a reverse proxy - the second and third blocks in my diagram up above. It even comes with built-in tools, such as a dashboard, request inspector, and replay functionality.

Ngrok is also recommended by several well-known third-party platforms (Twilio, Github).

I am not affiliated with Ngrok in any way; just think they have an impressive service.

The downside to Ngrok

Although Ngrok has a very generous free tier, one major restriction is that the public URLs that are generated are both random and not assigned - meaning that each time you start it up, you will get a different random subdomain, like https://df12f52a.ngrok.io. If you are trying to develop webhooks or something similar, this means that each time you need to restart Ngrok, you would have to login into your third-party platform, find the admin dashboard, and copy and paste your new Ngrok URL into the GUI. Or, if you are sharing URLs with coworkers, you would need to keep sending them updated URLs.

Ngrok does offer custom subdomains or personalized domains, but these come under the paid option, which starts at $5 a month. Not much, but this is basically the same price as running Ngrok or Nginx reverse proxy yourself on a paid server, like Digital Ocean.

My Solution - Add Another Proxy with Cloud Functions!

This is the kind of solution I am simultaneously both proud and ashamed of. Because I am cheap, and didn’t feel like buying a paid Ngrok plan just to test webhooks a few times a year, or setting up a dedicated server for it, I started thinking of how I might use free, or very low cost workarounds. What I came up with is kind of silly, but it works - just add another proxy, with a fixed URL, in front of the changing Ngrok URL!

Let me elaborate. As I researched options, I noticed that cloud functions (Google Cloud Functions or AWS Lambda) were often the cheapest to use, in that they start at a scale of zero by design (they only run when requested). At first I was thinking of replacing Ngrok with them entirely, but this quickly was squashed; they are designed to execute quickly and exit, and would not work for maintaining a SSH tunnel connnection.

However, I realized two important things; cloud functions often get a semi-permanent PUBLIC url to invoke them, and they can handle both incoming and outbound requests. With this in mind, I realized I could setup a cloud function with a public URL, that simply proxies inbound requests to a re-configurable Ngrok address. Here is what that looks like:

In this scenario, my Cloud Function public URL stays static, even as the Ngrok address it points to changes!

The only caveat here is that the cloud function has to be redeployed each time that the Ngrok URL changes. However, this is very easy to automate; in fact, I already have a command set up in my package.json, so if my Ngrok URL has changed, all I need to do to redeploy with the update is type npm run ngrok-deploy.

GCP

I ended up going with Google Cloud Platform to provide the serverless function that proxies requests. Their free tier includes 2 million (!!!) executions per month, plus 5GB of network traffic. This is WAY more than enough for some simple webhook testing.

You can find all my code here, which proxies inbound requests to a configurable destination: github.com/joshuatz/gcp-proxy-func.

Another cheap alternative

Besides the obvious routes of paying for a pro Ngrok plan or paying for a dedicated server to run your own reverse-proxy, there are some other alternatives I have yet to explore.

One option could be to wrap a reverse proxy in a container, and deploy it on a platform that supports “scale to zero” container scaling, and have it “wake up” on an SSH connection. Then, you would only have to pay for it while you are actively connected via the SSH tunnel and using it for developing.

Google Cloud Functions HTTP Proxy Relay with http-proxy-middleware

Joshua Tzucker — Sun, 24 Nov 2019 10:35:17 GMT

Quick post: I just want to say that I was (pleasantly) surprised to find how easy it was to setup and use a NodeJS powered HTTP request proxy, with Google Cloud Functions and "http-proxy-middleware" for Express JS. For example, here is a bare-bones setup:

Index.js:

const express = require('express');
const proxy = require('http-proxy-middleware');
const app = express();
app.all(’*’, getProxy());
function getProxy() {
return proxy({
target: ‘https://postb.in/123-456’,
changeOrigin: true,
followRedirects: true,
secure: true
});
}
module.exports = {
proxy: app
}

Package.json:

{
	"name": "gcp-proxy-func",
	"version": "1.0.0",
	"description": "Run http-proxy-middleware to relay requests",
	"main": "index.js",
	"scripts": {
		"deploy": "gcloud functions deploy proxy --runtime nodejs8 --trigger-http --memory=128 --timeout=60s"
	},
	"dependencies": {
		"express": "^4.17.1",
		"http-proxy-middleware": "^0.20.0"
	}
}

Deploy command:

gcloud functions deploy proxy --runtime nodejs8 --trigger-http

Note:

Although this didn't quite end up fulfilling the need that I had for a project, I thought it was neat how fast the function could spin up, respond to a request, proxy it, and respond.

Docker-Compose: Relative Env Files with Variable Substitution

Joshua Tzucker — Sat, 16 Nov 2019 02:36:06 GMT

One of the neat features of the Docker-Compose file syntax is that it allows for variable substitution, where you can put in a placeholder that will get filled in from your .env file. This is both useful and often necessary because you need to keep your docker-compose.yml checked into GIT / version-control, but you don’t want to store passwords directly in the code. Variable substitution lets you do exactly that:

docker-compose.yml:

version: '3.1'
services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_PASSWORD: ${PASSWORD}

.env (not checked into version control):

PASSWORD='abc123def456'

Problem: Docker-Compose does not read .env files outside of the working directory

Here is the issue: all of the above works fine, except if your .env file lives outside of the directory where the docker-compose.yml file resides. For example, let’s pretend that our scenario is that we have a .env file in the project root that we want to share with multiple compose files in subdirectories. Like this:

Project Root:

.env (contains variables)
.gitignore
/backend
- docker-compose.yml (consumes variables via substitution)
/frontend
- docker-compose.yml (consumes variables via substitution)

Now, ignoring that there might be ways to optimize our docker setup here and consolidate, how can we get both docker-compose files to use the top level .env file?

Best Option (if it works for you) - Official CLI Arguments

This issue / feature has been requested a lot from users of Docker, and it looks like the Docker dev team has taken a few passes at implementing the ability to use variable substitution with a env file outside of the working directory.

project-directory

One way that this is supposed to now be possible is through the --project-directory argument to docker-compose. The way it is supposed to work, is that you can use it to “specify an alternate working directory”, by passing in a PATH string (CLI ref page here). For example, if I wanted to start the backend while using the parent project directory, theoretically this should work:

cd backend
docker-compose --project-directory ./../ up

However, you can find many instances (1, 2, 3) where users are saying this straight up does not work, and I would have to agree from my personal experience. And the official response is essentially “yes, this is usually broken”, despite the argument staying in the docs since Compose 3.2 (PR here).

env-file

Another option that is supposed to be viable for this purpose is the --env-file argument. However, like the env_file option, this appears to actually only work for passing environment values directly to the container, and does not work for docker-compose variable substitution - in fact, it appears to only work with the run commands, which would support this hypothesis. This is despite the fact that an issue requesting that --env-file be useable with the compose commands was closed - you can see comments on the relevant pull request that show it still not working (this is my experience as well). If you can get this to work, this would probably be the cleanest solution to this issue.

Workarounds

Since neither of the above “official” approaches actually work for me, and many other users, I want to cover some common workarounds that do work. Albeit, not in the most optimal solutions.

Execute commands where the `.env` is located

Rather running docker-compose where the YML file is located and getting docker to pull in a .env file in a different directory, you can actually do the reverse; execute the command where the .env file is located, and tell docker the location of a compose file in a different directory. We can do this with the -f or --file option.

Going back to our original example, here is the example file structure:

.env (contains variables)
.gitignore
/backend
- docker-compose.yml (consumes variables)
/frontend
- docker-compose.yml (consumes variables)

And here is how we could start up both docker configs.

# In the project root dir
docker-compose -f ./backend/docker-compose.yml up -d
docker-compose -f ./frontend/docker-compose.yml up -d

Symlink the .env file in

This is a really common solution to this issue, and a good solution to remember in general when programs require that files be in the same directory. Instead of copying the file to the directory as a part of a build script, we can create a symbolic link, which we only have to create once, and will make docker think that the .env file really is in the same folder as the compose file.

For our example scenario, I just have to create the two symlinks in a one time process:

(Bash example):

ln -s ../.env backend/.env
ln -s ../.env frontend/.env

Although version control, like GIT, can actually handle storing symlinks, if you are not comfortable with this, you could always have the symlinks generated with a one-time script.

Copy the env file around

If you are opposed to using symlinks, you could simply copy the top level .env file around to various spots, using some standard shell scripting. I find this method a little messy and less optimal.

Simply passing env variable values from host to container

If you don’t care about variable substitution, and all you’re looking to do is simply pass a bunch of values through to the container, then most of this post is extra information that you don’t need. Passing environment values to the container itself is pretty simple, and has multiple easy-to-use options:

Full details
- You can use the env_file option to pass an entire .env file contents to the container
- Put under environment: but leave off value
- Pass via docker-compose run -e {VAR}={VAL} {imageName}

Chai Testing - Expecting Array To Not Include Members

Joshua Tzucker — Mon, 28 Oct 2019 05:50:10 GMT

One of the problems with "natural language" testing frameworks, or as I like to think of them, "semantic testing" frameworks, is that language is inherently ambiguous. This can lead to problematic instances where the developer (myself) thinks that, just because a test is written as an English sentence ("expect variable alpha to be a string equal to 'abc' and have length of 3"), my interpretation of how this is tested is the same as the frameworks. Often, it is not.

On to specifics: I hit this issue head on when I couldn't understand why Chai was letting an array pass a test that it should not have been.

Chai - Making sure an Array does NOT include members

Click here to jump right to the solution, or read on for background and what does *not* work.

Problem and attempt:

Let's start with the result we want. We want to ensure that {inputArr} (type of array), does not contain any of {blockArr} (type of array).

const blockArr = ['&', '@', '!'];

One of Chai's handy chaining functions is the "not" method. Per the docs, this "negates all assertions that follow in the chain". If we want to make sure that an array does not contain of the block elements, this seems like a good starting spot. So, we start with expect(inputArr).to.not

Next, lets combine this with the ".members()" method, which specifies array or object members to check against. So now we have expect(inputArr).to.not.include.members(blockArr). However, this will only fail if inputArr has every element of blockArr in it, so let's use the ".any" modifier to specify that inputArr should not have any element of blockArr in it.

Here is what we ended up with:

expect(inputArr).to.not.include.any.members(blockArr);

That looks good, right? It should work? WRONG! It does not work! This is why I mentioned language ambiguity. This reads as a valid sentence, and uses valid methods from Chai, so why is it not working?

Well, the answer comes in reading more into the docs and realizing that there is a bad combination of requirements.

We have to use any in the chain, because otherwise methods like include or contains expect that inputArr is a superset of blockArr. In practicality, this means expect(['a', 'b', 'c']).to.not.include.members(['a', 'b']) will successfully fail, but if we change it to expect(['a', 'b', 'c']).to.not.include.members(['a', 'e']), it lets the test pass instead of failing it.
1. This goes back to ambiguity. If I asked you "does the word DOG not contain the letters DB", you might answer "Yes, it does not contain those letters", or you might answer "No, it contains the letter D, but not the letter B"
We are required by point 1 to use any, but Chai only applies this rule to keys, not to members!

Because any only works on keys, including it with negated members actually has zero effect.

Solution: Workaround to use any with array member values

First, the way that Chai seems to recommend is to just keep chaining more precise rules together, like so:

expect(inputArr).to.not.include('&').and.not.include('@').and.not.include('!');

But this starts to get ridiculous with long lists of blocking elements, and is not usable with dynamic tests. Here is a solution that keeps things simple and lets you check that inputArr does not contain ANY element from blockArr:

expect(inputArr.reduce((running, curr) => {
	running[curr] = curr;
	return running;
}, {})).to.not.include.any.keys(blockArr);

That's it! All I've done is transformed inputArr from an array, to an object where the keys are the values in the array. This lets us use the any chain method with keys, so we don't have any issues like we did with members.

Keep in mind that this removes duplicates, but that shouldn't really matter if you are trying to check that an array does not contain elements.

Another option is to just iterate over the input array and check for blocked elements, then store the result in a boolean, and then check the value of the boolean, like so:

const foundBlocked = inputArr.reduce((blocked, current) => {
	return blocked || blockArr.indexOf(current) !== -1;
}, false);
expect(foundBlocked).to.equal(false);

Proof and demo:

If you want to see how this all works, and maybe play with the results yourself, check out this Repl I set up - https://repl.it/@joshuatz/Chai-Array-Does-Not-Include-Any-Members

Reddit APIs and Manually Building Reddit Embeds

Joshua Tzucker — Wed, 09 Oct 2019 19:02:01 GMT

[Click here to jump right to the Reddit APIs documentation]

First, a warning. Putting together your own dynamic embeds based on Reddit data, rather than using a service, is not a choice that lends itself to an easy or robust solution. Reddit often retires public endpoints, and there can be mixed documentation about those that do exist, which is part of why I am writing this post.

An alternative to rolling your own solution:

First, before getting into a custom solution, I'll just point out that Embedly, which is a service that lets you dynamically embed external content on your site, is an official partner of Reddit. Why does this matter? Well, if Reddit makes a change to their API, you better believe they are going to work with Embedly to make sure their embeds keep working. Some random dev that doesn't pay to use their API? Not so much.

If you hit the "share" button below a Reddit post, and select "embed" as the option, the embed code it generates for you to use is actually Embedly code! Or, you can generate the same code using Embedly's generator.

Embedly also has hundreds of providers other than Reddit. If you are looking to add other external content to your site or app, think about whether it is worth the time to build out a custom connector for each source, rather than one single integration with Embedly.

I'm not an employee of Embedly or even a user of their service, but just want to make it clear what your options are if you don't want to reinvent the wheel. Now, for those that do, or are looking for general information on Reddit APIs, read on.

Reddit API Endpoints, Notes, and Examples:

Main docs: https://www.reddit.com/dev/api/

Not advised:

https://www.reddit.com/oembed?url={reddit_post_or_comment_link}
- Example request: Link
- The problem with this endpoint, is that it does not return a proper CORs header
  - This means the request get blocked in all major browsers
  - This might not be a problem for you if you are building a desktop application or other non-web app
- The information returned from this is also pretty basic

Good:

The ".json" trick - e.g. {reddit_url}.json
- Example: Link
- This is actually an insanely cool trick. Currently, you can take pretty much any Reddit permalink URL (post, comment, even boards) and just add .json to the end of the URL to get back JSON formatted data!
- This approach even returns a wildcard access-control-allow-origin header, so you can easily use it with AJAX and not have CORs issues!
- Downside: The data that is returned from each URL is not really filtered...
  - This is bad if you simply want to get the title of a post and its preview image, as you add unnecessary latency and bandwidth use
  - However, this is good if you are looking to get tons of data at once (for example, to generate a preview of a popular thread).
- You can limit the number of returned children elements by using the limit={#} querystring parameter
- A benefit to this method is that you don't need to know the type (type prefix) of the object beforehand
Official /info API endpoint
- Example: Link
- This is probably the best API endpoint to use in terms of being documented, likely to be supported for a long time, and returns brief info
- Actually documented, here

Parsing API JSON data

Both the .json trick and /info API endpoints return JSON data.

For .json, it is returned as an array with two elements (objects).
- The first element is the "thing" itself (post, comment, etc.)
- The second element is the children of the thing
For /info, it is returned a single object, of the thing itself
Each element contains two sub-objects: data and kind (type of thing)
- datacontains a children array, which can contain many sub-elements

To get the title of a post / article with the .json trick, you would use `jsonData[0].data.children[0].data.title`, since the post itself is the first child of the first element (itself).

Image 403 Forbidden Error from API:

If you grabbed an image URL off the API response and tried to use it, you might have ran into a "Error 403 Forbidden" message and/or HTTP error status. The most likely culprit is that the URL got messed up due to encoding / escaping.

Whether you did or did not use raw_json=1 option with your API request, you simply need to decode some encoded HTML entities within the image URL. With JavaScript, you can use this code to decode it.

If you are not using JS, search for "decoding HTML entities" + your programming language to see if there is a library or pattern you can use. Or simply do a string search and replace of "&" with "&", since that is the usual culprit.

For embeds only:

If all you need to do is embed a single comment or comment chain in a website, there is a quick trick you can use, no API required:
- `redditmedia.com/r/{subred}/{article_id}/{article_name}/{comment_id}/?embed=true`
- Example: Link
- You can stick this directly in an iFrame, no API parsing required.
- It appears it uses the same querystring params as the options listed for the Comments API endpoint - here
I can't find a corresponding trick to generate an embed URL for regular posts instead of comments :(

Reddit IDs / ID36 / T_

One of the things you might see glossed over through API documentation and various help threads is references to a ID36 string, or t_ Reddit ID. What is this? Well, if you look closely at pretty much any Reddit URL, you will start to see a pattern:

reddit.com/r/food/comments/60jxzo/homemade_breakfast_sugar_cookies/df71am8/

See the highlighted parts? Those are Reddit IDs, specifically ID36, or base36 strings. Why the "36"? Well, the IDs will always contain only lowercase letters (26 characters in the alphabet) and/or digits (0-9, which is 10 choices) --> 26 letters + 10 digits = 36 choices!

Type prefix (t_) and fullnames

A "fullname" in terms of the Reddit API is the type prefix (t1_, t2_, etc.), plus the ID36. So, an example would be "t1_df71am8", which corresponds to a comment (type t1) with an ID of "df71am8". You can find the table of type prefixes here and more info about fullnames, here.

Building a Custom Embed - Demo:

To put together some of the API basics I've covered in this post, and demonstrate how you might use them, I've thrown together a quick demo. This is just a little bit of JS + HTML + CSS, that dynamically generates a Reddit post summary card, based on an input URL. It uses the /info API endpoint, and JavaScript's fetch() web API. You can press the "demo" button to test it with the example Reddit thread used throughout this post.

Music Meta DOM Scraper - Simple Meta Info Grabber

Joshua Tzucker — Mon, 30 Sep 2019 17:19:39 GMT

Source code / Repo:

https://github.com/joshuatz/music-meta-dom-scraper

Description:

There is not a whole lot to this project, and I don't really have plans to expand on its capabilities, but given how much use I get out of it, I thought it warranted a quick write-up and project page.

The "Music Meta DOM Scraper" is just a bit of Javascript that lets me quickly grab song meta information (track title, artist, release date, etc.) from a few different sources. Right now, those are just Bing search results and AllMusic discography pages.

It does not use any APIs or offer bulk functionality beyond scraping an album at a time; its main purpose is to just clean up the data on the webpage, normalize it a little, and make it easy to copy into my clipboard, so I can then paste it into my catalog tracker of choice (Google Sheets, SQL table, etc.).

Right now, I have it built as a simple bookmarklet, which you can install below:

Bookmarklet installer:

NodeJS: Injecting Values, Sharing Globals, and General Mischief

Joshua Tzucker — Fri, 27 Sep 2019 08:48:46 GMT

Warning: A lot of things discussed in this post are “hackish” and not best practice. However, there are some one-off scenarios where they are helpful, and exploring their use gives us some insight into how NodeJS works.

Example Scenario: Executing a file with side-effects from another

Here is the setup. We have file as-is.js, which for some reason, we cannot edit to add imports. Maybe it is third-party vendor code, maybe we need to use it later in an environment that does not allow imports, or maybe we need to execute it with special imported values only for a one off test.

Example “as-is.js”:

var mode = '';
if (window.location.host === 'localhost'){
    mode = 'debug mode';
}
else {
    mode = 'production mode';
}
console.log(mode);

Now, let’s say that we want to do multiple things:

Inject a fake value for “window.location.host”
Override “console.log” with our own method that redirects the output to a file
Capture the value of “mode”

We will do all things in a file called “with.js”:

const fs = require('fs');

// Inject window global
var window = {
    location: {
        host: 'localhost'
    }
};

// Mutate console.log method
console.log = function(input){
    input = input + '\n';
    fs.writeFileSync(__dirname + '/output.txt',input,{
        flag: 'a'
    })
};

// Run as-is.js
require(__dirname + '/as-is.js');

// Try to read captured `mode` variable
console.log(`Mode captured from "as-is.js" is ${mode}`);

Right now, that file doesn’t achieve our goals and doesn’t work.

Here are some options for implementation

Polluting the global variable scope

In NodeJS, one way to share variables between file is to make them global variables.

One way to do this is to completely remove the variable declaration keyword. If I change “var foo = 'bar';” to “foo = 'bar';”, I have now made “foo” a global variable.

You can also use “global.foo = 'bar'” as another way to declare a global in node.

Here is our modified “with.js”:

const fs = require('fs');

// Inject window global
window = {
    location: {
        host: 'localhost'
    }
};

// Mutate console.log method
console.log = function(input){
    input = input + '\n';
    fs.writeFileSync(__dirname + '/output.txt',input,{
        flag: 'a'
    })
};

// Run as-is.js
require(__dirname + '/as-is.js');

// Try to read captured `mode` variable
console.log(`Mode captured from "as-is.js" is ${mode}`);

The only change that has been made is “var window = ...” to “window = ...”, making it a global variable. Note that this change does not need to be made to “console”, since it is already a global.

However, there is still an issue with this code. The “mode” variable is not a global in “as-is.js”, so it can’t be read by “with.js”! The last line will of “with.js” fails!

Eval the file contents

Since we can’t touch “as-is.js” to change “var mode = ...” to “mode = ...” and make it a global, we need some other way to capture its value. A quick hack way to do this is to use “eval()”. By default, in non-strict-mode, eval runs code in the same lexical scope as where it is called, so variables can be shared across.

Modified “with.js”:

const fs = require('fs');

// Inject window global
var window = {
    location: {
        host: 'localhost'
    }
};

// Mutate console.log method
console.log = function(input){
    input = input + '\n';
    fs.writeFileSync(__dirname + '/output.txt',input,{
        flag: 'a'
    })
};

// Run as-is.js
const rawAsIsCode = fs.readFileSync(__dirname + '/as-is.js').toString();
eval(rawAsIsCode);

// Try to read captured `mode` variable
console.log(`Mode captured from "as-is.js" is ${mode}`);

Notice that we were able to change “window = ...” back to “var window = ...”, and it was still able to share its value with “as-is.js” due to the scope sharing of “eval()”.

Note: The value of “mode” still can’t be read, if the eval’ed code contains the “use strict” directive; in that case, “as-is.js” would get its own lexical environment, which is to say, its own scope.

Use NodeJS VMs

Something that is unique to Node is the vm module API.

With this exposed API, we can create mini sandboxes with specific context, to run code in. This is pretty similar to “eval()”, but offers a little more isolation and control.

Modified “with.js”:

const fs = require('fs');
const vm = require('vm');

// Inject window global
var window = {
    location: {
        host: 'localhost'
    }
};

// Mutate console.log method
console.log = function(input){
    input = input + '\n';
    fs.writeFileSync(__dirname + '/output.txt',input,{
        flag: 'a'
    });
};

// Run as-is.js
const rawAsIsCode = fs.readFileSync(__dirname + '/as-is.js').toString();
// Create scoped context with shared vars
var mode;
const vmContext = vm.createContext({
    console,
    window,
    mode
});
// Run code in context
vm.runInContext(rawAsIsCode,vmContext);

// Try to read captured `mode` variable
console.log(`Mode captured from "as-is.js" is ${vmContext.mode}`);

There are a few things important things to note about this approach:

The explicit context setup of the vm allows us to capture the value of “mode”, even if “as-is.js” is in “strict mode”
Like “eval()”, all the vm run methods expect code as a string.
Variables have to be explicitly shared via “context”

File concatenation

This is kind of a silly solution, but it works! We will basically pretend that the two files are one, and merge the code together with string concatenation.

CLI Solution

The only special part of this solution is that we need to move our final reading of the “mode” value to the end, by placing it in a separate file and concatenating it last. So the values of the files as they are now are:

as-is.js:

'use strict';
var mode = '';
if (window.location.host === 'localhost'){
    mode = 'debug mode';
}
else {
    mode = 'production mode';
}
console.log('Mode set');

with.js:

const fs = require('fs');
const vm = require('vm');

// Inject window global
var window = {
    location: {
        host: 'localhost'
    }
};

// Mutate console.log method
console.log = function(input){
    input = input + '\n';
    fs.writeFileSync(__dirname + '/output.txt',input,{
        flag: 'a'
    })
};

with-final.js:

// Try to read captured `mode` variable
console.log(`Mode captured from "as-is.js" is ${mode}`);

And finally, we can concatenate the code from these files and run them from the command line with a single line:

node -e "eval(fs.readFileSync('./with.js').toString() + fs.readFileSync('./as-is.js').toString() + fs.readFileSync('./with-final.js').toString());"

Wrap-up

These are just some fun examples of messing with NodeJS scope and context; they are far from the only options. Furthermore, depending on what you are looking to do (testing, module mocking, etc), there is likely a library or existing solution to help you out.

For example, if you are looking to mock the global window object, since that is a browser API and does not exist in NodeJS, please take a look at jsdom. Or for mocking modules, mock-require.

Finally, if you have the power to actually edit the files you need to share values with, then obviously you should use actual established methods for doing so, such as the module system of exports and imports.

Unix/Linux File Creation Stamps (aka birthtime), and NodeJS

Joshua Tzucker — Mon, 23 Sep 2019 09:01:06 GMT

This learning experience started off because of a test failing on Travis-CI.

This test used NodeJS "fs" ("filesystem"), or more accurately, "fs-extra", which extends the "fs" base. It simply created some files, waited a few seconds, modified one of the files to update its modification timestamp (mtime), and then checked to make sure that FS showed that the modification time minus the creation time was the span of the delay. This test passed with no issues on Node v12 and Node v10, but was failing 100% of the time on Node v8. Another very important fact (and clue to the real issue) was that Travis-CI uses Linux, whereas on my own PC, which runs Windows, the test passed on all versions, including Node v8.

The exact issue:

After adding some logs to my test, it became very clear what was causing the test to fail:

Diff between created and modified should have been 8, but was 0.

In this case, my test had waited 8 seconds to modify the file, so the difference between created time (stats.birthtime) and modification time (stats.mtime) should have been 8 seconds, but it was zero, indicating that modification time was identical to birthtime.

Since this issue was only appearing on Linux, I fired up a Ubuntu VM with the same version as Travis (16.04.6 LTS Xenial) to see what the actual OS had to say about this. The results were similarly disturbing:

Here you can see that Ubuntu, and more precisely the stat system call, is refusing to show the creation time of a file that it just created.

So obviously NodeJS is not finding a birthtime from the OS, and just using the modified stamp as a fallback.

But why is birthtime showing as blank? Some quick searches made it sound like getting a file birthtime is either impossible on Unix, or it requires some insane workarounds. However, knowing that my test was passing on later versions of Node, on the same exact Ubuntu OS, obviously Node had found some sort of solution. What was it?

NodeJS fs.stat change and libuv

First, to prove that I'm not making this up, here is proof that in older versions of Node, prior to 10.16, fs.stat returns the wrong birthtime, whereas in 10.16 and up, it returns the right one:

So, how the heck did Node fix this? The answer took some digging, but also gives some insight into how Node versioning and core dependencies work.

When I first tried to look through Node's source code to find how "fs" worked, I wasn't finding much of value. It turns out, that is because Node uses something called libuv for a lot of the heavy OS stuff, such as filesystem calls, and most of the code for FS stuff is in there.

Libuv and statx()

Knowing that I should be looking at libuv, I was finally able to get somewhere and found a proposal that explained how the libuv team found out about updates to the Linux system that would improve stat calls. This was primarily the introduction of the statx() syscall, which as opposed to stat, actually *does* return the birthtime!

The statx() syscall support was added to the Linux kernel in 4.11 (2017). It was then added to glibc v2.28 in 2018. glibc is the "Gnu C Library", essentially a wrapper around low-level system stuff that allows you to make calls from C/C++. This is important, because this is how libuv makes system calls from its C powered codebase, and getting statx support in glibc meant that it could be used in libuv, which also meant it could get used in Node!

The proposal to use statx in libuv was accepted, and the code to do it was merged with PR #2184, in Feb 2019, which shortly after made it into the official v1.27.0 (Stable) release in March 2019 (see changelog). This update to libuv, in turn, was manually pulled into the Node source code via this specific commit, on March 16th, 2019.

Here is where it gets a little confusing. That commit is tagged as "v12.10", so you might expect that the tag, plus how recently it was merged, means that the improved fs.stat code is only in Node V12.10 and up... but you would be wrong (like I was).

Node versioning is a little complicated (see this and this), but the basic summary is that the libuv update was applied to multiple versions of Node based on where they were in their lifecycle:

Node v8.x did not receive the update on any releases, even though it is LTS, since it is on "maintenance LTS" and not "active LTS"
Node v9.x did not receive the update on any releases, since it is odd numbered, and thus is not LTS and the update did not happen during its active dev stage
Node v10.x received it, starting with 10.16.0, since it is "active LTS"
Node v12.x received it on all versions since it is under current development

Testing for statx support

So, back to my original issue, now that I know all the above, how do I detect when the test is going to run on a Linux system where Node's fs.stat function will NOT use the improved statx calls instead of stat?

The pseudo code is something like this:

(node v < 10.16.0 && linux) OR (node v >= 10.16.0 && linux && kernel < 4.11)

If the above condition is found to be true in my code, I have Travis-CI simply skip the test that relies on an accurate birthtime. You can see how I implemented it if you look at the commits linked to from this issue. No more failing tests! :)

Implementing yourself

If you are looking to implement the use of statx in your own program, outside of using it through NodeJS, you have some considerable work set out for you. I would recommend looking at how libuv brought in the changes in their PR, and also taking a look at this StackExchange answer.

Git-Date-Extractor NPM Package and CLI Tool

Joshua Tzucker — Sat, 21 Sep 2019 04:08:30 GMT

Github and NPM:

Github repo: https://github.com/joshuatz/git-date-extractor

NPM link: https://www.npmjs.com/package/git-date-extractor

About this project:

In general, OS timestamps for files, such as when a file was created or last modified, are problematic for developers. They can be unreliable, ambiguous, and are not exactly portable between systems. For example, in many versions of Linux, trying to get the "birthtime" of a file, or when it was created, gives an empty result. Another wrinkle to this issue is that OS timestamps are not tracked as part of Git - so if you clone a repo, all the files that are created will have timestamps that point to the time of cloning, regardless of when the files were actually last modified.

I built this tool because I needed a solution for retrieving and storing file timestamps, and I realized that although Git does not store timestamps directly on files, they could be pulled programmatically by scraping the Git history of a file and seeing when it was created or modified as part of commits. That is what this tool does; you can give it a list of files (or a directory) and it will use Git as the preferred method for generating timestamps. It can even save the results as a JSON file with the filepaths as keys; this makes it a great alternative solution for storing file modification dates in a system like Gatsby, which has no traditional database for storing that kind of metadata. In fact, I myself use this solution for my own Gatsby site, which is discussed in this post.

Demo:

Tech stack:

NodeJS
Javascript + TypeScript annotations with JSDoc
Initial structure was scaffolded from Yeoman generator-nm
Uses Meow for CLI parsing
Unit and integration tests use Ava
Code coverage analysis with Istanbul JS / NYC
Automated build CI testing with Travis CI
XO for linting

Features:

Make sure to checkout the Github repo for the current up-to-date readme, but I'll list some key features below as well:

Usable from both CLI and JS
Very configurable (filter results by filename, directory)
Get output in console or save as JSON file
Creation timestamps are cached once generated, if using JSON file
Automatic re-staging of JSON file if calling script with githook option
... and more!

Gatsby - Better Last Updated or Modified Dates for Posts

Joshua Tzucker — Thu, 12 Sep 2019 04:59:46 GMT

A common need for documentation or blog posts that cover topics that rapidly change (such as technology) is a "last modified" or "last updated" date and time. It can quickly tell readers how up-to-date your post is and clue them in on how relevant it might be to the information they seek.

A nasty surprise that I got when I first deployed Gatsby was how it handled file modification stamps. I had setup Gatsby to use a commonly used system for pulling in file timestamps; in GraphQL, I queried file -> modifiedTime or file -> mtime, which is when the markdown file was last modified on the disk, and then in in my page template, simply used something like Markdown last updated: {(new Date(node.parent.modifiedTime)).toString()}. When building and testing locally, everything worked beautifully. But when I deployed to a different location (my actual production server), instead of showing the true time the file was modified, every page showed the same modification date: when the deploy ran, today!

I had forgotten a very simple lesson that I had previously learned; git does not record or keep file timestamp metadata. So unless you are syncing files directly from your computer to your server, they are not going to have matching timestamps. In my scenario, my server (Netlify) did a git clone, which set all the modified stamps to the current time, and subsequent deploys after git push or git merge also had incorrect dates. This is not limited to Netlify though, since again, the issue is that git simply does not store this data. In fact, you will lose correct edit timestamps if you delete your project and re-clone from a remote source (Github).

Solution A: Storing Dates in Front-Matter or Filenames

So how do we get around this? Many people opt to use Gatsby's support for "front matter" - these are fields that you place at the top of a source Markdown file, and contain metadata about the file that can feed into GraphQL without necessarily being rendered visible to the user. This is basically the official recommendation from Gatsby and Netlify (see Github Issue and Stack Overflow). Here is what that front matter might look like at the top of a markdown file:

--- date: 2019-08-24 last-modified: 2019-08-26 author: 'Liz Lemon' title: 'Favorite Snacks' tags: - food - list - favs ---

How to even begin with my favorite snacks! Well, my top 20 are…

People also often store the creation date within the filename itself (like "2019-08-24--my-blog-post.md"), or use that in combination with storing dates in front-matter. If you look at some very popular Gatsby repos on Github (such as this, this, or this), either of these solutions are extremely popular.

Now, I'm about to say something that might be controversial and rather blunt; I think this approach has a bad "code smell".

To start with, you're introducing a non-automated hand-written process into the system; authors have to write out dates by hand as a text string and remember to keep them updated if they edit a file. Furthermore, how are you parsing this string? If you are converting it into a JS Date, you better have error checking in case someone miss-typed a date or suddenly wrote in the wrong syntax! And what about timezones? Do you want to store just the date, or do you want the actual hour modified in there too? If so, that is an extra thing the author has to manually type out!

Solution A - part 2

It should be noted that, hybrid CMS approaches (aka a "headless CMS" that supports MD) , such as Netlify CMS, do automate the part of generating this frontmatter and saving it in Markdown. However, this is a bit of a moot point if you accept pull-requests from users who are not using a CMS, or hand-edit markdown in addition to using the CMS. Also, this still leaves my second complaint, which is more of a rant:

The second part of my complaint is partly directed at a tendency to overuse Markdown and SSRs to the point of misuse. Storing separate and accurate post publication and last modified stamps is something that is built-in to pretty much every major CMS that uses a database; it's just another column in the SQL table. Requiring that authors hand type timestamps or using hacked together code to paste date as text strings into files is going backwards, and is a hint that maybe relying 100% on markdown to generate a site is not always the best choice. It irks me a bit though that it is so trendy to hate on databases and shove everything in static files, when the truth is that it takes a lot of kludgy fixes to just replicate some simple things that come standard and are more "safe" with a database based CMS. /rant

An alternative automated approach: Using git commit timestamps

Without using a headless-CMS layer as an editor for our markdown, there is an approach that should work locally, as well as when deployed, since it will check the timestamps into source control. The downside of this approach is that the initial setup is a little complicated, since it relies on combining several advanced tools: git hooks, git formatting (aka "pretty-print"), bash scripting, and using Gatsby's Node-APIs.

The way I'm currently setting this up is to use a JSON file to hold all the timestamps; the filepath relative to the project root is used as the lookup key, and the value is an object that holds both a last modified and created timestamp, in UNIX timestamp format.

The pseudo code / flow looks like this:

A git pre-commit hook fires when a dev uses git commit, but before the commit actually finalizes
The git hook retrieves the filenames of the files that were changed in the commit, and passes them to a node script, via arguments
The node script then does the bulk of the work:
- A "created" timestamp is generated once per file by looking up the file history with git log
  - A very handy one-liner command to do this is git log --pretty=format:%at --follow -- myFile.txt | tail -n 1
- The modified stamp is pulled in with stat and mtimeMs, which gives the modification time in a UNIX timestamp
- Timestamps are updated in the JSON file, and the file is added to the current commit before it finalizes
After the git hook completes, the updated timestamp JSON file will be lumped in with everything else in the commit
To pull the timestamps from the JSON file into Gatsby GraphQL, gatsby-node.js does the heavy lifting:
- Within exports.onCreateNode, my code checks to see if a timestamp exists for a given file node, and if it does, it uses it with createNodeField to add it to the node
- Now within exports.createPages and within React page templates, I can pull in the newly created node fields with GraphQL queries.

~~I'm still working on cleaning up this code a bit; once I'm happy with it, I might release it as a NPM package, since I could see it being of a lot of use to many projects, beyond just Gatsby.~~

The node script which pulls the modification and creation times in via git scraping is now published as an NPM package - check it out here.

So far, it's working great! Check it how it automatically updated my timestamps in this commit, based on the files updated:

Cheatsheets and Snippets - Markdown Powered Gatsby Mini Site

Joshua Tzucker — Sun, 25 Aug 2019 03:26:02 GMT

As I've been trying to hone my chops as a self-taught developer, one thing that has helped immensely is compiling my own cheatsheets and reference material. I realize that there is a huge amount of programming cheatsheets already out there and my own are not likely to get much traffic, but I am a huge believer that writing things down helps cement them in your own brain. Plus, it is always easier to read your own writing style than someone else's!

I used to keep these cheatsheets and snippets in random files in either Dropbox or OneNote, but as the collection started growing, I felt the urge to get them in order. As a project for myself, I converted them all to Markdown, placed them into an organized Github repository, and then integrated Gatsby (GatsbyJS). Gatsby is a ReactJS based static-site generator that can take a variety of inputs, including Markdown, and allows you to customize how it builds the output HTML by integrating custom React components, CSS, and more.

My Gatsby setup is still pretty boilerplate at the moment, but I have added a few unique features, and I plan on adding more soon. So far, I've added:

Dark vs light theme picker
- Simple CSS, JS, and persisted state through localStorage
Auto-generated directory index pages
- See this post for implementation details
Some custom automated markdown link fixing (needed for my current unique repo setup)

I also set this mini-site up to be compiled and served by Netlify, which allows me to have the site auto-rebuild and deploy when ever I push changes to the actual Markdown cheatsheets.

The Gatsby/React/Netlify powered portal is now live at cheatsheets.joshuatz.com, and I plan on continuing to built out my collection.

I might even add some non-development related cheatsheets, so I don't have to keep googling things like recipe ingredient equivalents.

Gatsby - Automatic Directory Listing Pages - Beginner's Attempt

Joshua Tzucker — Wed, 21 Aug 2019 01:36:21 GMT

I just started building my first Gatsby project yesterday, and I've already ran into an issue that I'm having trouble finding existing solutions for. My goal is to have the directory structure of my source directory, which Gatsby is building from, closely mirror the generated site structure. This would include automated directory listing pages.

The issue is that Gatsby is excellent at creating static files on a 1-to-1 basis, but there is much less information out there on how to create virtual pages, especially when they need to be automated based on directory structure or folder content. This is hard to describe, so I'm just going to show you my exact issue:

The Issue:

I'm trying to use Gatsby to create a front-end for a Markdown-based cheatsheet repo I've been working on. Let's say that the Gatsby build will be hosted at "output.test". The repo is a work-in-progress, but currently the structure looks something like this:

Cheatsheets
- JavaScript
  - js-modules.md
  - js-devops.md
  - ...
- bash-sh.md
- ...
Random
- roku.md
- index.md
...

If I follow the Gatsby tutorial on generating a static site based on Markdown (also see tutorial part-7), I'll end up with a very close-to-usable result, where I can navigate to "output.test/cheatsheets/bash-sh/" or "output.test/random/roku/". However, what about "output.test/cheatsheets/"? Well, unlike "random", the "cheatsheets" directory does not have an index.md in it, so no corresponding static index.html file will be generated. On a server with disabled directory listings, this will result in a 404 error or directory listing denied error on trying to access it.

If you have not disabled directory index listings on your server, one option is that you can just use the default directory listing service offered by your server. For example, this is what it looks like on Apache:

However, there are many issue with this. The most important is that it breaks the whole PWA/SPA part of Gatsby - because this "index of" page is not handled by React/Gatsby, it can't be pre-fetched and rendered by the system, and links to it will be broken. The second issue is that these automatic server directory listing pages are ugly, and not customizable. Here is how it could look a lot nicer, within the Gatsby system, just using default styling:

So how do we create these types of directory listing pages within the Gatsby ecosystem?

Gatsby-Node and exports.createPages

A good starting point for figuring out how to automatically create "index of" React pages with Gatsby, is with the official tutorial! Step 7 is "programmatically create pages from data", and points us towards using "exports.onCreateNode" to listen for updates in source input (such as the filesystem), and "exports.createPages" to tell Gatsby to create new pages programatically.

That's all fine and dandy, but again, we aren't trying to create a 1-to-1 output of static files to input nodes. Here is how I approached this (pseudo code) in gatsby-node.js:

use exports.onCreateNode to listen for MarkdownRemark files / nodes
- As they come through, add a node field of "slug"
- So far, this is boilerplate code that matches the tutorial step 7
CUSTOM: use exports.createPages to create directory listing pages
- Use GraphQL to get allMarkdownRemark and allDirectory nodes
- Iterate through all the markdown files:
  - Create a page for the markdown file itself (same as tutorial)
  - Record what directory the page belongs to, and mark that this file is a child of it
- Iterate through all the directories that hold markdown files
  - Record which directory the current one is a sub-dir of, and mark as child of it
  - Check if the current directory already has an index page, and if it does not, add it to a "queue" of index pages that need to be generated
- Iterate through the queue of index pages that need to be created
  - use actions.createPage to generate the page, and pass a template file, and tell Gatsby to also send along "meta" information about the children of the index page, through context (which gets passed as props.pageContext)
Directory Listing Template - directory-index.js:
- Combine the child directories and child file lists that were passed via props.pageContext into a combined array.
- Use the combined listing array to generate links and icons on directory vs file

Here it is in action:

Not bad! And that is with very minimal styling or tweaking beyond the Gatsby defaults.

The Code:

This is not the prettiest or most efficient code, but it is a good jumping off point for some approaches on how to handle this. If you are going to try to replicate these result for your own needs, you will probably need to tweak both these files to get things how you want them.

gatsby-node.js:

const path = require('path');
const { createFilePath } = require(`gatsby-source-filesystem`)
const AUTOBUILD_INDEXES = true;
/**

Implement Gatsby’s Node APIs in this file.

See: https://www.gatsbyjs.org/docs/node-apis/
*/

exports.onCreateNode = ({node, getNode, actions}) => {
if (node.internal.type === ‘MarkdownRemark’){
const fileNode = getNode(node.parent);
// Programmatically create slug field and value, and append to node to be consumed by page creator
// path should mirror md directory, and should be extracted
let slug = createFilePath({node, getNode, basePath: ”});
if (node.frontmatter && node.frontmatter.customPageSlug){
// replace parsedName with customPageSlug
slug = slug.replace((new RegExp(\\/${fileNode.name}\\/$)),/${node.frontmatter.customPageSlug}/);
}
actions.createNodeField({
node,
name: ‘slug’,
value: slug
});
}
}
exports.createPages = async ({graphql, actions}) => {
let subdirsWithIndexPages = [];
let subdirIndexesToCreate = [];
let subdirIndexPages = {};
const result = await graphql(    query {       allMarkdownRemark {         edges {           node {             fileAbsolutePath             fields {               slug             }             parent {               ... on File {                 relativePath                 base                 name               }             }           }         }       }       allDirectory(filter: {sourceInstanceName: {eq: "cheatsheets-md"}}, sort: {fields: name, order: ASC}) {         nodes {           absolutePath           base           relativeDirectory           relativePath           name         }       }     }  );
function recordAsChild(subdir, child, isDir){
subdirIndexPages[subdir] = typeof(subdirIndexPages[subdir])===‘object’ ? subdirIndexPages[subdir] : {children:{
dirs: [],
md: []
}}
const target = isDir ? subdirIndexPages[subdir].children.dirs : subdirIndexPages[subdir].children.md;
target.push(child);
}
result.data.allMarkdownRemark.edges.forEach(async ({node}) => {
// Get dir of file
const subdirAbs = path.dirname(node.fileAbsolutePath);
// Note that this md file is child of dir
recordAsChild(subdirAbs,node,false);
// Create page for file itself
actions.createPage({
path: node.fields.slug,
component: path.resolve(./src/templates/generic.js),
context: {
// Becomes available as GraphQL variables within page queries
slug: node.fields.slug
}
});
});
// Iterate over directories that contain MD, and check if they need an index page created
let directoryIteratorPromise = new Promise((resolve,reject) =>{
result.data.allDirectory.nodes.forEach(async (node, index, arr)=>{
const subdirAbs = node.absolutePath;
const subdirRel = node.relativePath;
const parentDir = path.posix.dirname(subdirAbs);
// Note child of dir
recordAsChild(parentDir,node,true);
// Create page for subdir that file is in, if it is missing an index page. Skip for homepage (’/’), or top level page (/test.md)
const alreadyHasIndexPage = (subdirsWithIndexPages.indexOf(subdirAbs)!==-1 || subdirRel === ”);
if (!alreadyHasIndexPage){
// Check for index.md
const indexPath = path.posix.join(subdirAbs,‘index.md’);
const existResult = await graphql(         {           allMarkdownRemark(filter: {fileAbsolutePath: {eq: "${indexPath}"}}) {             totalCount           }         });
if (existResult.data.allMarkdownRemark.totalCount < 1) {
console.log(There is no index for ${indexPath});
subdirIndexesToCreate.push({
subdirAbs: subdirAbs,
subdirRel: subdirRel,
parentDir: parentDir
});
}
subdirsWithIndexPages.push(subdirAbs);
}
if (index === arr.length-1) resolve();
});
});
if (AUTOBUILD_INDEXES){
directoryIteratorPromise.then(()=>{
console.log(’=============== BUILDING SUBDIR INDEX PAGES! ======================’);
for (let x=0; x<subdirIndexesToCreate.length; x++){
const subdirPaths = subdirIndexesToCreate[x];
// Create index page!
actions.createPage({
path: subdirPaths.subdirRel,
component: path.resolve(./src/templates/directory-index.js),
context: {
slug: subdirPaths.subdirRel,
hasIndex: false,
meta: subdirIndexPages[subdirPaths.subdirAbs]
}
});
}
});
}
}

src/templates/directory-index.js:

import React from "react"
import { Link, graphql } from "gatsby"
import Layout from ”../components/layout”
import Image from ”../components/image”
import SEO from ”../components/seo”
import {strMethods} from ”../helpers/helpers”
function makeDisplayName(slug){
slug = slug.replace(/-/gim,’ ’);
slug = slug.split(///).map((e)=>{return strMethods.toTitleCase(e)}).join(’/’);
return slug;
}
export default (props) => {
const displayName = makeDisplayName(props.pageContext.slug);
const children = props.pageContext.meta.children;
// Compile listings - subdirectories and files
let listings = [];
listings = listings.concat(children.dirs.map(function(d){return {
path: ’/’ + d.relativePath,
display: ’/’ + d.name,
isDir: true
}}));
listings = listings.concat(children.md.map(function(m){return {
path: m.fields.slug,
display: m.parent.name,
isDir: false
}}));
return (
<Layout>
<SEO title={${props.pageContext.slug}} />
<h1>Generated Index Page - {${displayName}}</h1>
<div className=“directoryListingWrapper”>
<div className=“directoryListingRow”>
<i className=“material-icons”>folder</i>
<Link to={${document.location.pathname + '/../'}}>..</Link>
</div>
{listings.map((listing)=>{
return (
<div className=“directoryListingRow” key={${listing.path + ((new Date()).getTime())}}>
<i className=“material-icons”>{listing.isDir ? ‘folder’ : ‘description’}</i>
<Link to={${listing.path}}>
{listing.display}
</Link>
</div>
)
})}
</div>
</Layout>
)
}

WordPress - Relative Stylesheets and Scripts, Plus wp-config.php Order

Joshua Tzucker — Sun, 18 Aug 2019 15:00:38 GMT

I recently wanted to force WordPress to make all my <script> and stylesheet (<link>) tags load relative URLs instead of absolute URLs, and ended up learning a bit about the internals of WordPress (probably more than I wanted to know). For reference, I wanted to turn this:

Into this:

I wanted to do this because I was working with a local development environment, using Laragon on Windows as an Apache host, and proxying through Ngrok or something similar was not working with absolute domain-tied links, for a number of reasons. Furthermore, relative paths that were being passed to wp_enqueue_style were not being preserved.

Click here to jump right to the solution, or read on if you are interested in the details of how WordPress forces absolute paths.

First wrinkle: WP forces absolute links

The first issue I ran into was WordPress does not respect the format you use to pass links into the enqueue system, even if you make 100% sure you pass in a relative link. For example, look at this code:

wp_enqueue_style('joshuatzwp-style',wp_make_link_relative(get_stylesheet_uri()),array(),$cacheBustStamp,'all');

You would expect that the use of wp_make_link_relative would ensure a relative link, but the style tag that this generates actually is absolute with the domain:

Why is this happening? Well, it is not a fun answer:

WordPress do_item complexity

To figure out what was going on, I ended up tracing backwards from where the <script> and <link> tags are actually echoed out into the page, to see how that HTML is generated. Here is a rough overview of what that looks like:

Both the script class (wp-includes/class.wp-scripts.php) and the style class (wp-includes/class.wp-styles.php) have a method called do_item() that is responsible for actually processing each enqueued "dependency" and echoing it out
- In class.wp-scripts.php, there is inline code that prefixes any relative URL with $base_url
- In class.wp-styles.php, it uses a function called _css_href(), which also prefixes relative URLs with $base_url
The $base_url, which is a member of the shared extended class for both scripts and styles, and used to prefix URLs above, is actually filled with a value in wp-includes/script-loader.php
- This value ends up either being the result of site_url(), or wp_guess_url()
  - site_url() pulls it from get_site_url(), which pulls it from get_option('siteurl'), which comes from your admin settings
  - wp_guess_url() checks the global constant defined 'WP_SITEURL', and if that fails, tries to scrape it off the incoming request

None of this knowledge was strictly necessary to force relative paths, but it gives some insight into how the whole process works. I wouldn't actually want to change any of these files, since they are part of the core of WP, and would be overwitten on a system update anyways.

The solution for relative paths:

The solution here is to hook into a WordPress filter. I've written about this before, when trying to add async/defer attributes to scripts and style tags, in this post. In fact, our solution is going to look pretty similar.

Basically, we will hook into the filter for 'script_loader_tag', which is the filter for <script> tags, and 'style_loader_tag', which is the filter for stylesheet <link> tags. In our hook, we will take the source that WordPress has prefixed and forced into an absolute URL, and remove the domain part that they added.

Here is the code:

function makeInternalLinkRelative($src){
    return str_replace(get_option('siteurl'),'',$src);
}
add_filter(‘style_loader_src’,function($src, $handle){
return makeInternalLinkRelative($src);
},10,4);
add_filter(‘script_loader_src’,function($src, $handle){
return makeInternalLinkRelative($src);
},10,4);

Pretty simple, huh! Just add this code to your theme's (or child theme's) function.php file, and you should be good to go!

The Solution for Dynamic Domains

If you don't want to use relative paths, but just want WordPress to know when you are serving it through a different domain (such as when using Ngrok or swapping domains) and automatically adjust all the links, there is another solution we can employ. We can modify the wp-config.php file in the root of our WordPress install to dynamically set the "siteurl", which as my earlier research pointed out, is used internally by WP to auto-prefix relative links.

Here is what I have added to my wp-config.php file:

$host = $_SERVER['HTTP_HOST'];
$protocol = ((!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] !== 'off') || (isset($_SERVER['SERVER_PORT']) && $_SERVER['SERVER_PORT']===443)) ? 'https://' : 'http://';
define('WP_SITEURL', $protocol . $host);
define('WP_HOME', $protocol . $host);

Now, here is the really important thing. Make sure this code comes before this line of code that should already be in the file:

require_once(ABSPATH . 'wp-settings.php');

This matters because wp-settings.php is responsible for the chain of events that fills in the $base_url variable we were looking at earlier, and is ultimately responsible for prefixing URLs. If that runs before you set the "siteurl" global, it will have the domain from your admin settings, not from our dynamic override. I found this out the hard way!

Coding a CSS Theme Switcher - a Multitude of Web Dev Options

Joshua Tzucker — Fri, 16 Aug 2019 15:00:36 GMT

Introduction:

Front-end web theming seems to be a current hot topic, and I have to admit that I felt the itch to jump on the bandwagon and add a dark theme to a Single-Page Application I’m working on. But where to get started? There are so many guides out there, and yet I couldn’t find many that I felt adequately talked about CSS theming in a generic way that covers many different approaches, rather than a tutorial that is essentially “copy and paste this into your code”.

So I set out to make one (this post), a guide that briefly discusses each of the many available options in a broad manner, with some short snippets of example code.

First though…

What is theming?

In this context, CSS theming or CSS theme switching refers to a set of shared styles (colors, etc) that are grouped as a theme, and being able to switch between themes instantly on a webpage or within a SPA, without refreshing the page.

A very simple example of a theme, that I use throughout this post, is as follows:

Dark theme:
- Primary Color: Black
- Secondary Color: White
Light Theme:
- Primary Color White
- Secondary Color: Black

Ok, now lets look at different options for implementation:

Method A: Using CSS custom properties / variables

This is fast becoming one of the most common current approach to implementing switchable themes, and uses a newer CSS feature called “Custom Properties”, which are often referred to as “CSS variables”. In short, CSS custom properties allow you to define a variable, such as “primaryColor”, which can then be referenced in hundreds of other spots throughout your CSS.

This is crucial for theming, because it means that you can share a property among thousands of elements, and to change it, you just need to update that one variable value, rather than needing to individually update each element’s CSS.

To get into the details of how to use CSS variables to deliver switchable CSS themes, let’s break this method into a few different approaches:

Method A-1: Using specificity to change variable inheritance

Ok, long title, but I promise you that this method is actually pretty simple. In fact, this is one of the most common methods, and can be seen in current use across several major websites. Basically, you define the default theme as a set of variables, and then override the exact same variables through another block of CSS. The override will work because we will add something specific (class or attribute) to a top level node (like <body>) to trigger the more specific rule.

Here is some example CSS, without using custom properties:

body .myElement {
    background-color: white;
}
body.dark .myElement, body[data-theme="dark"] .myElement {
    background-color: black;
}

And here is the same output, but using CSS variables:

body {
    --primaryColor: white;
    --secondaryColor: black;
}
body .myElement {
    background-color: var(--primaryColor);
}
body.dark, body[data-theme="dark"] {
    --primaryColor: black;
    --secondaryColor: white;
}

Then you can simply toggle the class or attribute with JS, either directly with something like .setAttribute(‘data-theme’,’dark’), or with a reactive binding within something like React or Vue.

Here is a quick demo using setAttribute on <body> to switch themes:

See the Pen <a href="https://codepen.io/joshuatz/pen/xxKOZVr/">Method A-1</a> by Joshua T(<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

You could technically do this without using CSS custom variables, but you end up with a lot of handcoding and repeating rules. For every rule/element that has different styling for a specific theme, you only have to write that block once if you are using variables, but if not, you have to write it 1 x as many themes as you have. Example below, where you can see that with three elements and three themes, the minimum number of CSS blocks without custom properties is 9 (3×3), whereas with variables, we only need 3 (one for each element):

Method A-2 Re-Assign variable values with JS

This method usually is used when you aren’t just theme switching, but also allowing for live theme property changing by the user, through something like a color picker. Let’s pretend that the user has just picked a new custom primary color for their theme, and they want to preview how it will look. Here is the initial CSS:

:root {
    --primaryColor: red;
    --secondaryColor: blue;
}

With just a little bit of JavaScript, we can easy modify any of these values. Here we go changing the primary color:

// This will target :root variables
document.querySelector(':root').style.setProperty('--primaryColor',newColor);
// Or, same thing...
document.documentElement.style.setProperty('--primaryColor',newColor);

You can take this a step further and easily implement theming logic to swap out entire themes and switch between multiple:

const themes = {
    light: {
        '--primaryColor': 'white',
        '--secondaryColor': 'black'
    },
    dark: {
        '--primaryColor': 'black',
        '--secondaryColor': 'white'
    },
	red: {
		'--primaryColor': 'red',
		'--secondaryColor': 'white'
	}
}
function activateTheme(theme){
    for (let prop in theme){
        document.querySelector(':root').style.setProperty(prop, theme[prop]);
    }
}
// Switch to the dark theme:
activateTheme(themes.dark);

Demo:

See the Pen <a href="https://codepen.io/joshuatz/pen/yLBJeJY/">Method A-2</a> by Joshua T(<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

By default, when you change a CSS property via Javascript, the change is not persisted when the user reloads the page. To save their changes to a theme, you have a bunch of different options, which is beyond the scope of this post. However, here are some generic options:

On value change, also AJAX to server and save with User data
- Then, when user logs in, send back theme data either as on-the-fly generated CSS, or JSON which gets parsed and mapped back to CSS custom vars
Persist theme to localStorage and map back on page load
- If you are using a state management system and storing the user’s theme config there, this should be very easy to do with something like “redux-persist”
- You can also just inject a <style> tag into the page with overriding variable values; this is how dev.to is currently handling theming — on page load, it injects a different set of variable values through a <style> tag, depending on the value of a localStorage value

Method B: Using SASS (Sass/Scss)

Technically, you could use Sass to accomplish both methods A & B, since Sass transpiles to CSS, and you could also just stick both methods, as CSS, into Sass, and it should be valid.

However, there are some Sass specific approaches for theme switching, which I’ve outlined below.

Sidenote: Throughout this post, my demos all use the SCSS style of Sass, but all of this should be possible with indented SASS; you would just need to reformat.

Method B-1: Themes as Nested Maps

One way to approach this in Sass is with nested maps. The inspiration for this code, and one of the best guides on this method is this post.

Nested maps in Sass resemble JSON, and allow nesting of key-pair maps that can hold any valid Sass value. This makes them a great way to hold multiple named themes, and sub-properties for each theme. Here are my two example themes as a nested map saved to $themes:

$themes: (
    'dark': (
        'primary': black,
        'secondary': white
    ),
    'light': (
        'primary': white,
        'secondary': black
    )
);

This by itself does not generate any CSS, so you have to write your own function/code to loop through your themes and generate the CSS you want. It is best to define re-usable helper functions to handle this, that use mixins:

/**
* Mixin to use to generate blocks for each theme
* Automatically takes @content
*/
$scopedTheme: null;
@mixin themeGen($allThemesMap: $themes) {
    @each $themeName, $themeMap in $allThemesMap {
        .theme-#{$themeName} & {
            // Creating a map that contains values specific to theme.
            // Global is necessary since in mixin
            $scopedTheme: () !global;
            @each $variableName, $variableValue in $themeMap {
                // Merge each key-value pair into the theme specific map
                $scopedTheme: map-merge($scopedTheme, ($variableName: $variableValue)) !global;
            }
            // The original content passed
            @content;
            // Unset
            $scopedTheme: null !global;
        }
    }
}
/**
* Function to call within themeGen mixin, to get value from the current theme in the iterator
*/
@function getThemeVal($themeVar){
    @return map-get($scopedTheme,$themeVar);
}

Finally, using this with an element:

/**
* Actually using theme values to generate CSS
*/
.myComponent {
    @include themeGen() {
        background-color: getThemeVal('primary');
        color: getThemeVal('secondary');
    }
}

Here is a demo showing the final CSS this can generate:

See the Pen <a href="https://codepen.io/joshuatz/pen/aboNPLV/">Method B-1</a> by Joshua T(<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

Technically, you could hand code the CSS that this generates (and some old-school devs do), but there is not a good reason to do so, and if anything, you are more likely to write error-ridden CSS.

A related, but slightly different approach (avoids global vars) can be found here. And here’s another variation.

Method B-2: Simple Mixin “Dump”

This method is not as elegant as C-1, but less complicated. Essentially you dump a huge block of CSS into a mixin, which then repeats it, but with a named parent theme class.

@mixin scopeToTheme($themeName,$prop,$val){
    .#{$themeName} & {
        #{$prop}: #{$val}
    }
}

button {
  background-color: red;
  padding: 20px;
  margin-bottom: 4px;
  border-radius: 10px;
  @include scopeToTheme(
    'dark-theme',
    'background-color',
    'black'
  );
  @include scopeToTheme(
    'light-theme',
    'background-color',
    'white'
  )
}

And here is the CSS that this generated:

button {
  background-color: red;
  padding: 20px;
  margin-bottom: 4px;
  border-radius: 10px;
}
.dark-theme button {
  background-color: black;
}
.light-theme button {
  background-color: white;
}

Method B-3: CSS Custom Properties to Sass

I’m not sure I’ve seen this done anywhere, but an interesting approach could be to combine Method B with Sass by pulling the CSS variables into Sass. Like so – CSS:

:root {
    --primaryColor: red;
    --secondaryColor: blue;
}

Sass:

$primaryColor: var(--primaryColor);
$secondaryColor: var(--secondaryColor);

Method C: Mutating State

I won’t get into the specific of how to code this out, but this is pretty easy to think about conceptually. In a reactive framework, you can hold CSS properties, such as colors, as part of state. Then, using a CSS-in-JS framework, or even just sticking state in inline css, you can plug the value from state into your scoped CSS.

React Demos:
- Simple inline style binding
- Styled components + ThemeProvider
React Native Web
- Inline style binding with “atomic” CSS classes
Vue Demos:
- Simple inline style binding

It looks like Twitter uses this method, with React Native Web.

Method D: Crazy Old-School

There are so many reasons not to do it this way, but I feel obligated to share this hackish idea anyways. If you really wanted to insist on using CSS, but not using any variables or touching of the <body> class or attributes, you could hand code different themes as completely separate CSS files. Like dark-theme.css and light-theme.css. Then, to switch the theme, you could use AJAX to pull in the new theme file. Or, if you wanted to support browsers without Javascript, you could even use server side code to change which theme gets included and force a page refresh.

As a side note, an effect of this approach is actually better utilization of bandwidth, since you are only sending what theme is needed.

Wrap-Up / Final Thoughts:

First, although this was an extensive look at theming options, it should not be considered fully comprehensive; there is an endless number of ways you can implement custom theming.

Second, if you have made it all the way through the post, you might still be wondering “well, which option is the best one?”. The truth is that it really matters what your objective is. If your goal is to support as many browsers as possible and/or have functions within styling (like converting hex to RGBA and then darkening), then SASS is probably the way forward (Method B). Or, if you want to have the smallest CSS filesize and/or binding of CSS values to JS values, then CSS custom properties are likely the best fit (Method A and Method C, or Method B with special setup).

The short answer is use what fits best with the framework or existing structure of your site/SPA/App. And have fun theming!

This writeup was first published at: https://joshuatz.com/posts/2019/coding-a-css-theme-switcher-a-multitude-of-web-dev-options/

Sass-Embed: Dynamic embedded SASS/SCSS-to-CSS Playground

Joshua Tzucker — Thu, 15 Aug 2019 09:11:38 GMT

Github repo: https://github.com/joshuatz/sass-embed

Purpose:

I needed to embed SASS/SCSS side-by-side into a blog post as a demonstration of some code, and realized that it was extremely difficult to find any "live" SASS-to-CSS playgrounds that supported embedding into another website. There are code playgrounds that support SASS conversion and embeds, but the conversion is usually done server side when the playground is saved, which is incompatible with an anonymous iframe embed. For examples of those types of solutions, see the section on alternative (easier) solutions.

This project is the end result of roughly a day and half of work, to see how much functionality I could shoehorn into a single-page that could be embedded as an iframe.

What it is:

The whole point of this project is being able to embed it into posts like this, so why don't I just show you! Here is a live embedded demo:

As you can see, above is a two-pane embed, showing SCSS input side-by-side with CSS output. What is unique about this though, is a few things:

It is using in-browser SASS-to-CSS conversion
It is a live playground
- With "autorun" on, if you type in the left panel and then pause for a second or two, it will automatically be converted
- You can manually trigger conversion by clicking the "convert" button
- You can toggle indented syntax (SCSS vs indented SASS)
You can adjust the width of each panel by sliding the divider in the middle
Even though the playground is embedded within an iframe, you can modify the contents from the parent window by sending messages. For example, try clicking here and looking at the above embed.

Embed-Gen

Since the embed is a simple single-page application without any sort of database, in order to preload SASS and various options, you have to either use special querystring parameters (part of the URL of the embed), inline tags placed into the index.html file, and/or postMessage events.

Embed-Gen is a simple tool that I built to simplify crafting the necessary embed codes to do all this. You find it hosted up here.

How it works:

For how it works, and more details on how to use it, please refer to the readme file in the project repo.

Alternative (easier) Solutions:

If you just want to embed a side-by-side display of SASS input and CSS output, there are some easier solutions. One of the easiest is to use an embeddable code playground tool, like JSFiddle, use the SASS input panel, and then force the editor to show the generated CSS. My first attempt at this used Javascript, which you check out here.

This is already pretty simple, but I was able to simplify it even further thanks to this tweet from Matt Stow:

There's an even simpler, faster way of making a Sassmeister clone in @CodePen: CSS! pic.twitter.com/lg6DpAvIFR

— Matt Stow ✨🦄 (@stowball) April 20, 2019

That snippet of CSS is not only a more simple solution than the JS solution, but it also easily works across different code playgrounds (with a slight tweak), since most tend to inject the generated CSS through an inline <style></style> tag, which is what this snippet relies on to work.

Here are examples using this method, which you can easily clone and paste your SASS into before generating an embed code:

JSFiddle - Link
CodePen - Link

And here is a nice demo:

<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span> See the Pen <a href="https://codepen.io/joshuatz/pen/rNBeyaW/">SASS Embed - Show CSS - With CSS</a> by Joshua T (<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

Or, if you don't need the SASS/SCSS input side-by-side and are OK simply toggling between, create a CodePen with nothing but SASS input and un-select the result pane when generating the embed code. Then the embed will look like the below and you can simply toggle between the input SASS and the generated CSS by clicking the "View Compiled" button in the bottom right:

See the Pen <a href="https://codepen.io/joshuatz/pen/qBWZrdp/">SASS Embed - CodePen Toggle Between</a> by Joshua T (<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

If you don't need to embed at all, you can use one of these live Sass-to-CSS playground:

Wordpress - Script and Style Tags - Adding Defer, Async, and Lazy Load

Joshua Tzucker — Sun, 11 Aug 2019 09:29:24 GMT

One of the downsides to WordPress is that they don't always have the best exposed methods / APIs for modifying how HTML generated by their own system (rather than your content like posts or pages) gets echoed out. For example, WP devs are familiar with 'wp_enqueue_script()' in PHP, which will generate a <script></script> tag output in HTML, but there are no arguments on that method to add attributes or even to simply grab the raw text output of it. To do anything like this requires using WordPress Filter, which is not always straightforward.

I started looking into this since I wanted to be able to add defer or async loading to scripts and styles, which requires modifying attributes of the outputted tags.

Where are the filter calls actually generated

If you start to search for how to modify the HTML that is generated by "wp_enqueue_style" and "wp_enqueue_script", you will quickly come across posts recommending that you hook into filters, "style_loader_tag" and "script_loader_tag" respectively. But why does this work? How?

Since WordPress is open source, we can find out!

After a little digging, we can see that the function that actually turns queued scripts and styles into echoed out <link> and <script> tags is do_item($handle), which is a public method on the WP_Scripts and WP_Styles classes. Both methods also call a named filter. Here is a summary:

Type	HTML Generated by	Uses filter:	Actual filter call:	Arguments passed to filter
Script	WP_Scripts::do_item($handle, $group)	script_loader_tag	Link	$tagHtml, $handle, $src
Style	WP_Styles::do_item($handle)	style_loader_tag	Link	$tagHtml, $handle, $src, $media

Modifying the output of do_item()

The way both of the do_item() filters work is that it passes a bunch of arguments to a function you hook into the filter with add_filter(), and it expects that your function returns a string that represents the new <link> or <script> tag that it should use. For example, if you wanted to change all non-https scripts to https, just in case a plugin tries to add one, you could do something like this:

<?php
add_filter('script_loader_tag', function($tag, $handle, $src){
    $tag = preg_replace('/^http:\/\//', 'https://', $tag);
    return $tag;
},10,4);

You can do pretty much anything you want to the tag HTML string; this is both beneficial and dangerous. On one hand, this gives a great deal of flexibility, but on the other hand, it is very easy to craft malformed HTML. Plus, like I mentioned, all you can return is the string of HTML for WP to embed - there is not a method for adding or removing a particular attribute, which means most of the time relying on Regex to modify the tag passed by WP.

Putting it together in a reusable way

There are a lot of hackish ways you can add filters all over your functions.php to file modify the output of queued scripts and styles, but there are also some ways to craft reusable methods that will make modifying script and style tags easier. For one, since both filters have almost identical arguments, we can add a single function as the callback if we want to. Here is how I have done that:

<?php
/**
 * Callback for WP to hit before echoing out an enqueued resource
 * @param {string} $tag - Will be the full string of the tag (`<link>` or `<script>`)
 * @param {string} $handle - The handle that was specified for the resource when enqueuing it
 * @param {string} $src - the URI of the resource
 * @param {string|null} $media - if resources is style, should be the target media, else null
 * @param {boolean} $isStyle - If the resource is a stylesheet
 */
function scriptAndStyleTagCallback($tag, $handle, $src, $media, $isStyle){
    // ...
    // Function body omitted
}
add_filter(‘script_loader_tag’,function($tag, $handle, $src){
return scriptAndStyleTagCallback($tag, $handle, $src, null, false);
},10,4);
add_filter(‘style_loader_tag’,function($tag, $handle, $src, $media){
return scriptAndStyleTagCallback($tag, $handle, $src, $media, true);
},10,4);

Configurable defer, async, and lazy CSS loading

Next, I wanted to be able to easily enqueue certain scripts and styles in a way that would generate output HTML with async or defer loading. My current approach is as follows:

First, have a variable that holds the handles (unique identifiers) of any scripts or styles that need to be given special treatment:

$specialLoadHnds = (object) array(
    'scripts' => (object) array(
        'async' => array(),
        'defer' => array()
    ),
    'styles' => (object) array(
        'async' => array(),
        'asyncPreload' => array()
    )
);

Items are added to this variable through helper functions I've added:

<?php
// Same signature as wp_enqueue_style, + $loadMethod as last arg
function wp_enqueue_style_special($handle, $srcString, $depArray, $version, $media, $loadMethod){
    global $specialLoadHnds;
    array_push($specialLoadHnds->styles->{$loadMethod},$handle);
    wp_enqueue_style($handle, $srcString, $depArray, $version, $media);
}
// Same signature as wp_enqueue_script, + $loadMethod as last arg
// Reminder - $inFooter should probably be false for both async and defer
function wp_enqueue_script_special($handle, $srcString, $depArray, $version, $inFooter, $loadMethod){
    global $specialLoadHnds;
    array_push($specialLoadHnds->scripts->{$loadMethod},$handle);
    wp_enqueue_script($handle, $srcString, $depArray, $version, $inFooter);
}
/**

Example: Adding a script with ‘async’
wp_enqueue_script_special(‘jquery’,‘https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js’,array(),false,false,‘async’);
*/

Next, within scriptAndStyleTagCallback(), I check the incoming $handle variable and see if it is contained in any of the special arrays. If it is, I apply a treatment as follows:

Scripts:
- 'async'
  - add 'async="true"' to <script> tag
- 'defer'
  - add 'defer' attribute to <script> tag
Styles
- 'async'
  - set 'media="none"' and 'onload="if(media!='all')media='all'"'
  - Add <noscript></noscript> block that contains raw $tag, in case user has JS disabled
  - Based on this post
- 'asyncPreload'
  - set 'rel="preload"' and 'as="style"' and onload="this.onload=null;this.rel='stylesheet'"
  - Add <noscript></noscript> block that contains raw $tag, in case user has JS disabled
  - Based on loadCSS approach, and should be used with their preload polyfill for browsers that don't support the 'preload' attribute

Here is everything all put together, into a single file that I require() into my functions.php file:

https://github.com/joshuatz/joshuatz-wp-theme/blob/71a12bca12c5fa5b25b6eb3a70cd498845076834/inc/special-loader.php

How to Get Github to Recognize a Pure Markdown Repo

Joshua Tzucker — Wed, 31 Jul 2019 15:00:44 GMT

File this under "not really necessary, but nice to have". If you have ever used Github, you have probably noticed the language details statistics bar (or "language breakdown bar"):

It doesn't really have an impact on anything, but it can tell other devs at a glance what kind of tech is likely in your repo, and it does influence the search results when searching for a repo across Github.

Unfortunately, if you create a repo and fill it with nothing but markdown files (.md), the language breakdown bar will either not appear at all, or if you introduce some non-markdown files, it will only show those. What gives?

Both the problem and the solution are revealed with a little bit of background knowledge.

How is the language stats calculated?

Internally, Github uses a tool called Linguist to detect and classify code files. What you might not know though, is that Linguist also has a lot of rules for preventing a file from contributing towards statistics. A file is excluded from stats if it is classified as any of these:

Vendored-code
Generated-code
Documentation
Data
Prose

For details, see this section of the readme.

Why does Markdown not show up?

The first reason why Markdown does not show up is because it is immediately classified as "Prose", which we know from above, is ignored when it comes to repo language statistics.

The second reason might or might not apply to you; filename and file structure. For example, if all your markdown files are named "readme.md" or are just in a folder called "Documentation", they would be classified as "Documentation" and excluded. You can see this from the Regular Expressions patterns in the "documentation.yml" file.

Solution:

It used to be that there was not a solution to this issue. However, over time, Github added "overrides" to Linguist, so now users can modify the default classification of files by overriding with git attributes (see issue #3964, which was solved by PR #3807, and example given in #3806).

How do we implement overrides? It is actually surprisingly pain-free.

First, create a ".gitattributes" file in the root of your repo. Then, use the standard git attributes syntax within that file to modify file attributes based on filename or pattern. Git Attributes are a way to assign specific, custom, and arbitrary attributes to specific files.

The linguist attributes you can override are:

linguist-documentation
linguist-langauge
linguist-vendored
linguist-generated
linguist-detectable

For example, here is a .gitattributes file where I have forced Github to count all Markdown files towards my language stats, *except* for readme.md:

# Linguist overrides
*.md linguist-detectable=true
README.md linguist-detectable=false
readme.md linguist-detectable=false

Technically, I believe toggling "linguist-detectable" should be all it takes to change whether or not a file is ignored, since it acts as a final override. If you wanted to be 100% sure that markdown is not excluded, you could also override any attribute that might prevent it from being recognized, like so:

*.md linguist-vendored=false
*.md linguist-generated=false
*.md linguist-documentation=false
*.md linguist-detectable=true

Final notes:

Keep in mind that the solution found here is applicable towards many issues with Github stats; you can easily override any Linguist classifications with Git attributes.

Also, Pro Tip: You can easily check if your file patterns are correct in your .gitattributes file by using the "git check-attr" command. For example, to check the attributes of a markdown file with the filename "web-dev.md", in my folder "cheatsheets", I would use:

git check-attr -a cheatsheets/web-dev.md

Vue - Mixing SASS with SCSS, with Vuetify as an Example

Joshua Tzucker — Tue, 30 Jul 2019 09:34:28 GMT

The Vue CLI is an impressive tool that, similar to create-react-app, boostraps and automates a bunch of the Vue setup process. Unfortunately, the "magic" that makes the CLI and Vue so easy to use, also abstracts away a lot of what it is doing under the hood and makes it a little difficult to understand how to deal with unexpected issues, such as a SASS vs SCSS conflict when trying to use SCSS with Vuetify:

Error: Semicolons aren't allowed in the indented syntax.

I got this error when trying to load a scss file through vue.config.js, which is the method that Vue officially recommends. If you search around, you can find that this is a known issue with Vue and how it uses webpack. If you selected SASS as one of the options while setting up the Vue CLI, under the hood Vue has installed and configured both "vue-loader" and "sass-loader" to handle the Sass->CSS conversion. The specific issue is that "vue-loader", using "sass-loader", can handle both SASS (indented syntax) and SCSS (CSS3 superset) formatted Sass files, but will run into a conflict if trying to mix them at a global level with loaderOptions.sass.data. And in my situation, I'm trying to combine Vuetify, which uses the SASS syntax, with my own style file, which uses SCSS syntax.

If you want to jump right to the solution click here, otherwise, keep reading as I dig into the specifics of this issue.

What this issues is not about:

First, let's try to make it clear what this issue is not. This issue is not simply about getting SASS/SCSS to work with Vue. That is pretty simple, assuming you have "sass-loader" installed. There are a few options, but the easiest is to simply stick your Sass into a style tag, like so:

<template>
    <div class="login">
        <button>Click to Login</button>
    </div>
</template>
<script>
</script>
<style lang=“scss”>
.login {
button {
color: green;
}
}
</style>

Or, you could "import" the Sass style file in your main JS file that loads Vue. For example:

import Vue from 'vue';
import App from './App.vue';
import router from './router';
import store from './store';
import './registerServiceWorker';
import vuetify from './plugins/vuetify';
import ’./styles/App.scss’;
Vue.config.productionTip = false;
new Vue({
router,
store,
vuetify,
render: (h) => h(App),
}).$mount(‘#app’);

The real issue: Global vs Non-Global

The real issue here is how to inject global styles across your entire Vue app, using Sass. The common reason why this is desired is Sass variables. For example, a common dev practice is to create a Sass variable file, maybe "_variables.scss", which has variables such as "$darkColor: #222222;", and then in a Vue component file (SFC), reference that variable value in the style tag, such as with:

button { background-color: $darkColor; }

This is deceptively complicated, as evidenced by this massive Github issue thread on vue-loader, which explores different approaches and issues. The solution that most people have landed on is modifying the vue.config.js file and tweaking the loaderOptions for CSS - there is a great guide by CSS-Tricks on how to do this here, and another guide by VueSchool here. Here is a sample vue.config.js file that follows their guide:

const path = require('path');
module.exports = {
css: {
loaderOptions: {
sass: {
data: @import "@/styles/_variables.scss";
}
}
},
}

This solution works for many users, but those using third-party libraries formatted with Sass syntax, such as Vuetify, will likely end up with this dreaded error:

Error: Semicolons aren't allowed in the indented syntax.

What is happening is that vue-loader/sass-loader is essentially trying to inject your import code, which is formatted as SCSS syntax (with a semicolon), into your third-party library, which uses SASS syntax (semicolons are not allowed.

Now what? We need some way to tell sass-loader to remove the semicolon when using with SASS syntax, and/or don't inject it at all into .sass if possible.

August 2019 Update: There is now a much easier way to handle this thanks to an update to the Vue CLI. See "the easy solution" below!

Mixing Sass-syntax and SCSS-syntax in Vue

If you don't need to use variables, and simply want some SCSS converted to CSS and applied across globally, across your entire app, you have a few options, some of which I've already mentioned:

Import in your main JS file (with "import './styles/myStyle.scss'")
Import in your root Vue template file (e.g. App.vue), either through the script section, the style section, or a style link tag.
Use your own pre-processor of choice to process your files and push them into the right file locations.

However, if you want to use variables, mixins, or anything else that needs to be "reference-able" across components, this won't cut it.

To figure out how to get this to work, I started combing through a few relevant threads, starting with this Github issue for Vue-CLI. Some extra hints came from this, and especially this.

The easy solution:

Thanks to the v3.11.0 update to the Vue CLI, there is now a built-in solution for handling mixed SCSS and SASS syntax imports within the same project. Make sure your CLI is updated, and then you can use the following format within your vue.config.js file:

module.exports = {
  css: {
    loaderOptions: {
      sass: {
        // Here we can specify the older indent syntax formatted SASS
        // Note that there is *not* a semicolon at the end of the below line
        data: `@import "@/styles/_variables.sass"`
      },
      scss: {
        // Here we can use the newer SCSS flavor of Sass.
        // Note that there *is* a semicolon at the end of the below line
        data: `@import "@/styles/_variables.scss";`
      }
    }
  }
}

That was easy! If you are curious, here is the PR that fixed this issue, and here is the updated guide section showing the new usage and examples.

If the above did not work for you, or you are stuck on an old version of the Vue CLI, you might have to use one of the hard solutions below:

The hard solution:

We can tap into webpack settings, by using the "chainWebpack" function in our vue.config.js file, which is a way to modify the default behavior of Vue's "behind-the-scenes" webpack configuration. Here are some relevant links:

Documentation for Vue config file
Vue's Documentation for using chainWebpack
Documentation for "webpack-chain", which is the module that powers the chainWebpack function.

Let's get to work:

First: Make sure vue.config.js exists

Depending on how you use the Vue CLI to create your project, the vue.config.js file might not actually exist yet. If it does not, go ahead and create it in the root of your project.

Now that we have a config file, let's move onto the solutions:

Solution - Option B - "Tap" into webpack settings using chainWebpack

This solution is provided by a Vuetify dev on a Github issue opened about this specific problem. I've modified it just a tiny bit to accept an array of files:

vue.config.js:

const globalSassFiles = [
    '~@/styles/_variables.scss',
    '~@/styles/App.scss'
]
module.exports = {
css: {
loaderOptions: {
sass: {
data: globalSassFiles.map((src)=>‘@import ”’ + src + ’”;‘).join(‘\n’)
}
}
},
chainWebpack: config => {
[“vue-modules”, “vue”, “normal-modules”, “normal”].forEach((match) => {
config.module.rule(‘sass’).oneOf(match).use(‘sass-loader’)
.tap(opt => {
return Object.assign(opt, {
data: globalSassFiles.map((src)=>‘@import ”’ + src + ’”‘).join(‘\n’)
});
});
});
}
}

This solution should work for most users, although a couple issues with it should be noted. For one, it injects your files into *every* used sass file, including all Vuetify files. This means that if you put something into Sass that is going to generate actual CSS code, it also ends up in all vendor CSS files. This also means an increased rebuild time for hot-reload, as touching one of those global sass files means all vendor files have to be reprocessed to re-inject your imports.

If you just need your Sass variables available to your own Vue template code, a better solution might be something like this:

const globalSassFiles = [
    '~@/styles/_variables.scss',
    '~@/styles/App.scss'
]
module.exports = {
css: {
loaderOptions: {
sass: {
data: globalSassFiles.map((src)=>‘@import ”’ + src + ’”;‘).join(‘\n’)
}
}
},
chainWebpack: config => {
[“vue-modules”, “normal-modules”, “normal”].forEach((match) => {
config.module.rule(‘sass’).oneOf(match).use(‘sass-loader’)
.tap(opt => {
delete opt.data;
return opt;
});
});
}
}

As you can see, "vue" has been removed from the match array, and for anything other than "vue" (template files), the import data is removed.

Solution - Option C - Use "Sass-Resource-Loader"

"Sass-Resource-Loader" is a loader plugin for Webpack that is specifically designed for injecting SASS variables as globals across imports. It even has a guide for using with vue.config.js and the Vue CLI 3+, which you can find here.

To get the same results as above, but with this solution, this is what my vue.config.js looks like:

const globalSassFiles = [
    './src/styles/_variables.scss',
    './src/styles/App.scss'
]
module.exports = {
chainWebpack: config => {
const oneOfsMap = config.module.rule(‘scss’).oneOfs.store
oneOfsMap.forEach(item => {
item
.use(‘sass-resources-loader’)
.loader(‘sass-resources-loader’)
.options({
resources: globalSassFiles
})
.end()
})
}
}

Final notes:

If none of this worked for you, my advice would probably be to either open a Github issue with Vue or Vue-loader, or use webpack outside of vue.config.js (either by switching to Vue CLI 2, which exposes more webpack stuff, or by setting up webpack from scratch outside of the Vue CLI). A great tool to use to try and determine how Vue is internally using Webpack is the "vue inspect" command. For example, to see what webpack config is used for production builds, and save it to output.txt, use:

vue inspect -v --mode production > output.txt

Finally, on a personal note, I would have to say that this whole post exemplifies one of the problems with "magic" tools like the Vue CLI, which abstract away how things are actually working below the surface. It really shouldn't be this difficult to tweak behavior, but it is. There is a great related thread on the balance between ease of use and exposing controls when it comes to the Vue CLI, and this reponse in particular is something that resonates with me.

Google Data Studio - Custom Toggl Time Tracker Connector

Joshua Tzucker — Fri, 05 Jul 2019 16:01:05 GMT

Project status: Final stages

Source code: https://github.com/joshuatz/gds-toggl-connector

Installation:

This is not yet live; when it is published I will update this post with the link to install it. In the meantime, if you would like to notified when this goes live, you can fill out a form here.

Once this connector is live, the installation process should be as easy as clicking a link, and then copying and pasting your Toggl API user token into the setup page.

Background:

I currently use a service called "Toggl", an online time tracking software, to keep tabs on how I spend my time; both for personal and business purposes. I'm not alone; with 1.6+ million users and over 70,000 businesses enrolled, Toggl is definitely a popular time tracker. As a detail oriented, technology-obsessed person, I also have been milling around the idea of building a personal "dashboard" using Google Data Studio (GDS), where at a glance I could see how much time I had worked on various projects in the current week, how many website visitors this site got, and things like that. For those unfamiliar, Google Data Studio is a (free) online reporting and visualization platform, similar to Tableau or Microsoft Power BI. It lets you create and share interactive reports and widgets that pull from multiple massive data sources.

Toggl would be a big part of this personal dashboard, but I was disappointed to find that they did not have a "connector" available to pull in data from Toggl into Google Data Studio. However, being who I am, and loving a challenge, I thought this was a great opportunity to see what went into creating a data connector, and try my hand at creating it from scratch.

Implementation:

Having just recently tried out TypeScript and being impressed with how it worked and kept large complex codebases tidy, I figured this would make a good first project to try it with. So the first part of this project was simply to get TypeScript to play nicely with Google Apps Script (the engine that powers GDS connectors), which was its own ordeal, and has a write-up here. After I had figured that out, it wasn't actually all that difficult to build out, although I ran into a lot of "gotchas" and strange undocumented bugs of the GDS connector library, which I have written up here.

Of course, the main crux of the connector is fetching results from the Toggl API and then translating the results into GDS formatted rows. The Toggl API is well documented, but unfortunately does not have TypeScript definitions available, so that was another thing that I had to code out from scratch.

I have not yet deployed this connector in an "official" capacity, but once I have and it has been approved, I will update this post.

Marketing Applications:

I filed this project under "marketing" for several reasons. For one, this shows my skills working with data sets and data transformation, both of which are a key part of the digital marketing eco-system. Google Data Studio is also, primarily, a marketing dashboard service, and I could have just as easily have built this as a connector for Tableau instead of GDS.

Finally, Toggl itself can be very useful for marketing-specific businesses. For example, an advertising agency might use the "client" grouping feature to track their billable time per client, then pull the data into a GDS report that combines the time spent per client with KPIs, to prove the value of their time and encourage clients to request more billable hours. Or, for a business that does not bill by the hour, or an in-house department, you could use it to track what type of tasks are burning more time and pull insights like "QA for creatives is taking 2x as long this month as last month; is that something we need to look into with our design department?".

Custom WordPress Theme and PHP Helpers: joshuatz-wp-theme

Joshua Tzucker — Mon, 01 Jul 2019 23:36:31 GMT

Source Code:

You can find the full theme here: https://github.com/joshuatz/joshuatz-wp-theme

NOTE: This repo serves mainly as a host for deployments and to showcase my theme building abilities. I'm not planning on releasing this theme into the WordPress theme directory, as I really only built it with myself in mind, and continue to constantly tweak it to suit my own personal needs.

Details:

I've been meaning to post this for a while now; this custom WordPress theme has actually been in use for a while now, for over half a year. Originally I built this site on the Blogger platform (Google's point-and-click blog platform), then moved to static HTML files, then to some simple PHP, and finally to WordPress, where I built and implemented a custom theme that would allow me to do whatever I like with the site.

Although WordPress is not necessarily liked by all developers, and I understand some of their frustrations, I enjoy the flexibility that the platform allows. For example, I've built a lot of custom code into my theme that tries to help SEO by keeping low quality posts from being indexed; if my code detects that a post is a placeholder (less than x number of characters) or a "virtual" post that immediately redirects, it will set a flag to stop Google and other search engines from indexing it. Another feature I implemented that I really like is automatic disclaimers based on date; if a post was created longer than [x] number of years ago, my code will automatically add a disclaimer at the top notifying users that the post content might be out of date.

I've also created a "helpers" PHP class, that abstracts a lot of code to solve annoyances and peculiarities with WP into helpful methods. For example, a single function to get the age of a post. You can find that file, helpers.php, here.

Developing a Google Data Studio Connector - Tips and Gotchas

Joshua Tzucker — Mon, 01 Jul 2019 05:11:58 GMT

I'm working on building a Google Data Studio (GDS) Connector; for those unfamiliar, Google Data Studio is a free BI (Business Intelligence) reporting platform, similar to Microsoft BI or Tableau. A connector, or "Community Connector" (a connector not built by Google) is an Apps Script powered bundle of code (Javascript) that enables Google Data Studio to connect to any third part service, by acting as the relay in-between.

As I've been working, I've been more than a little frustrated by the lack of in-depth documentation and troubleshooting guides available, so this post is both a way for me to fill in some of those gaps for others, as well as a way to vent my anger :)

Tip: Review the community connectors repo

If you get lost or stuck, and are wondering "how have others done this?", you might find it helpful to check out the "community-connectors" repo on Github. These are connectors that various people have developed, that are open-sourced and varied in usage.

Tip: Testing new code without deploys

If you want to test how new code actually works with GDS, you might be tempted to either reconnect to Data Studio or create a brand new deployment. Both are actually unnecessary if you just want to test for just yourself and are not pushing to production. When you click "Deploy from manifest..." in the "Publish" menu dropdown, that alone will actually push your code to any connectors that were setup by clicking the special link under "Latest Version (Head)".

So, for example, if I've just added a new "console.log()" line to my code and want to instantly see the new message appear in StackDriver, I would add the line of code, save, then go to "Menu -> Publish -> Deploy from manifest", and then, in another tab, simply refresh a Google Data Studio report that uses the connector.

Tip: Use TypeScript

Due to the complexity of writing a GDS connector and the amount of different types involved, I recommend using TypeScript, especially if you are used to using statically typed languages. I started writing my connector in vanilla Javascript, realized how complex things were getting, and rewrote it quickly into TS. During the rewrite process, I caught several bugs in my code purely through looking at TypeScripts type-checking errors. That is not to say that TS will prevent you from writing broken code, but I find it helps to keep things orderly and avoid what I would call "silly" programming mistakes. Part of this is personal preference as well.

I would only advise this if you have at least a little familiarity with TypeScript, or are looking to learn and have the extra time for it.

For how to use TypeScript with GDS: I've already written a guide for using TypeScript with GAS projects, and it applies to Google Data Studio Connectors as well.

Tip: Use TypeScript definition files

Even if you are not using TypeScript, one of the awesome things about VSCode is that you can still get the intellisense / auto-suggest capability to take advantage of TypeScript definition files. Just make sure you have a package.json file, and install "@types/google-apps-script". Once you do that, even if you are writing your connector in vanilla JS, you should be able to take advantage of the improved intellisense:

Tip: Additional resources:

There are a bunch of great resources out there for help on building a connector, beyond the official guides.

Here are some in particular that stood out to me:

Looking at code in the "community-connectors" repo.
"Building a custom Google Data Studio connector from A-Z" by Jan Bajena
This massive compilation of GDS resources by @Helpfullee.
The "DataVis DevTalk" series on Youtube by Google Cloud Platform (especially talks by @_mkazi_)

Gotcha: Unexpected data formatting rules

I'll just go ahead and say it: the way Google Data Studio expects data to be formatted is... well... a little strange. The format of the data in a field that you return to GDS is usually referred to as a "semantic type". The GDS docs have a nice table of the possible formats here.

Noticed anything odd about that table? I sure did! There are several random semantic types that, despite always holding an integer value with no zero-padding, such as "QUARTER" and "DAY_OF_WEEK", expect a string.

Furthermore, there is no field to hold a full datetime/timestamp value! The date field with the highest precision is "YEAR_MONTH_DAY_HOUR", and that only gets as precise as the hour. If you really wanted to pass a full timestamp to GDS, you could pass an epoch / unix timestamp in a NUMBER field, and then leave it up to the user to format it into any date format they would like with a "calculated field" and the "TODATE()" function.This solution also allows you to ignore timezone discrepancies, as the epoch does not need to take those into account (at least not for the purpose of passing it to GDS for their TODATE to handle).

EDIT: On 09/09/2019, they added support for "YEAR_MONTH_DAY_SECOND" (format: YYYYMMDDHHMMSS), which is a massive improvement considering that previously the highest precision date-time field was a specific hour (YEAR_MONTH_DAY_HOUR). Thanks GDS!

Gotcha: Debugging, Logging messages and errors

Debugging:

There isn't really a way to use the built-in GAS debugger as the connector runs, such as when a user navigates a report that uses the connector within Google Data Studio, or runs through the configuration setup. However, you could create functions that "mock" those actions, and then debug them by selecting the function in the "Select function" dropdown, and hitting the "debug" button.

Logging:

Since there is no real way to "debug" a community connector as it runs, you will likely find yourself putting in lots of print statements when you are trying to track down an error. If you are not seeing your messages show up, check how you are logging them.

The old school way is to use "Logger.log(message)", but that only works if your code is initiated directly within the Apps Script IDE. In general, you should avoid Logger, and instead use "console.log()". Calls to console methods log to StackDriver, which you can find by going to "view -> StackDriver Logging" inside the Apps Script IDE.

Also, note that the syntax for "console.log()" in GAS is not exactly the same as in your browser. Here is the reference page.

Gotcha: Timezones

There are a few things to note about using Timezones with GDS. First, both the GDS platform and the Apps Script environment use UTC standard time (aka GMT) as the default. So you will need to take that into account when returning date/time fields.

Timezones are complicated (and annoying), so I won't even attempt to try to pretend I can solve this for you, but I will give some possible approaches:

Always return everything in UTC-0. This is probably the safest solution, but requires coding offsets for every piece of data you are relaying from another API to GDS.
- Worse yet, if the API does not have a specific endpoint to query what the users timezone is, you might have to "sniff" it based on date/time strings...
Offer users a configuration setting where they can enter their timezone in hours offset from UTC, then inside your connector, add or subtract that number from every date field. This is the approach used by the openweathermap community connector.
If the API you are requesting data from allows for you to specify a timezone, give it UTC-0 as the desired format, and then you won't need to offset the data it returns before passing to GDS.

Gotcha: Promises, async/await

Here is the deal; yes, you can actually use promises and async/await inside Google Apps Script (and thus Google Data Studio connect0rs), but, you probably don't want to.

The issue is that pretty much all required functions are synchronous in nature, and are expecting to be returned a non-promise value. For example, if you are wrapping API requests in promises, there is no way for you to return the value inside the promise to getData, and then return it to GDS. You can't make getData async to use await inside it, so you might be tempted to write:

function getData(request){
    let apiResponse;
    myApiPromise.then((result)=>{
        apiResponse = result;
    });
    return apiResponse;
}

...but in this case, your promise would start to execute, and then while it is still executing, your return statement would hit the return line and instantly return "undefined" to GDS. This would actually trigger a "column number does not match request" error (see this gotcha item).

If you insist on using promises, there are just a few steps to take to get them working.

Add the es6-promise polyfill to your project. The easiest way is to simply copy and paste from the minified source into an Apps Script code tab.
Since es6-promise relies on setTimeout, which does not exist in the GAS environment, you need to manually polyfill that as well. Copy and paste this code into your project:
1. ```
function setTimeout(cb, ms) {
    Utilities.sleep(ms);
    cb();
}
```
Done! You should now be able to use polyfill and async/await just like normal! Just don't use them in places where they are not expected.

Gotcha: The order of requested fields

FIXED! You can now ignore this section.

This issue has been addressed and actually fixed by GDS :)

You can find their PR mentioning a patch here.

I'll leave the below section for historical reasons, or if this issue ever resurfaces.

Previously encountered issue:

When GDS requests a set of data, there are three places where field order has the potential to change; in the request.fields, in the order that you return schema in the getData() return object schema property, and the order you return columns in the return object rows->values array. The first one does not really matter, but what does matter is that your schema field order and rows column order match, or else it will throw an error, usually "Failed to fetch data from the underlying data set".

Here is the "gotcha"; the way that Google recommends you build schema in response to a getData request is actually very likely to cause an issue with this. They recommend that you use getFields().forIds(), and pass in the array of field IDs that were requested by GDS. The catch? This is not documented, but I caught forIds() returning fields in an order that did not match the order of the array passed to it!

Here is an example. Take a look at this code:

function getData(request) {
    // Raw request field ids - ['day','time','cost',...]
    var requestedFieldIds = request.fields.map(function (field) { return field.name; });
    // Should return special fields obj collection
    var requestedFields = getFields().forIds(requestedFieldIds);
    // Map back to an array of field ids - ['day','time','cost',...]
    var requestedFieldsMappedBackToIds = requestedFields.asArray().map(function (field) { return field.getId(); });
myConsole.log({
    "Raw Request Ids": requestedFieldIds,
    "GDS getFields().forIds() mapped back to IDs": requestedFieldsMappedBackToIds
});

// ... rest of getData() left out
}

One might assume that both arrays of field IDs should be identical in my console.log() result. After all, to generate the second array, the first ordered array was used as the input. But...

{
    "Raw Request Ids": [
        "billableMoneyTotal",
        "entryDescription",
        "time"
    ],
    "GDS getFields().forIds() mapped back to IDs": [
        "entryDescription",
        "billableMoneyTotal",
        "time"
    ]
}

Argh! forIds() randomly swapped the order of two of the columns! Because I was using the variable requestedFieldIds to generate a response to getData() in later code, my connector was failing because my schema had the flipped order, but my rows did not. Definitely something to be aware of (and would be great if Google could document somewhere). So basically, the rule of thumb should be to not rely on the order of fields/columns requested in getData(request), but instead focus on your result.schema and rows.rows[{values[]}] columns matching.

Gotcha: Unhelpful errors

"The number of columns received in the data returned from the community connector does not match the number of columns requested by Data Studio"

Despite this error being one that you are most likely to encounter, there is very little detail about it on the internet, and virtually none from Google's own documentation. This StackOverflow question is one of the few search results (although maybe now this page will come up!).

This error message is directly linked to whatever you are returning from "getData()". As the error text would imply, Google is saying that the number of columns returned by your code in getData does not match the number of columns that it requested, which you are supposed to read by checking the contents of the "request", the only argument to getData.

Let's walk through an example scenario. If I use "console.log(request)" inside "getData(request)", this is what the request object looks like in our example case:

{
    "configParams": {
        "workspaceId": "9234225"
    },
    "dateRange": {
        "endDate": "2019-06-29",
        "startDate": "2019-06-29"
    },
    "fields": [
        {
            "name": "day"
        },
        {
            "name": "time"
        }
    ]
}

The important thing to note in the above request from GDS is that there are exactly two fields requested, "day" and "time". Now, in our return object, both returnObject.schema and returnObject.rows[x].values, there should be exactly two columns returned by our code. Here is an example that would fail with the "number of columns does not match" error for the above request:

function getData(request){
    return {
        "cachedData": false,
        "schema": [
            {
                "dataType": "STRING",
                "name": "day",
                "label": "Date",
                "semantics": {
                    "conceptType": "DIMENSION",
                    "semanticType": "YEAR_MONTH_DAY"
                }
            },
            {
                "dataType":"STRING",
                "name":"entryDescription",
                "label":"Entry Description / Title",
                "semantics": {
                    "conceptType":"DIMENSION",
                    "semanticType":"TEXT"
                }
            },
            {
                "defaultAggregationType": "SUM",
                "dataType": "STRING",
                "name": "time",
                "description": "Logged Time",
                "label": "Time",
                "semantics": {
                    "isReaggregatable": true,
                    "conceptType": "METRIC",
                    "semanticType": "DURATION"
                }
            }
        ],
        "rows": [
            {
                "values": [
                    "20190629",
                    "Entry Alpha",
                    "1245"
                ]
            },
            {
                "values": [
                    "20190629",
                    "Entry Bravo",
                    "1981"
                ]
            },
            {
                "values": [
                    "20190629",
                    "Entry Charlie",
                    "1570"
                ]
            }
        ]
    }
}

Did you spot the error? I returned an extra field, "entryDescription", which was not requested! Here is a different returned object that should satisfy the request and make the error go away.

function getData(request){
    return {
        "cachedData": false,
        "schema": [
            {
                "dataType": "STRING",
                "name": "day",
                "label": "Date",
                "semantics": {
                    "conceptType": "DIMENSION",
                    "semanticType": "YEAR_MONTH_DAY"
                }
            },
            {
                "defaultAggregationType": "SUM",
                "dataType": "STRING",
                "name": "time",
                "description": "Logged Time",
                "label": "Time",
                "semantics": {
                    "isReaggregatable": true,
                    "conceptType": "METRIC",
                    "semanticType": "DURATION"
                }
            }
        ],
        "rows": [
            {
                "values": [
                    "20190629",
                    "1245"
                ]
            },
            {
                "values": [
                    "20190629",
                    "1981"
                ]
            },
            {
                "values": [
                    "20190629",
                    "1570"
                ]
            }
        ]
    }
}

Now, obviously, you would not want to be blindly returning hardcoded objects like this; your getData function should inspect every request, check what fields are needed, and dynamically construct the proper return object based on those fields. As such, your getData function is likely to be the most complicated method in your project

If the above did not solve your problem:

Now, the reason why I put this error in the "Gotcha" section is that the above example is usually why this error appears, but it will actually appear if getData() returns ANYTHING other than what matches the request. What I mean by this, is that this code will trigger the error:

function getData(request){
    return false;
}

Although Google's error about the column number not matching is technically correct in this case (0 columns !== number > 0 in request), it is not helpful. They really should have a separate error for when getData returns a completely malformed response, that has 0 columns, is not an object, is lacking the "rows" object property, etc. Something like "getData response does not match expected format".

The above code is an extremely simplified example; I actually made this mistake myself, and here are a bunch of ways you can end up accidentally returning the wrong thing in getData and triggering this error:

Using promises or other async code in getData and trying to return from inside them. GDS will not see the final result, even if your promise succeeds, and it will see null/undefined/void as the returned object
Using a bunch of if/else blocks to return different objects based on the request, but forgetting to have a "default" or missing a scenario not covered by your if/else blocks
Not returning due to an error case (maybe in a try/catch set of blocks), but not throwing an exception to replace the "column number" error.
- You should use "cc.newUserError()" to throw a user-facing error. See the Google Data Studio guide on handling exceptions for details.

Gotcha: Publishing and sharing your connector

So, your connector is now functional and you are ready to share it with others. Deploying to your own account has been as easy a few clicks, so publishing and sharing the connector with others should be relatively straightforward too, right? Not so fast...

"Open Source" Community Connector vs Partner Connector

The first thing to note is that GDS offers two very distinct paths forward for "publishing" your connector in a way that it will show up in the public connector gallery (here). A table comparing these two options can be found on the "Publishing Overview" connector guide page. However, I would like to add some additional commentary here that I feel obligated to share.

Appsscript.json sources array - requirement for both options

Regardless of which publishing option you pick, there is a requirement that is easy to miss at first glance. In your appsscript.json meta file, there is a property called "sources", which should have an array of strings corresponding to various sources that your connector uses. The catch here is that you should actually be only using source strings that exist as part of a source enum, which is maintained in a separate repository, the "Data Registry Repository". So if you are building a connector that uses a brand new source that no other connector uses, you will need to open a PR into that repo to add the new source, before you publish your connector. See this page for details.

The "Open Source" Community Connector Option is... unusual...

First, you can find the full list of requirements for publishing the connector as an open source community connector here. What I would like to add, is that it is a little odd how this process works.

Community Connectors are maintained in a single mono-repo, googledatastudio/community-connectors, which are apparently synced across automatically to the connector gallery. To add your connector, you would first make sure it meets all the requirements, then make a pull-request, like this one, to add your connectors code to the repo. It also looks like one of the requirements to get your PR approved is to have signed the Google CLA (Contributor License Agreement).

Once the PR is approved, and your connector is in their repo, Google will start handling pretty much all parts of your connector; they are responsible for deployments, users are instructed to open issues with the connector on their repo instead of yours, and the only way to update the code is to make a new PR against the repo and wait for Google to merge and then redeploy. Another strange thing is that this whole workflow seems to be currently handled by just one Google employee ("diminishedprime"), but it looks like he is doing an impressive job responding to all the PRs and even taking time to give pointers to devs who are missing things on their PRs. Still, it seems like Google can afford to lend some more employees to cover the GDS connector repo.

The main downside to this publishing route is that you are relinquishing a huge amount of control to Google; they control the deploys, code updates, or could remove your connector entirely if you fail to update it to address issues. It looks like this might not always be apparent to everyone, as Google has taken to warning devs of new PRs about the downside to publishing as a community connector:

This connector might be better served as a partner connector. Please see Publish A Community Connector for an outline of why you might pick open-source vs Partner.

We can certainly merge this is an an open-source connector, but keep in mind that our support for this is limited.

If you need to make a fix/update to the connector, you'll have to make a PR to this repo and we can only review those as we have time. If (potentially) long delays are okay with you (and your customers), we can move ahead with the review, but our guidance is anything business critical should go through the partner-path, or share deployment links directly.

Partner Connector Option

The partner connection option is more straightforward than the Open Source Community Connector option. The page of requirements and steps to take can be found here.

What about private publishing?

If you have built a connector that you just want to share with a few other people (or maybe internally at the company you work for), and not publish to the connector gallery, you might be tempted to just copy and paste the install link from the "deploy from manifest" popup in Apps Script editor. However, if you do this after starting from the default Apps Script environment, and send it to another user, they will likely get this error when they try to open the link:

The linked connectorId was invalid. Check the connectorId and try again.

OK, so how do we share a connector privately with other users? You might try next having the other use try adding the connector by ID instead of hyperlink, by going to the "select connector" page and then hitting the "DEVELOPERS" button on the upper right and pasting your deployment ID. But, if they did that, they would likely get this message:

The connector manifest could not be retrieved or is invalid. Check the connector and try again.

Hmm. Luckily this Google community post points us in the right direction. For both these errors, the issue is the permissions setting on the actual Apps Script project code. In order for other uses to "see" our manifest file, and actually load the connector, the permissions for the connector project needs to be set to give those users "view" privileges, at a minimum. You can change sharing permissions by clicking the "share" button in the upper right of the Apps Script IDE, or going to "File -> Share".

If you work at a company, and want to keep the connector internal, you could set the permission to share with "people at this organization". Otherwise, you should set it to "Anyone who has the link can view", or, individually add the email addresses of everyone you want to share the connector with, and give them view access. Note that giving users view access technically means that they can view the source code of the connector, but they would need to know the URL of the script, whereas you will be sharing a deployment ID or deployment URL with them.

Google Sheets: UNICODE() Char Code to UTF-8, Decimal and Binary

Joshua Tzucker — Tue, 25 Jun 2019 23:58:28 GMT

Warning: This post is going to get into the "dangerously nerdy" territory :)

Quick links to solution:

Convert to UTF-8 Binary
Convert to UTF-8 Decimal Char Code
- Pure formula
- Custom Script-based Formula

The issue - Google Sheets does not use UTF-8

I recently ran into a head-scratcher of an issue with Google Sheets, where a formula that used the output of the UNICODE() and/or CODE() functions was not matching with the expected value. After a bit of digging, I discovered that, although this is not really documented, most of the Google Sheets functions use UTF-32 (UTF-32BE) as the encoding scheme, whereas I was expecting UTF-8. This is made even more confusing by the fact that the Apps Script part of Google Sheets has an enum specifically for the UTF-8 charset.

To show this more clearly, here is table directly from Google Sheets, that show for various input characters, the CODE() function always returns the UTF-32BE value:

Why is this a problem? Well, most of the time it is not; as long as you use the same encoding scheme on both ends (input and output) it really doesn't matter. However, when you are feeding the result of UNICODE() into something where you don't control the decoder part, it becomes an issue. To be specific, I was running into this with writing a custom Base64 encoder function (see full details on that here).

Usually, when people talk about converting text to Base64, without even saying it, the implication is that we are going to be using UTF-8 as the encoding scheme. In fact, the default charset used in Apps Script with Utilities.base64Decode() is UTF-8, and specifying UTF-32 for decoding is not an option! If you pull up various online "text to base64" converters online, chances are that 99% of them use either plain ASCII or UTF-8 for the encoding/decoding of text.

In my case, I was feeding the result of UNICODE() into my base64 generator, which was causing it to spit out a base64 encoded string that could only be properly decoded with a decoder that uses UTF-32, which most decoders do not use.

Sidenote: JavaScript suffers the same issue!

This might surprise some people, but most JS engines do not use UTF-8 as the encoding scheme for strings - they use UTF-16!!! Similar to how I tested Google Sheets, this is easy to verify by checking the char code of a non-ASCII character:

And guess where this gets special mention... the MDN page on base64! I am not alone in realizing the UTF-* issue with Base64!

Why is UTF-8 so different?

The reason why UTF-8 is so different, and also why it is so popular, is because it is a variable length encoding scheme, with a minimum length of just one byte, and a maximum of 4 bytes wide. In comparison, UTF-16 is also variable, but has a fixed minimum of 2 bytes in width, and UTF-32 is not variable, and fixed at 4 bytes in width. The variable nature of UTF-8 means that each byte also has to have information in it on whether or not it is the last byte, which is the reason why you can't just plug the binary from a 2-byte-wide UTF-8 char into UTF-32, and vice-versa.

The solutions!

There is a lot to this, so I’m going to break down my overall solution (Text -> UTF-16 Unicode Decimal -> UTF-8 Decimal -> UTF-8 Binary) into smaller chunks, in case you need just one part of it.

Google Sheets: Text Character to UTF-8 Array

This section talks about converting UTF-16 to UTF-8. Originally, I thought Sheets was using UTf-16, but I am now aware it is UTF-32BE. However, the approach below still seems to work.

My starting point for this project was to try and convert a single UTF-16 charcode/codepoint to a UTF-8 codepoint. Luckily, the MDN page on base64 encoding and decoding had a very clean Javascript function I could use as a base.

Here is the relevant part of their Javascript code, that converts a UTF8 charcode to UTF-16:

nChr = sDOMStr.charCodeAt(nChrIdx);
if (nChr < 128) {
    /* one byte */
    aBytes[nIdx++] = nChr;
} else if (nChr < 0x800) {
    /* two bytes */
    aBytes[nIdx++] = 192 + (nChr >>> 6);
    aBytes[nIdx++] = 128 + (nChr & 63);
} else if (nChr < 0x10000) {
    /* three bytes */
    aBytes[nIdx++] = 224 + (nChr >>> 12);
    aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
    aBytes[nIdx++] = 128 + (nChr & 63);
} else if (nChr < 0x200000) {
    /* four bytes */
    aBytes[nIdx++] = 240 + (nChr >>> 18);
    aBytes[nIdx++] = 128 + (nChr >>> 12 & 63);
    aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
    aBytes[nIdx++] = 128 + (nChr & 63);
} else if (nChr < 0x4000000) {
    /* five bytes */
    aBytes[nIdx++] = 248 + (nChr >>> 24);
    aBytes[nIdx++] = 128 + (nChr >>> 18 & 63);
    aBytes[nIdx++] = 128 + (nChr >>> 12 & 63);
    aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
    aBytes[nIdx++] = 128 + (nChr & 63);
} else /* if (nChr <= 0x7fffffff) */ {
    /* six bytes */
    aBytes[nIdx++] = 252 + (nChr >>> 30);
    aBytes[nIdx++] = 128 + (nChr >>> 24 & 63);
    aBytes[nIdx++] = 128 + (nChr >>> 18 & 63);
    aBytes[nIdx++] = 128 + (nChr >>> 12 & 63);
    aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
    aBytes[nIdx++] = 128 + (nChr & 63);
}

And, here it is replicated in Google Sheets, as a formula:

=IF(UNICODE(A2)<128,UNICODE(A2),IF(UNICODE(A2)<2048,{(192+BITRSHIFT(UNICODE(A2),6)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<65536,{(224+BITRSHIFT(UNICODE(A2),12)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<2097152,{(240+BITRSHIFT(UNICODE(A2),18)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<67108864,{(248+BITRSHIFT(UNICODE(A2),24)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},{(248+BITRSHIFT(UNICODE(A2),30)),(128+BITAND(BITRSHIFT(UNICODE(A2),24),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))})))))

Or, if Google Sheets doesn’t like the nested IFS returning arrays of different sizes, this slightly longer formula uses SPLIT() to produce the arrays:

=IF(UNICODE(A2)<128,UNICODE(A2),IF(UNICODE(A2)<2048,SPLIT(JOIN("|",(192+BITRSHIFT(UNICODE(A2),6)),(128+BITAND(UNICODE(A2),63))),"|"),IF(UNICODE(A2)<65536,SPLIT(JOIN("|",(224+BITRSHIFT(UNICODE(A2),12)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|"),IF(UNICODE(A2)<2097152,SPLIT(JOIN("|",(240+BITRSHIFT(UNICODE(A2),18)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|"),IF(UNICODE(A2)<67108864,SPLIT(JOIN("|",(248+BITRSHIFT(UNICODE(A2),24)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|"),SPLIT(JOIN("|",(248+BITRSHIFT(UNICODE(A2),30)),(128+BITAND(BITRSHIFT(UNICODE(A2),24),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|"))))))

And here is confirmation that it works:

Google Sheets: Text to UTF-8 Binary

The formula above is cool, but it spits out the code points as an array, chunked in bytes, represented in decimal. What if we want it in binary? And how about as a single cell instead of as an array? That would be neat!

OK! Let's do it! First, lets convert the decimal array into binary. To do that, the easiest way will be to apply the DEC2BIN() function to each element of the array, converting from decimal to binary, which can be done by wrapping the above formula in =ARRAYFORMULA(DEC2BIN(...)). Easy!

And if we want it as a single cell output instead of an array? Just use JOIN()! Here is the final massive formula, to convert a text cell into UTF-8 binary:

=JOIN("","",ARRAYFORMULA(DEC2BIN(IF(UNICODE(A2)<128,UNICODE(A2),IF(UNICODE(A2)<2048,SPLIT(JOIN("|",(192+BITRSHIFT(UNICODE(A2),6)),(128+BITAND(UNICODE(A2),63))),"|"),IF(UNICODE(A2)<65536,SPLIT(JOIN("|",(224+BITRSHIFT(UNICODE(A2),12)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|"),IF(UNICODE(A2)<2097152,SPLIT(JOIN("|",(240+BITRSHIFT(UNICODE(A2),18)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|"),IF(UNICODE(A2)<67108864,SPLIT(JOIN("|",(248+BITRSHIFT(UNICODE(A2),24)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|"),SPLIT(JOIN("|",(248+BITRSHIFT(UNICODE(A2),30)),(128+BITAND(BITRSHIFT(UNICODE(A2),24),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))),"|")))))))))

Google Sheets: Text to UTF-8 Decimal (Pure Formula)

Based on what we have so far, we should just be able to shove the final large binary block generated by the above formula in to BIN2DEC(), right?

Error: Function BIN2DEC parameter 1 is too long. It is 24 characters; the maximum length is 10 characters.

Uh-oh! Well, that is disappointing. But here is an alternative; instead of using "character -> utf-16 -> utf-8 split into bytes as decimal -> split into bytes as binary -> binary", we can stop at the 3rd step, where we have an array of bytes as decimals, and add them together into a binary product. I have to admit that binary math is a bit beyond the scope of what I normally am comfortable with, but I managed to hack this together with a related answer I found for working around the BIN2DEC limit with Excel. Here is the final formula!

=SUMPRODUCT(--MID(JOIN("","",ARRAYFORMULA(DEC2BIN(IF(UNICODE(A2)<128,UNICODE(A2),IF(UNICODE(A2)<2048,{(192+BITRSHIFT(UNICODE(A2),6)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<65536,{(224+BITRSHIFT(UNICODE(A2),12)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<2097152,{(240+BITRSHIFT(UNICODE(A2),18)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<67108864,{(248+BITRSHIFT(UNICODE(A2),24)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},{(248+BITRSHIFT(UNICODE(A2),30)),(128+BITAND(BITRSHIFT(UNICODE(A2),24),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))})))))))),LEN(JOIN("","",ARRAYFORMULA(DEC2BIN(IF(UNICODE(A2)<128,UNICODE(A2),IF(UNICODE(A2)<2048,{(192+BITRSHIFT(UNICODE(A2),6)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<65536,{(224+BITRSHIFT(UNICODE(A2),12)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<2097152,{(240+BITRSHIFT(UNICODE(A2),18)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<67108864,{(248+BITRSHIFT(UNICODE(A2),24)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},{(248+BITRSHIFT(UNICODE(A2),30)),(128+BITAND(BITRSHIFT(UNICODE(A2),24),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))})))))))))+1-ROW(INDIRECT("1:"&LEN(JOIN("","",ARRAYFORMULA(DEC2BIN(IF(UNICODE(A2)<128,UNICODE(A2),IF(UNICODE(A2)<2048,{(192+BITRSHIFT(UNICODE(A2),6)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<65536,{(224+BITRSHIFT(UNICODE(A2),12)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<2097152,{(240+BITRSHIFT(UNICODE(A2),18)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<67108864,{(248+BITRSHIFT(UNICODE(A2),24)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},{(248+BITRSHIFT(UNICODE(A2),30)),(128+BITAND(BITRSHIFT(UNICODE(A2),24),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))}))))))))))),1),(2^(ROW(INDIRECT("1:"&LEN(JOIN("","",ARRAYFORMULA(DEC2BIN(IF(UNICODE(A2)<128,UNICODE(A2),IF(UNICODE(A2)<2048,{(192+BITRSHIFT(UNICODE(A2),6)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<65536,{(224+BITRSHIFT(UNICODE(A2),12)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<2097152,{(240+BITRSHIFT(UNICODE(A2),18)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},IF(UNICODE(A2)<67108864,{(248+BITRSHIFT(UNICODE(A2),24)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))},{(248+BITRSHIFT(UNICODE(A2),30)),(128+BITAND(BITRSHIFT(UNICODE(A2),24),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),18),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),12),63)),(128+BITAND(BITRSHIFT(UNICODE(A2),6),63)),(128+BITAND(UNICODE(A2),63))})))))))))))-1)))

And here is the confirmation it works:

Pretty neat, huh?

Google Sheets: Text to UTF-8 Decimal (Javascript)

As a "cleaner" solution, that does not use a huge formula, we can take advantage of Google Sheets custom formulas, which we can setup with Javascript in "Tools -> Script editor". Here is the code to use:

function utf8Char(input) {
    var decArr = strToUTF8Arr(input);
    var binString = '';
    for (var x = 0; x < decArr.length; x++) {
        binString += (decArr[x]).toString(2);
    }
    return parseInt(binString, 2);
}

// Adapted from https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding#Solution_2_%E2%80%93_JavaScript's_UTF-16_%3E_UTF-8_%3E_base64
function strToUTF8Arr(input) {
    var aBytes, nChr, nStrLen = input.length, nArrLen = 0;
    for (var nMapIdx = 0; nMapIdx < nStrLen; nMapIdx++) {
        nChr = input.charCodeAt(nMapIdx);
        nArrLen += nChr < 0x80 ? 1 : nChr < 0x800 ? 2 : nChr < 0x10000 ? 3 : nChr < 0x200000 ? 4 : nChr < 0x4000000 ? 5 : 6;
    }
    aBytes = [];
    for (var x = 0; x < nArrLen.length; x++) {
        aBytes.push(nArrLen[x] < 0 ? nArrLen[x] + 256 : nArrLen[x]);
    }
    for (var nIdx = 0, nChrIdx = 0; nIdx < nArrLen; nChrIdx++) {
        nChr = input.charCodeAt(nChrIdx);
        if (nChr < 128) {
            /* one byte */
            aBytes[nIdx++] = nChr;
        } else if (nChr < 0x800) {
            /* two bytes */
            aBytes[nIdx++] = 192 + (nChr >>> 6);
            aBytes[nIdx++] = 128 + (nChr & 63);
        } else if (nChr < 0x10000) {
            /* three bytes */
            aBytes[nIdx++] = 224 + (nChr >>> 12);
            aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
            aBytes[nIdx++] = 128 + (nChr & 63);
        } else if (nChr < 0x200000) {
            /* four bytes */
            aBytes[nIdx++] = 240 + (nChr >>> 18);
            aBytes[nIdx++] = 128 + (nChr >>> 12 & 63);
            aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
            aBytes[nIdx++] = 128 + (nChr & 63);
        } else if (nChr < 0x4000000) {
            /* five bytes */
            aBytes[nIdx++] = 248 + (nChr >>> 24);
            aBytes[nIdx++] = 128 + (nChr >>> 18 & 63);
            aBytes[nIdx++] = 128 + (nChr >>> 12 & 63);
            aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
            aBytes[nIdx++] = 128 + (nChr & 63);
        } else /* if (nChr <= 0x7fffffff) */ {
            /* six bytes */
            aBytes[nIdx++] = 252 + (nChr >>> 30);
            aBytes[nIdx++] = 128 + (nChr >>> 24 & 63);
            aBytes[nIdx++] = 128 + (nChr >>> 18 & 63);
            aBytes[nIdx++] = 128 + (nChr >>> 12 & 63);
            aBytes[nIdx++] = 128 + (nChr >>> 6 & 63);
            aBytes[nIdx++] = 128 + (nChr & 63);
        }
    }
    return aBytes;
}

Simply copy and paste this code into the script editor, save, and reload your spreadsheet. After that, you can use:

=utf8Char(A2)

...and get back the UTF-8 charcode in decimal!

2/28/2021 Update: I previously thought that Sheets was using UTF-16, but after a revisit to this post, I realized it is actually UTF-32BE; the post has been updated to reflect this.

Google Apps Script - Authorization in a cross-origin iframe

Joshua Tzucker — Sun, 16 Jun 2019 00:14:10 GMT

I've noticed more than a few questions on StackOverflow that have to do with using Google Apps Script or other authenticated google embeds in combination with iframes, and it has prompted me to do some digging. First and foremost, if you want to publish a Google Apps Script that can render within an iframe, that is actually surprisingly simple. To avoid any X-Frame-Options errors, simply use "HtmlService.XFrameOptionsMode.ALLOWALL". For example:

Code.gs


function doGet() {
    var template = HtmlService.createTemplateFromFile('output');
    return template.evaluate().setXFrameOptionsMode(HtmlService.XFrameOptionsMode.ALLOWALL);
}

output.html


<!DOCTYPE html>
<html lang="en">
<head>
<meta charset=“UTF-8”>
<meta name=“viewport” content=“width=device-width, initial-scale=1.0”>
<meta http-equiv=“X-UA-Compatible” content=“ie=edge”>
<title>Document</title>
</head>
<body>
<p>Hello World!</p>
</body>
</html>

And then simply load it as the source of an iframe:

Note that the above will only work if

A) The script is set to be authorized by your own account (executing as yourself), not the users,

Or:

B) The script is set to be authorized by the end user ("execute the app as user accessing the web app"), but they are logged in and already authorized by the time they load the page the iframe embed is on.

Dealing with Authentication / Logged Out Users - Intro

If your script touches any other Google product (email, Google sheets, etc.) there is a very good chance you are using the user's authentication to perform actions on their behalf. If you are using your user's authorization, your "deploy web app" dialog should look like this:

The problem with this, is that if you embed your App Script into an iframe, and a user visits the page it is embedded into and they are logged out of Google, then this happens:

If you open your devtools when this happens, you will also see this descriptive error message:

Refused to display 'https://accounts.google.com/ServiceLogin?passive=1209600&continue=https...' in a frame because it set 'X-Frame-Options' to 'deny'.

This is because Google realizes that your script is going to require an authenticated Google account to function, and redirects the iframe to the Google Login page. However, the Google Login page is set to not allow itself to be iframed into any site (thus the X-Frame-Options value of "deny"), so the browser blocks the iframe from loading the login page and errors out.

Obviously, this leads a poor user experience; how is the end user supposed to know what the grey sad face means, or how they should proceed?

Easiest solution - warn in advance:

The simplest solution is to just add some sort of warning message above or near where the script is embedded. Something like: "If the widget below appears as a grey box, please login to your Google account here [link to Google Login page] and then reload this page and accept permissions request."

Dealing with Logged Out Users with iframes - Advanced

Obviously, a lot of devs are not going to like just having a warning message about "seeing a grey box appear", and even I would have to agree that there feels like there should be a better way. Is there a way to detect if a user is not logged in and is going to get redirected inside the iframe to the non-functioning login page? Can we preemptively redirect them? This is where things started to get a little complicated and I tried some creative approaches...

If you don't care about what doesn't work, and are just looking for what I consider to be "the solution", which I list last, you can jump right to it by clicking here. Otherwise, read on to see all the ways I attempted this before getting to the solution!

Using postMessage()

One of my first ideas was to pass info about the success of the user getting authenticated, from the child iframe embedded App Script, to the parent window, which is wherever the widget is embedded. Although iframes are sandboxed so that the content can't be accessed from the parent (or vice-versa), postMessage is a method you can use for passing small snippets of data back and forth. My idea was to have my App Script immediately send a message to the parent window saying something like "Successfully loaded", and then the parent could start a timer, and if it does not receive that message within x # of seconds after the page loads, it knows the script got blocked from loading due to a login redirect. However, trying to implement even just the most basic of postMessage tests, I noticed none of the events were coming through, and I saw this in the console:

dropping postMessage.. was from unexpected window

What? What is that?

Oh... So it looks like Google Apps Script double iframes your content in. So really my custom HTML code is nested inside of Google proprietary wrapper code, which then loads into the iframe I actually embedded into my website. Google's custom wrapper iframe listens for postMessages, but only ones that match its own origin, so they are ignoring mine. Dang. I don't think there is an easy way around this. Let's try something else...

Create our own Ajax endpoint

My next idea: instead of only returning HTML, what if we set up our app script so it could respond with JSON in response to an ajax request? Then, from our website, we can use fetch(), and if the response is anything other than the JSON we are expecting, such as a redirect, we know the user has not been authenticated. Let's set this up:

function doGet(e) {
    if (e.queryString && e.queryString.indexOf('ajaxCheck')!==-1){
        // Respond back! With JSON.
        var response = {
            success: true
        }
        return ContentService.createTextOutput(JSON.stringify(response)).setMimeType(ContentService.MimeType.JSON);
    }
    else {
        // Display HTML in iframe
        var email = Session.getActiveUser().getEmail();
        var template = HtmlService.createTemplateFromFile('output');
        template.authedEmail = email;
        return template.evaluate().setXFrameOptionsMode(HtmlService.XFrameOptionsMode.ALLOWALL);
    }
}

As you can see, we can even reuse the same script, but change the response type (HTML vs JSON) based on the querystring used to request the script. This seems like it might work, but in practice...

What the heck? The request redirected? Well, it looks like for some reason (likely to do with character escaping or something like that), Google will not serve JSON from our script directly, instead it redirects the request to a shared common endpoint for echoing content, but with a unique querystring to tell the server what to serve:

https://script.googleusercontent.com/macros/echo?user_content_key=[unique-key]&lib=[?]

OK, well this is another wrench in the works. This would not be a deal-breaker on its own, but this is:

Access to fetch at 'https://script.google.com/macros/s/.../exec?ajaxCheck' from origin 'https://joshuatz.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

We just can't catch a break! Apps Script is not sending back the proper CORs header, so our requests are being outright blocked. There is basically no workaround for this - if we set fetch to use a mode of 'no-cors', than all we get back is an opaque response, which really tells us nothing. We also can't use a CORs proxy, like cors-anywhere, because we need to include credentials with our request, so Google can see if they are a logged in Google user or not, and that obviously won't work if the domains don't match.

This is starting to get a little frustrating, but trying to get JSON responses working actually made me stumble onto something promising...

The Solution!!! Using <script> tag with/without JSONP pattern

Script tags are not subject to the same CORs/CORB restrictions that are placed on AJAX, and I stumbled upon Google actually suggesting this method as a way to get script data into webpages here. They recommend using the JSONP pattern, but you could also use it to inject whatever JS content you want, which I'll explore as an option.

Option A) Recommended JSONP Pattern:

There are only a few unique things about the JSONP pattern and how it applies here:

You must define a callback function that is a window global, that will receive the response from the app script
Pass the string literal name of the callback function as a querystring key/value pair
In your app script, return JS that will call the function by name and pass the results

You can check out the example from the Google docs here, or I have prepared a complete working solution to the original problem we were trying to solve; detecting a completely logged out user and redirecting. My solution basically works as follows:

In HTML, have an overlay div that is positioned over the iframe embed, with text that tells user they need to authenticate, and button to do so
- Define callback function for JSONP that hides the "auth required" overlay if the script successfully executes
- Load the script through a <script> tag and pass my callback through the querystring
Script checks querystring to see how it should respond
- If JSONP, it returns JS that calls the callback with the authed users email
- If embed, returns success HTML to show in actual widget
- If accessed directly, shows message about being able to close tab, since auth was successful.

Here is what the result looks like:

You can see that on the left side, where I am in an incognito window and have not authorized the script, it detects so, and prompts me to authorize. In the window on the right, I am signed into Google and have authorized the script, so it hides the authorization prompt and displays the success HTML.

Minimal code to reproduce:

I've uploaded the full code used to create the above demo, which includes both the Apps Script side and the embed side, to Github - here. If you don't feel like visiting that, the two most important snippets of code are below:

Embed HTML:

<!-- Script loaded as iframe widget with fallback -->
<div class="appsWidgetWrapper">
    <iframe class="appsWidget" src="https://script.google.com/macros/s/SCRIPT_ID/exec?embedIframe"></iframe>
    <div class="loggedOut">
        <div class="loggedOutContent">
            <div class="loggedOutText">You need to "authorize" this widget.</div>
            <button class="authButton">Log In / Authorize</button>
        </div>
    </div>
</div>
<!— Define JSONP callback and authbutton redirect—>
<script>
function authSuccess(email){
console.log(email);
// Hide auth prompt overlay
document.querySelector(‘.loggedOut’).style.display = ‘none’;
}
document.querySelectorAll(‘.authButton’).forEach(function(elem){
elem.addEventListener(‘click’,function(evt){
var currentUrl = document.location.href;
var authPage = ‘https://script.google.com/macros/s/SCRIPT_ID/exec?auth=true&amp;redirect=’ + encodeURIComponent(currentUrl);
window.open(‘authPage’,‘_blank’);
});
});
</script>
<!— Fetch script as JSONP with callback —>
<script src=“https://script.google.com/macros/s/SCRIPT_ID/exec?jsonpCallback=authSuccess”&gt;&lt;/script>

Code.gs

function doGet(e) {
    var email = Session.getActiveUser().getEmail();
if (e.queryString &amp;&amp; 'jsonpCallback' in e.parameter){
    // JSONP callback
    // Get the string name of the callback function
    var cbFnName = e.parameter['jsonpCallback'];
    // Prepare stringified JS that will get evaluated when called from &lt;script&gt;&lt;/script&gt; tag
    var scriptText = "window." + cbFnName + "('" + email + "');";
    // Return proper MIME type for JS
    return ContentService.createTextOutput(scriptText).setMimeType(ContentService.MimeType.JAVASCRIPT);
}

else if (e.queryString &amp;&amp; ('auth' in e.parameter || 'redirect' in e.parameter)){
    // Script was opened in order to auth in new tab
    var rawHtml = '&lt;p&gt;You have successfully authorized the widget. You can now close this tab and refresh the page you were previously on.&lt;/p&gt;';
    if ('redirect' in e.parameter){
        rawHtml += '&lt;br/&gt;&lt;a href="' + e.parameter['redirect'] + '"&gt;Previous Page&lt;/a&gt;';
    }
    return HtmlService.createHtmlOutput(rawHtml);
}
else {
    // Display HTML in iframe
    var rawHtml = "&lt;h1&gt;App Script successfully loaded in iframe!&lt;/h1&gt;"
        + "\n"
        + "&lt;h2&gt;User's email used to authorize: &lt;?= authedEmail ?&gt;&lt;/h2&gt;";
    var template = HtmlService.createTemplate(rawHtml);
    template.authedEmail = email;
    return template.evaluate().setXFrameOptionsMode(HtmlService.XFrameOptionsMode.ALLOWALL);
}
}

Option B) Inject whatever you want:

Since the browser does not know the difference between the response from Google Apps Script and any old Javascript file, there really doesn't have to be anything special about the JS code that we return; it just needs to be valid JS and match the MIME type. If we want, we can return all kinds of hardcoded values, functions, or whatever we want. Here is quick example that sets a global window variable to a boolean TRUE value:

function doGet(e) {
    // Respond back with JS that sets global var
    var windowGlobalKey = 'userPassedAppsAuth'
    // Create script content
    var scriptText = "window['" + windowGlobalKey + "'] = true;"
    return ContentService.createTextOutput(scriptText).setMimeType(ContentService.MimeType.JAVASCRIPT);
}

If you embed your apps script as a <script> tag, in any code that runs after it loads, you could query "window.userPassedAppsAuth" and the result should be TRUE if the apps script successfully loaded and was authenticated. You could then take action based on that result, such as redirecting the user to login if the value is FALSE or undefined.

Another Solution: contentWindow.length

Thanks to this S/O answer, I learned of another approach that can be used as a solution, which relies on a specific quirk of how Google Apps Scripts returns HTML, and proxied window properties exposed on iframes.

For security purposes, most cross-origin Iframes block access to their content and properties from the parent window, but certain properties have their values proxied. One of those is window.length, exposed to the parent window as {iframeElement}.contentWindow.length. It is a numerical value, which corresponds to the number of frames within that window. Although it is normally an annoyance, the fact that Google wraps returned HTML from our script in its own iframe gives us a (roundabout) way to check for successful authorization: when our script successfully loads, this value should be 1, and if it fails to load, for example when our user has not authorized it yet, it will be 0.

Here is the relevant code for this approach:


// Give iframe some time to load, while re-checking
var retries = 5;
var attempts = 0;
var done = false;
function checkIfAuthed() {
    attempts++;
    console.log(`Checking if authed...`);
    var iframe = document.querySelector('.appsWidget');
    if (iframe.contentWindow.length) {
        // User has signed in, preventing x-frame deny issue
        // Hide auth prompt overlay
        document.querySelector('.loggedOut').style.display = 'none';
        done = true;
    } else {
        console.log(`iframe.contentWindow.length is falsy, user needs to auth`);
    }
if (done || attempts &gt;= retries) {
    clearInterval(authChecker);
}
}
window.authChecker = setInterval(checkIfAuthed, 200);
document.querySelectorAll(‘.authButton’).forEach(function(elem){
elem.addEventListener(‘click’,function(evt){
var currentUrl = document.location.href;
var authPage = ‘https://script.google.com/macros/s/SCRIPT_ID/exec?auth=true&amp;redirect=’ + encodeURIComponent(currentUrl);
window.open(authPage,‘_blank’);
});
});

Solutions that would work, but take a lot of effort:

Here are some solutions that would work for making sure a user gets redirected to Google login if they are logged out and visit your site with your widget embedded. However, all of these would take some considerable time, skills, and possibly extra paid services, like paid hosting, to implement.

Random IDs + webhook to server

This idea work as follows: On the load of your webpage, you create a completely random string (maybe timestamp + random output) and append it as a query string to the iframe source that loads the App Script. In your app script, you retrieve it inside your doGet() function, and immediate use URLFetchApp to send it back to your website host. You would need to have an endpoint set up that URLFetchApp could use, maybe something like "example.com/authCallback.php".

When Apps Script pings your endpoint, you save the random ID to a database table, where it can easily be looked up. You also expose your own internal endpoint, which can be queried with an ID to see if it exists in the table.

Finally, back on your webpage, you have a short timeout delay, maybe 15 seconds. After the delay ends, you query the ID against your own database, to see if Apps Script has pinged you for that specific user yet. If it has not, you know that the script most likely failed due to the blocked login, and you can at that point redirect the user / display a popup / do whatever you want.

Integrate Google Sign-In to your website

This feels like a lame answer, because there is nothing unique about it, but I'm guessing that if you were to use Google Sign-In for Web as a requirement to entering your site, their SDK would have some sort of method that would let you check to see if the current user is authed. Of course, this has all sorts of other issues, including yet another thing that your user has to agree to and be comfortable sharing permission with.

Checking out a Chrome Crash Report on Windows

Joshua Tzucker — Thu, 13 Jun 2019 05:58:54 GMT

After recently updating Chrome, I ran into certain sites hard-crashing the entire application on certain actions. I'm not talking about the "this tab has crashed" message, I'm talking acts-like-a-kernel-panic, no error shown, every single tab and entire app window exists type crash. Understandably, I got frustrated, but I never like being one of those people that just complains without knowing why, so I wanted to see if I could dig a little deeper. Trying to use Chrome's dev tools panel wasn't much help as, well... it crashed along with the browser whenever this happened. So next I looked to using Chrome's crash reports.

Using Chrome's Crash Reports:

A quick Google search pulled up this SuperUser post, that says you need to enable the sharing of usage stats and crash reports with Google Corporate in order to use them, which I did not want to do. However, I noticed that even with that setting off, and and chrome://crashes/ showing "Crash reporting is disabled.", the actual crash dump files were being generated and saved anyways. I found them in "C:\Users\[USERNAME]\AppData\Local\Google\Chrome\User Data\Crashpad\reports", but your location may differ. They are saved as files with the "DMP" extension, which would indicate "Dump files", but they are actually "minidump" files, which I confirmed by checking the magic bytes (first 6 bytes read as "MDMP“§") and then referring to the readme for Crashdump and Breakpad, which are the Chromium tools that actually are responsible for saving and generating the files.

(EDIT: This helpful page from Chromium on Crash Reports cleared this up more. Breakpad is what used to be used, but is getting replaced by Crashpad, which is now used by default on Windows and Mac.)

Now how do we read these files? If you are a newbie to this part of software, like I am, you might be a little lost. The page from Chromium on decoding crash dumps makes it kind of sound like this is impossible to do on Windows and requires very specialized Unix tools. However, this is not true; minidump is a standardized dump format, actually created by Microsoft for Windows specifically, and there is more than just one tool to analyze them. On Windows, the default tool to use is WinDbg, which you might already have on your computer if you are a developer; it is available as part of the Windows 10 SDK. The full installation guide is here. WinDbg is also not the only program out there to analyze minidumps - for example, whocrashed.

Without any special setup, simply loading the minidump from Chrome into WinDbg on my machine let me see a lot of helpful details; the most helpful being the call stack. I could see that in every single of my 10+ crashes, the same exact function call was triggering the crash; blacklist::SuccessfullyBlocked(). This is a function that triggers when Chrome's anti-DLL-injection blacklist records that it successfully blocked a "rogue" DLL from injecting code into the browser. I'm going to have to do some more digging, but this is probably related to the real-time web browsing protection offered by my antivirus.

If you want a more in-depth experience with WinDbg, definitely read the docs on WinDbg, and at a minimum, you might want to enable symbol resolution by providing WinDbg with some symbol paths. For example, to pull in symbols for Windows API and Chrome, and I used this command to set the symbol file path:

.sympath srv*c:\symbols\chrome*https://chromium-browser-symsrv.commondatastorage.googleapis.com/;.symfix+ c:\symbols\ms

After which, you can use

.reload

and then

!analyze -v

to run a quick analysis.

Another option for trying to examine Chrome crashes is through logging.

Using Chrome Logs

To run Chrome with error logging on, where it logs critical errors to a specific file, use this command flag when you start the application:

--enable-logging --v=1

In Windows, you can save the flag with the shortcut for the application, or even save it is a separate shortcut, so you can choose at any time to launch the non-logging version of Chrome versus the logging one.

Once logging is set up, you can view the log file, likely at C:\Users\[USERNAME]\AppData\Local\Google\Chrome\User Data\chrome_debug.log

You can also use an application called Sawbuck (also made by Google) that can show you logging entries in realtime, and let you filter on levels. See the Chromium page on logging for details.

In my scenario, the log did not help or turn up anything useful, as the crash was actually happening so fast, nothing about the crash itself was getting written out to the log before the application exited completely.

Update: A further note about DLL injection

Another resource to check for which installed DLLs are potentially injecting code into Chrome and could cause a conflict, is this internal Chrome page - chrome://conflicts/. Mentioned here.

Using TypeScript with Google Apps Script and Google Ads Scripting

Joshua Tzucker — Sun, 09 Jun 2019 09:23:10 GMT

I just recently started a project that runs as a Google Apps Script (GAS), and as I also just recently tried TypeScript and loved it, I wanted to see if I could use it and compile it to code that could execute as a regular Google Apps Script. When you think about it, TypeScript is an excellent fit for Google Apps Script, as the Apps Script platform has limited debugging capabilities and uses tons of complex SDK-specific interfaces; TypeScript's static type checking and support for type declaration files make coding for Apps Script easier and less error-prone.

However, using TypeScript with GAS or Google Ads Scripting is not as easy as simply calling the TypeScript compiler/transpiler (TSC) to transpile your .ts to .js. True, Google Apps Script is an extension of Javascript and that should in theory mean that transpiled JS should work fine, but it has a few key differences that require building for it be treated differently, primarily how it treats function and variable scope, which you can read about at the bottom of this post.

Typescript Transpiling Options:

I've created a simple repo that shows each of these options with the same example code. You can find it here: https://github.com/joshuatz/typescript-to-gas-demos

Option A: Google's Recommendation - Using clasp

Clasp, or Command Line Apps Script Project, is a command line tool that lets you develop Apps Script code locally, in Javascript or TypeScript, and push via terminal to the Apps Script cloud, in the process magically getting transpiled and formatted for GAS. To begin using Clasp, the best place to start is here, or here if you are specifically interested in using it for TypeScript.

The trade-off for ease of use is:

Trust: You have to give the Clasp tool scoped permission to touch your Apps Script files, which is handled by requesting and then storing on your harddrive an OAuth token. Of course, you can revoke this at any time, and you can check the source code for Clasp itself to see what it does with it.
Increased "black-box" complexity: Although Clasp is open source, the amount of moving parts it adds to an Apps Script project and how it abstracts certain things so you don't have to worry about them, means that you might understand less about what is happening behind the scenes.
- For example, if you are using TypeScript with Clasp, when you use the command clasp push, behind the scenes it actually calls a tool called "ts2gas", which is a TypeScript to Google Apps Script transpiler, which in turn, uses the TypeScript Compiler API and a final (heavily modified) call to transpileModule, which is similar to running TSC. Then Clasp kicks back in and uploads the transpiled JS files to your Google Drive / Apps Script Project.
- With your code going through multiple steps like this, can you reliably predict what the transpiled JS will look like after being converted from TypeScript? Are you aware of all the limits of each step of the process?
Lack of compatibility with other Google Script environments
- I can find no evidence that Clasp can work with Google Ads Scripts (formerly AdWords scripting)

Note that neither Clasp nor ts2gas seem to import the Google Apps Script type definitions, so see my notes here on adding them.

I wanted to understand more about how TypeScript could be transpiled to GAS compatible GS code, which is the impetus behind this post and the further options I explore below.

If you are developing for a Google Data Studio community connector, you might want to use dscc-gen, which is a wrapper around Clasp that provides some convenience methods for connector specific work. I could not get it to work on Windows, which was also part of my research into how all these pieces fit together.

Option B: Use ts2gas directly

There is nothing preventing you from using ts2gas directly, instead of through Clasp. However you would need to add a little bit of wrapper code, since unlike TSC which can take files as inputs, ts2gas will only take a string, and also returns a string, not a file. You can find an example of this in the sample repo here, and I've also pasted the simple build script below:

const ts2gas = require('ts2gas');
const fse = require('fs-extra');
// paths
const inDir = ’./’;
const outDir = ’../output-gs/’;
// Iterate over TS files in this folder
fse.readdir(’./’,(err,fileList)=>{
fileList.forEach((fileName,index)=>{
if (/.ts$/i.test(fileName)){
let tsCode = fse.readFileSync(inDir + fileName,‘utf-8’);
// Send to ts2gas
let gasCode = ts2gas(tsCode);
// Write out transpiled GS
fse.writeFileSync(outDir + fileName.replace(‘.ts’,‘.gs’),gasCode);
}
});
});

Option C: Transpile without modules

The fastest and easiest way to quickly generate a GAS compatible export from TypeScript is to simply turn off modules, telling TSC that everything will be in a shared scope. This is a sample tsconfig.json that meets these requirements:

{
    "compilerOptions": {
        "target": "es3",
        "module": "none",
        "outDir": "./out",
        "strict": true
    }
}

These also (should) automatically change a few crucial things in your IDE (works best with Visual Studio Code). Most importantly, because it knows the output shares a scope, it lets you reference functions and global variables across files, without using imports/exports.

Option D: Transpile to single file (simple and advanced)

The main benefit of transpiling to a single GS file is that, if you chose not to use Clasp due to privacy concerns, having a single file output makes it easier to copy and paste directly into the online editor. The simple version of this example simply uses the outFile option of TSC to force the output to a single file:

{
  "compilerOptions": {
    "target": "es3",
    "module": "none",
    "outFile": "../output-gs/compiled.gs",
    "strict": true
  }
}

The advanced version uses a custom build script to remove places where the source file uses "import" or "export", so that you can use imports/exports in your source TS without it messing up the exported GS.

Using TypeScript definitions for Google Apps Script and Google Ads Scripting

First, you are going to want to actually install the TypeScript definition packages:

For Google Apps Script:
- Package: @types/google-apps-script
- Created from: DefinitelyTyped/types/google-apps-script
- Install: "npm install --save @types/google-apps-script"
- Add to top of TS: import 'google-apps-script'
For Google Ads
- Package: @types/google-adwords-scripts
- Created from: DefinitelyTyped/types/google-adwords-scripts
- Install: "npm install --save @types/google-adwords-scripts"
- Add to top of TS: import 'google-adwords-scripts'

Although, like any node package, you can install types globally with the -g flag, VSCode will not automatically pull from the global folder. The only way I know of to have it recognize a globally installed type module is to use the absolute path - for example, this worked:

import 'C:\\laragon\\bin\\nodejs\\node-v10.14.2-win-x64\\node_modules\\@types\\google-apps-script';

But of course, that is terrible practice, since it requires other devs to have the same global install path. Even if you are just developing for yourself, you might as well set up a package.json and locally install the type packages.

You might be wondering how this is handled as far as transpiling goes. Well, Clasp/ts2gas will automatically comment out your "import" lines when it is transpiled to JS, and if you are using a different option, you could manually comment them out by hand before running TSC, or use an automated build script that does so (see my options above for details).

Apps Script - JS Caveats - Scope and Hoisting

First, and please excuse any slightly inaccurate terminology, but Apps Script works a little differently than JS that runs on websites, or NodeJS environments. One of the key differences is how function & variable hoisting works. Normally, functions are hoisted to the top, along with global variables, meaning that even if you define a function at line 200 of your file, you can call it at line 1, because the interpreter "hoists" the function declaration and parses it first. This lets you do this kind of thing in plain JS:

Apps Script does this too, but with an important distinction as to what "scope" means and order of evaluation. With normal JS, functions are hoisted to the top of their scope, when the code block they are in is evaluated. In practice, this means that although separate script tags can share the same global scope, since they are evaluated separately as the page gets parsed top to bottom, functions declared in a separate script tag can only be called in a different script tag if where they are called from is either a "lower" script tag, or waits to call the function until where it is declared is evaluated by the JS engine. Since examples are much easier to understand than my rambling, here is a very practical example in plain Javascript:

<!-- Script A -->
<script>
    // call alpha before it is defined.
    alpha();
    function alpha(){
        console.log('running alpha');
        // call beta before it is defined.
        beta();
    }
</script>
<!— Script B —>
<script>
function beta(){
console.log(‘running beta’);
}
</script>
<!— Final —>
<script>
// Call alpha again
alpha();
</script>

In the example above, the first time alpha() runs, the script tag that the beta() definition is in has not yet been evaluated, so we get a fatal error. Then that script tag hits, and beta() gets defined, and automatically hoisted to the top of the scope (becoming window.beta()), and finally, the final script tag hits, calls alpha() again, and this time, although alpha is in a different script tag, both alpha() and beta() are now hoisted and can be called and alpha() runs without the fatal error. This is why order matters when adding script tags (either inline or calling external js files), and most web devs are very familiar with accidentally loading dependencies in the wrong order and getting undefined errors.

How does Google Apps Script differ in function and variable hoisting versus normal JS? Well, it is hard to find documentation on this, but from experimenting and what little public information there is, the conclusion is that AppScript actually considers different script files to not only share the same scope, but also be evaluated at the same time so function & var declarations are hoisted to the top, across files and regardless of order. Using the same example as above, slightly modified for the Apps Script Environment:

File-A.gs

test();
function test(){
  // call alpha before it is defined - in this file
  alpha();
  // call final before it is defined - in separate file Final.gs
  final();
}
function alpha(){
  Logger.log('running alpha');
  // call beta before it is defined - in separate file File-B.gs
  beta();
}

File-B.gs

function beta(){
  Logger.log('running beta');
}

Final.gs

function final(){
  // Call beta again
  beta();
}

If this was a webpage, and those were JS files getting called via script tags, you would expect all sorts of errors when test() gets called. For example, you would expect to get the same "beta is not defined" error, since beta is in a separate script file and should not yet be hoisted. However, because this is GS, we get no errors, and a perfect output:

[19-06-08 19:25:14:102 PDT] running alpha
[19-06-08 19:25:14:103 PDT] running beta
[19-06-08 19:25:14:103 PDT] running beta

So what exactly is going on? My suspicion is that Google Apps Script basically acts like it is joining all your GS files, no matter how many you have, into one single GS file that it then evaluates. Pretty much the only official documentation I have been able to find that backs this up is here:

You are not limited to a single server Code.gs file. You can spread server code across multiple files for ease of development. All of the server files are loaded into the same global namespace, so use JavaScript classes when you want to provide safe encapsulation.

The other reason why this is a logical conclusion is because it is basically a necessity given the way that the Apps Script online editor works. Unlike the web, where you can reorder <script> tags, you cannot control the order of .GS files within your project and thus, if the rule about sharing scope was not true, you would literally have to guess as to what order your files would be evaluated in and subsequently their functions hoisted.

LinkedIn Profile to JSON Resume Exporter - Extension and Bookmarklet

Joshua Tzucker — Thu, 06 Jun 2019 17:19:15 GMT

Installation:

You can now get this tool as a Chrome Extension, from the official webstore page.

Or, if you want to sideload it, you can grab a zip file from the latest release.

Source Code:

https://github.com/joshuatz/linkedin-to-jsonresume

What is the LinkedIn to JSON Resume Export Tool?

This is a project that I initially quickly threw together, mostly out of frustration that it didn’t already exist. As I outline in the “readme” file in my Github repo for this project, I wanted to export my LinkedIn profile to JSON Resume, which is an exciting standard for storing, sharing, parsing, and generating resumes based on a shared underlying data structure, or schema.

The official LinkedIn APIs are restrictive and require an approval process, and the manual data export option offered by LinkedIn can take up to 72 hours. I was frustrated by this fact, so I built this tool to instantly export a LinkedIn profile page to the JSON Resume standard, which appears in a little popup modal that you can easily copy and paste out of.

On 8/3/2019, I rewrote the tool from a bookmarklet to a browser extension (Chrome extension) to get around some restrictions placed by LinkedIn that were breaking the functionality of the bookmarklet version.

Features

Since the initial release as a simple bookmarklet, this project has seen some major improvements and gained additional features.

Rewritten as a browser extension, to work with CSP restrictions
Added support for downloading of JSON file
Reworked to simultaneously support multiple different schema versions
Added support for multilingual profiles, and ability to pick export language
Added vCard export (for import into address book)
Constant reworking to support changes in the LinkedIn endpoints, specifications, and restrictions

Demo

What I Used

JavaScript
TypeScript
JSON + JSON Schema
Webpack
Chrome extension APIs
Web APIs
Fetch
Vanilla JS + HTML + CSS
And more!

Embedding iframes - Static, Third-Party, and Dynamic Options

Joshua Tzucker — Sat, 01 Jun 2019 03:26:35 GMT

I feel like iframes are a very under-appreciated part of the web-development world. Especially as concern grows about the amount of invasive tracking and its negative effect on page performance, iframes continue to be an excellent way to force third-party code into sandboxes, where they can't interact with the host page (unless explicit data passing has been setup).

I personally use iframes somewhat often, but infrequent enough that I often forgot things like "can I force this site's widget into an iframe?", or "what is the fastest way to throw this into a publicly accessible iframe embed?". This post is a way for me to organize some notes on the topic and keep a list of resources related to iframe embeds, specifically for developers.

Table of Contents:

The Basics: Static HTML Upload
Third Party Generators
- JSFiddle
- CodePen
- JS Bin
Dynamic iframe generation: client-side and inline attributes
Generating iframe content server-side & some warnings

The Basics: Static HTML upload

I would be remiss if I didn't mention the default way to host up an iframe that others can embed; simply throw your code into a static HTML file (or generate with server-side code, more on that later), and shove the file onto your publicly accessible host (shared host, VPS, AWS bucket, etc). Lets pretend you uploaded it to the root of your domain, at example.com/widget-A.html. Then to embed, you could simply pull it in via the standard iframe element: <iframe src="/widget-A.html"></iframe>.

I use this simple method a few times throughout this site, so that I can pull some custom widgets into a WordPress page and not have to worry about conflicting CSS or messing with HTML escaping in WordPress post content.

This is always a good fall-back method to wrap anything in an iframe. However, if code relies on access to JS globals or things like window.location.href, you might need to add some extra code to explicitly pass data from the parent window to the embed HTML file. And of course this doesn't help at all really with dynamic iframe generation, where you want to put user submitted content within iframes.

Third-Party Generators:

If you are a web developer that likes to share widgets, you are probably already familiar with sites like jsFiddle and CodePen. However, you might not know that many of these sites allow for direct embedding, and some even allow you to natively use iframes as the embed type. However, please note that the intended purpose here is to embed code snippets to share with other developers or show off little demos. Embedding entire applications or exceeding what is considered "normal use" is likely against the TOS and is not a good idea anyway for anything beyond proof-of-concept demos and the such.

jsFiddle - An Interesting Lesson in Iframe Mitigation

Inside the JSFiddle editor, there is an "Embed" button, which lets you configure an embedded widget that you can copy the code for and paste into your destination of choice. You can easily configure it to generate an embeddable iframe:

The resulting Iframe can be set to just show the "result" of the fiddle, which is desirable if you are trying to show off a demo to users, rather than to other developers. However, JSFiddle injects a top menu bar into all Iframes with a tab selector and "Edit in JSFiddle" text, regardless of how many tabs are set to display:

You might have noticed that the iframe embed code generated by JSFiddle uses a src of "jsfiddle.net/{{user}}/{{fiddleID}}/embedded/result/", but the window.location.href in our fiddle shows that what actually loaded was "fiddle.jshell.net/{{user}}/{{fiddleID}}/show/{{theme}}". Those trying to get rid of the injected menu bar might try setting the iframe source manually to the "fiddle.jshell.net" URL, but would be disappointed to see that the menu bar still gets injected. What gives? How is JSFiddle doing this?

The answer is interesting, and outlines a good technique for blocking users from abusing an iframe embed system. First, JSFiddle uses a "wrapper" around the final result, which is to say that it double-nests iframes. So the setup looks kind of like this:

Level A: Your Website
- Level B: JSFiddle default IFrame Embed (<iframe src="jsfiddle.net/.../.../embedded/result"></iframe>)
  - [Automatic] Wrapper(html)
    - Tabs Bar (html)
    - Level C: Secondary iframe (<iframe src="fiddle.jshell.net/.../.../show/..."></iframe>)
      - Actual JSFiddle Result

Second, and this is the interesting part, it relies on the fact that when resources are loaded, either through AJAX or through resource tags like <script src="">, the browser automatically includes what website the request is coming from, as the "referer" header. If a request comes from within an Iframe, the referrer matches the host of the Iframe content, not the site that the Iframe is embedded on. JSFiddle is checking this, and if the request is coming from a website outside of its control, it knows that the wrapper/tabs/secondary iframe must not be injected yet, since the wrapper should be loaded inside an iframe that would send a JSFiddle controlled domain as the referrer. Since the tab bar is injected into a wrapper that contains the fiddle result in an iframe, rather than directly into the fiddle result, it also means that your fiddle code can't modify the wrapper, so you can't do something like set "display:none" on the tab bar through your fiddle code.

To summarize:

HTTP Request to fiddle.jshell.net/.../.../show/
- ...From a JSFiddle domain:
  - Response HTML will be your JSFiddle code
- ...From outside a JSFiddle domain:
  - Response HTML will contain injector code that creates wrapper, tabs, and secondary iframe that requests fiddle.jshell.net/.../.../show/ all over again.

I'm not going to go into detail on how to remove the top bar, since that isn't really something you should be doing, but I will point out that if you set your original Iframe embed src to fiddle.jshell.net/.../.../show instead of the default embed src, then the secondary iframe will actually have the same origin as the wrapper, which means that you can access and modify it with "window.parent.document" from inside your actual Fiddle code.

CodePen: Don't Mess with Us!

Like JSFiddle, CodePen also has an IFrame embed option. In fact, they pretty much use exactly the same methods for blocking abuse of the IFrame and removal of the menu and "Edit on CodePen" text:

Nested iframes: "codepen.io/.../embed/..." loads a wrapper, with the menu bar, which in turn nests a secondary iframe with the result at "s.codepen.io/.../fullembedgrid/..."
- Because the secondary Iframe has a different domain (subdomain), your CodePen code cannot touch the menu bar in the first, due to cross-origin policy.
CodePen checks the referer header on the result Iframe src

Here is where it is different. Whereas JSFiddle will re-embed the iframe if it detects a mis-matched referrer, CodePen will simply block your request to load the result entirely if it detects the referer not matching a CodePen owned host. If you try to set "s.codepen.io/.../fullembedgrid/..." as the source of an iframe that is not inside your fiddle, you will see this:

To be clear, this error message is misleading; the browser is sending a referer header, it just isn't one that CodePen wants to see. What this error message should really say is something like "We see what you are trying to do... cut it out!".

JS Bin: Ajax Fine, Browser Not

JS Bin is a little different than CodePen and JSFiddle in how it treats embeds. It looks like they do allow embeds, but only for Pro users. Also, instead of checking the referer header to block cross-origin result embedding, they are using X-Frame-Options, which is a cleaner and less error-prone way of blocking cross-origin embeds, but only blocks in browsers that support it. JS Bin has set X-Frame-Options on their "output" subdomain to "sameorigin", which as its name would imply, will block the embed if the origin if the frame does not match the origin of the page.

I noticed kind of a strange thing with JS Bin, which is that they are not enforcing a referer match and they actually are allowing cross-origin requests. This probably has to do with their live-streaming / hot-reloading features. Anyways, this strange combination of security means that this actually worked for me:

fetch('https://output.jsbin.com/{{ID}}').then(function(response){
    response.text().then(function(html){
        var iframe = document.createElement('iframe');
        document.body.appendChild(iframe);
        iframe.contentWindow.document.open();
        iframe.contentWindow.document.write(html);
        iframe.contentWindow.document.close();
    });
});

Advanced: Dynamic iframe content injection with client-side Javascript or inline attributes

One thing that you might not be aware of is that you can actually use iframes that show content that is not from an actual domain - e.g. dynamically generated content with Javascript or by inline attributes. There are tons of caveats to using iframes in this way though, and I'll cover them for each of the available methods to dynamically generate iframe content client-side. These are listed in order from most compatible but least friendly, to most friendly but least compatible.

"iframe.contentWindow.document.write" - Maximum cross-compatibility

This is pretty much the only solution that works for IE, since things like srcdoc and data-uri do not (or have very limited support), and I found it through this StackOverflow answer. This method works by creating a new iframe element that has the same origin as the current page and then streams raw HTML text into the document of that iframe with document.write(html):

function sameOriginIframeInject(html,target,append,cb){
    cb = typeof(cb)==='function' ? cb : function(){};
    var iframe = document.createElement('iframe');
    if (append){
        target.appendChild(iframe);
        iframe.style.width = '100%';
        iframe.style.height = '100%';
    }
    else {
        target.replaceWith(iframe);
    }
    iframe.addEventListener('load',cb);
    iframe.contentWindow.document.open();
    iframe.contentWindow.document.write(html);
    iframe.contentWindow.document.close();
}
// Example:
sameOriginIframeInject(‘<script>alert(“success!”);</script>‘,document.querySelector(‘#iframeWrapper’),true);

The downside to this method is that it pretty much negates the entire point of iframes in the first place; because the iframe shares origin with the parent, any code you put in the iframe can access your site, and vice-versa. However, standard iframe benefits of separately scoped JS variables and scoped CSS still apply, which is probably why this method still gets some use.

BONUS Script: Inject a cross-origin URL into a same-origin iframe. This will let you do some crazy stuff, like use iframe.contentWindow.print(), which would normally be unusable with an iframe that displays cross-origin content. Note that this method is very hackish and is likely to break on a large portion of sites.

function injectSameOriginIframeFromRemote(remoteSrcUrl, target, append, OPT_cb, OPT_useProxy) {
    var PROXY_BASE = 'https://cors-anywhere.herokuapp.com';
    useProxy = typeof(OPT_useProxy)==='boolean' ? OPT_useProxy : true;
    cb = typeof (OPT_cb) === 'function' ? OPT_cb : function () { };
    var iframe = document.createElement('iframe');
    if (append) {
        target.appendChild(iframe);
        iframe.style.width = '100%';
        iframe.style.height = '100%';
    }
    else {
        target.replaceWith(iframe);
    }
    var xhr = new XMLHttpRequest();
    xhr.onreadystatechange = function() {
        if (xhr.readyState === 4){
            if (xhr.status === 200){
                var rawHtml = xhr.responseText;
                var baseString = '';
                rawHtml = rawHtml.replace(/]+>/gim,'');
                rawHtml = rawHtml.replace(/<\/head>/gim,baseString + '\n' + '');
                iframe.contentWindow.document.open();
                iframe.contentWindow.document.write(rawHtml);
                iframe.contentWindow.document.close();
            }
            else {
                cb(false);
            }
        }
    }
    xhr.open('GET',PROXY_BASE + '/' + remoteSrcUrl);
    xhr.send();
    iframe.addEventListener('load', cb);
}

Data URI as Src attribute: Hackish, but works

As a method that is in-between the old way (same-origin contentWindow.document.write) and the newest method (srcdoc), using a Data URI as the src of an iframe element is a solution that has decent browser support, but also decent limitations.

What is a Data URI? - A Data URI is, well... data as a URI. Ok, I know, that's not very helpful, but that really is all it is. Normally URIs are strings that identify a resource, and they often they "point" to another location - URLs are a form of URIs that point to web addresses. Or "tel:{{number}}" is a URI that identifies a phone number. Data URIs are unique in that instead of "pointing" to the location of the data, they actually have the raw data inside of the URI. An analogy would be that a book could be theoretically shared by an ISBN URI of "ISBN 0-123-45678-9", but a Data URI that represents a book could be something like "text/book;charset=US-ASCII,{{ENTIRE_TEXT_OF_BOOK}}".

The way that we can use Data URIs with iframes is by setting the src of the iframe element to a data URI that holds raw HTML content, rather than to a URL that points to a web address. Sample:

function dataUriIframeInject(html,target,append,cb){
    cb = typeof(cb)==='function' ? cb : function(){};
    append = typeof(append)==='boolean' ? append : true;
    var iframe = document.createElement('iframe');
    // Construct Data URI
    var dataUri = "data:text/html;charset=utf-8," + escape(html);
    iframe.addEventListener('load',cb);
    iframe.setAttribute('src',dataUri);
    if (append){
        target.appendChild(iframe);
        iframe.style.width = '100%';
        iframe.style.height = '100%';
    }
    else {
        target.replaceWith(iframe);
    }
}

There are several downsides to using Data URIs, and especially with IFrames

Maximum length
- This is not standardized across browser, and sometimes is specified as a maximum character count, and sometimes as a maximum byte size (which would mean max chars depends on encoding type).
- Good rule of thumb is probably to stay under 2K characters.
- See this StackOverflow answer, and this page.
Limited browser support
- Data URIs are supported on IE starting with IE8, but only for assets like images and CSS, not HTML. That makes it unusable with iframes on IE, even up to version 11.
- See caniuse.com/#feat=datauri for more details.
Caching
- The page that contains the Data URI can be cached, but the actual Data URI is not cached itself.
Character escaping
- Trying to escape strings is notoriously annoying, and although functions like escape() make this easy for the most part, don't be surprised if you run into something not working like you were expecting it to.
Limited functionality, and limits are different across browsers
- It can be hard to find this kind of info, but there are special security restrictions with content loaded through a data URI. For example, in Chrome, reading and writing cookies is blocked/disabled inside data URIs. Even first-party same-origin cookies.

The main benefit of injecting content into iframes with data URIs over the previous method is that, even though the iframe is essentially loading a virtual location, it maintains a separate origin from the document it is embedded into! This means that cross-origin policies are enforced, and javascript inside of the iframe cannot access your website, and vice-versa. However, be aware that some old versions of browsers might actually treat them as same-origin (or more accurately, inheriting the origin of the parent). For example, it looks like Firefox versions below 57 did so.

Srcdoc attribute specifically for iframes: the hip new thing

Srcdoc is an attribute that you can use on iframes as an alternative to setting src. You set Srcdoc to a string of (valid) HTML, and it will load it into the iframe:

<iframe srcdoc="<p>Hello World!</p>"></iframe>

Like the Data URI method, the iframe maintains a separate origin than the page it is embedded into, so cross-origin policies are enforced and javascript cannot break out of the iframe and interact with the parent, and vice-versa.

Srcdoc is basically the official solution that all of the previous methods tried to hack together unofficially. It is super easy to use, has its own spec, and is gaining in adoption. Furthermore, it seems to have better security controls and documentation on how it sandboxes content. Unfortunately, the main drawback is still browser-compatibility. In fact, srcdoc has zero support on all versions of IE, and only proposed support on Edge. See caniuse.com/#feat=iframe-srcdoc for a full breakdown of browser support. You also still have to be somewhat careful about escaping strings - for example, the specs say that ampersands need to be double escaped.

If you want to use srcdoc, but support as many browsers as possible, you might want to look at this polyfill. It lets you use srcdoc normally without worrying about which browsers your users are using, and then goes through and switches to an older method if necessary for the user.

Generating iframe content server-side:

Because of the complexities and more error-prone nature of generating iframes on the client-side with Javascript, or by serving iframes with inline Data URI srcs or srcdocs, a lot of people choose to generate iframe content on the server and use the src attribute of the iframe to point to the generator endpoint with information on what to serve. For example, consider a large commenting system where users can leave comments with embedded code demos - definitely something you want to enforce security on so users can't break out of the comment area and interact with the DOM or client JS. For this example, you might do something like:

All comments are stored in a DB with unique IDs
Create a generator script that has an endpoint at:
- ```
"embeds.example.com/comments/"
```
In the comment area, embed iframes with corresponding comment ids, like:
- ```
<iframe src="embeds.example.com/comments/?id=220"></iframe>
```
Your script would pick up the ID from the querystring, lookup the comment from the DB, and echo back the raw HTML as a document, which would load into the iframe.

Of course, there are many different ways to do this, and no one way is the "correct way", but when it comes to security, you definitely want to be careful. If you don't need users to be able to embed any code, and just want simple comments, you would not use iframes, and instead just escape out any HTML and JS users try to put in comments. Again, this is complex stuff, so you probably just want to find a reputable commenting system rather than rolling your own. Improper escaping of comment forms and XSS (cross-site-scripting) vulnerabilities have plagued even the largest companies.

Microsoft Power BI - Playing Around with the Javascript Embed Library and APIs

Joshua Tzucker — Fri, 17 May 2019 04:34:04 GMT

In trying to answer a StackOverflow question, I quickly got sidetracked into playing around with the Microsoft Power BI Javascript Embed API / Library / SDK - something that I had never used before (or Power BI at all for that matter). Here are some "oddities" that I ran into that might help others to report on:

How to reset a Slicer state via the JS Embed Library:

"Slicers" are a form of Power BI "Visuals" that can contain filters, which is to say they are embedded widgets within a report that let the user filter the data that is shown across the report page. Unlike the global filters, which can be cleared by calling embed.removeFilters(), there is no built-in method for resetting a slicer, since slicerVisual.removeFilters() does not necessarily reset the actual slicer state. However, you can easily reset it by calling setSlicerState and setting an "empty" state, like this:

slicerVisual.setSlicerState({filters:[]})

Fun fact - the above is basically how removeFilters() also works - basically just an alias for setFilters([]);

How about a practical example - what if you want to reset every slicer filter when a single "reset all filters" button is pressed? You could have the following code execute:

async function resetAllSlicers() {
  let report = powerbi.embeds[0];
  let pages = await report.getPages();
  for (let x = 0; x < pages.length; x++) {
    let visuals = await pages[x].getVisuals();
    for (let x = 0; x < visuals.length; x++) {
      if (visuals[x].type === 'slicer') {
        // Clear state, but wait before moving on to next one, since clearing a filter can have a cascade effect
        try {
          let cleared = await visuals[x].setSlicerState({ 'filters': [] });
        }
        catch (e) {
          // setSlicerState or getSlicerState will fail if visual element is not initialized
          // visualConfigIsNotInitialized
        }
      }
    }
  }
}

How to use "Hierarchy" filters with reports or slicers

There is a strange column type (if you can call it that) within Power BI - a "Hierarchy". It is similar to column groups, but different in that it is more structured and, like its name would imply, is a hierarchical set of structured data/fields, that can be nested multiple layers deep. Here is what it looks like inside the built-in filter side panel:

So, how do we filter on a column/field inside a hierarchy? It doesn't seem to be well documented, but it is pretty simple, for example:

var report = powerbi.embeds[0];
var filter = {
  "$schema": "http://powerbi.com/product/schema#basic",
  "target": {
    "table": "Geo",
    "hierarchy": "GeoRollup",
    "hierarchyLevel": "Zip"
  },
  "operator": "In",
  "values": [
    "73555"
  ]
}
report.setFilters([filter]);

Or, it seems like you can also just filter on the nested field itself, as if it belonged directly to the table. I'm guessing this is only working for me because there is no naming conflict between a nested field and a regular field:

var report = powerbi.embeds[0];
var filter = {
  "$schema": "http://powerbi.com/product/schema#basic",
  "target": {
    "table": "Geo",
    "column": "Zip"
  },
  "operator": "In",
  "values": [
    "73555"
  ]
}
report.setFilters([filter]);

Some helpful resources:

Using Fabric.js (Fabric) without Cairo for web use - notes on NPM

Joshua Tzucker — Sun, 12 May 2019 07:20:39 GMT

Fabric.js is an amazing open-source library for Canvas manipulation with Javascript - I ended up using it heavily for this project, and have been meaning to post this writeup for a while on some ways to bundle Fabric into your dependencies, without requiring that the system your repo is built on has Cairo installed. These solutions only work if you don't need any of the node-canvas features, which is true for many web uses of Fabric.

By default, if you add Fabric.js as a dependency to your package.json and/or run npm install, it will kick off a build process that will fail if Cairo is not installed globally on your build system. This is because node-canvas (published under NPM as just "canvas") is a dependency of Fabric.js, and in the canvas package.json, the install command will run a build command that requires Cairo.

Easy option: Just ignore the build errors and grab JS from /dist

Although it probably goes against best practices, the nice builders of Fabric have checked in pre-built versions of the Fabric.js library into their repository, built without node-canvas support. This means that, as long as you just need the browser version of Fabric, you can simply pull the pre-built file out of /node_modules for use in your app. For example, as a build step, simply copy like this:

cp node_modules/fabric/dist/fabric.js public/lib/js/fabric.js

Because Canvas is an optional dependency, NPM install will succeed, even though it will spit out all those build errors without Cairo installed.

As a side note, you could even pull Fabric.js from its Github location rather than NPM, since NPM install lets you specific Git repos as sources for dependencies.

Another option - check in your own pre-built version:

This is true for almost any scenario, not just for Fabric; you can always build the file the end user will need yourself, in your dev environment, and then check in the fully built file into your source code, and optionally excluding the raw dependencies as part of your package. In general this is a bad idea for multiple reasons; it goes against some principles of version control, makes it harder for other devs to change the build system, and could have legal implications depending on the license of the third party library. However, for a scenario like this one, I think checking in a single pre-built JS file into your source control could outweigh requiring every dev who wants to build your app to install an entire separate build tool like Cairo.

Off on a tangent; exploring using CDNJS as a dependency

When I initially ran into this issue, one silly thought I had that I wanted to explore to see if it would even be possible, was to pull the prebuilt fabric.min.js file from a URL, ideally from CDNJS. This is not something that should ever be used in a production setting, but it was a fun idea to explore and see if it could easily be scripted with Node (it was). I was able to very quickly script together a file to download the Javascript file directly from CDNJS and place in the spot within my project. I then called that script as my "preinstall" script. I took a little time today to clean that script up and make it into a generic file downloader that can be called via the CLI and optionally can check the downloaded file's hash, to make sure it matches what is expected (e.i. that the code within the file has not been changed). I have uploaded it as a Github gist here.

Simple Node File Downloader – With SRI (Integrity) Hash Checker

Joshua Tzucker — Sun, 12 May 2019 07:17:07 GMT

Reusable Node script that can be passed CLI arguments for a remote file to be downloaded, and optionally integrity checked against a known file hash.

Windows Batch Script to Restart Dropbox Desktop Client

Joshua Tzucker — Sun, 12 May 2019 00:00:56 GMT

Batch script to kill and restart dropbox client on Windows.

Building a VSCode Extension - Some Tips and Tricks

Joshua Tzucker — Sat, 11 May 2019 01:32:45 GMT

I'm trying out building a VSCode extension for the first time, and having a lot of fun learning the basics of TypeScript and the VSCode API.

I'm impressed by the level of documentation for the VSCode APIs, but thought I would share some additional tips and tricks I have been picking up.

Watch for Changes in a Specific Document

onDidChangeTextDocument and onDidCloseTextDocument are both excellent event listeners to subscribe to for watching for changes in a document. However, they are "attached" to the workspace, not to specific documents, so if you attach a listener to either, you will actually be getting notified when any document is changed or closed, not just the one you are interested. However, even though vscode.TextDocument is a complex type, using an equality operator works just fine, so the best way to listen for changes on a specific doc is to store a reference to the TextDocument of interest, then check if it matches on each event. Here is a quick sample:

let currOpenEditor = vscode.window.activeTextEditor;
if (currOpenEditor){
// In this example, we want to start watching the currently open doc
let currActiveDoc = currOpenEditor.document;

let onDidChangeDisposable = vscode.workspace.onDidChangeTextDocument((event: vscode.TextDocumentChangeEvent)=&gt;{
	if (event.document === currActiveDoc){
		console.log('Watched doc changed');
	}
	else {
		console.log('Non watched doc changed');
	}
});

let onDidCloseDisposable = vscode.workspace.onDidCloseTextDocument((closedDoc: vscode.TextDocument)=&gt;{
	if (closedDoc === currActiveDoc){
		console.log('Watched doc was closed');
	}
	else {
		console.log('non watched doc closed');
	}
});
}

Getting the range of entire document

Right now, it looks like there is no single API call that will return the full range that the text content of a document spans. There are a few different ways you can determine the range:

The popular method - something like:

// where document is of type vscode.TextDocument
let entireDocRange: vscode.Range = new vscode.Range(0,document.lineAt(0).range.start.character,document.lineCount-1,document.lineAt(document.lineCount-1).range.end.character);

Neat method - Create a range that is purposely larger than the actual range of the document (start at 0,0, then go to end of doc + 1 line), then use vscode.TextDocument.validateRange() to shrink the larger range down to the actual text content. Found on StackOverflow, demonstrated below:

let entireDocRange = document.validateRange(new vscode.Range(0,0,document.lineCount,0));

Webview - the built-in web renderer

If you are looking to display simple HTML content to a VSCode user, you can use the built-in webview API to show a rendered webpage in a new tab - no dealing with deploying a local server, Chrome extensions, etc. You can even communicate with the content inside the webview by using the standard postMessage syntax.

Amp for Email - carousel not showing up - htaccess solution

Joshua Tzucker — Fri, 03 May 2019 16:21:34 GMT

I recently ran into the issue of an <amp-list> carousel not working inside an AMP email template. More specifically, it was not showing up in the "Gmail AMP for Email Playground" tool - where the slide carousel should have appeared, there was just empty blank space. Opening up the browser dev tools, it was pretty easy to spot that this had to with security issues - the JSON src of the list was fetched just fine (200 status), but the Playground was refusing to parse and show it due to a bunch of missing CORs header.

A quick search got me what I was was looking for - the AMP for Email specification does indeed require special security related headers for remote JSON sources that are fetched for <amp-list> elements:

I quickly uploaded the JSON source for my list to a special subfolder on my server, and added an HTACCESS file for that directory so I could play around with adding the correct headers. I followed the recommendations given by Google from the first link, and updated the email template with the new source. However, even though I could see that my JSON fetch now returned the right headers, such as "amp-access-control-allow-source-origin", I was still getting a very specific error: "The amp-access-control-allow-source-origin must be equal to the amp source origin sent in the request."

This made even less sense when looking closely at the network request - my response header exactly matched the _amp_source_origin request parameter!

It took me longer than I would like to admit, but after staring at the documentation again, this time at the readme for the full list of CORs headers for AMP, not just for email, I finally realized I was missing one additional header: "access-control-expose-headers". This is important, because in order for the browser / AMP playground to be able to read back the non-standard AMP-Access-Control-Allow-Source-Origin, it needs to be explicitly exposed through that header. I added it to my HTACCESS file, and lo-and-behold, the carousel showed up and started working. Google should probably update the primary docs for the AMPHTML security page to make this more clear.

Here is my full working HTACCESS:


# CORs
Header add Access-Control-Allow-Origin "*"
Header add Access-Control-Allow-Source-Origin "AMP-Access-Control-Allow-Source-Origin"
# AMP specific
Header add AMP-Access-Control-Allow-Source-Origin "amp@gmail.dev"
# Don't forget this!
Header add access-control-expose-headers "AMP-Access-Control-Allow-Source-Origin"

monitorEvents() Demo - Javascript DOM Events Virtual Console

Joshua Tzucker — Thu, 02 May 2019 22:58:25 GMT

Quick demo to show a stream of Javascript DOM events emitted when users interact with a page.

Google Tag Manager - Triggers Breakdown and Javascript Alternatives

Joshua Tzucker — Thu, 02 May 2019 22:33:27 GMT

Google Tag Manager (GTM) is a great tool for digital marketing, devops, A/B website testing, analytics, you name it. With just a few clicks, you can easily assign tags to fire based on complex combinations of rules - such as firing a conversion pixel when a user submits a contact form with a certain dropdown field value - without having to write Javascript (JS) by hand, or update the actual source code of the site. However, I also think that GTM gets used a little too often, without marketers understanding the underlying technology they are using, which can lead to tunnel vision and an inability to come up with solutions when GTM cannot be used for a given problem.

This post will attempt to explain how GTM works on multiple levels, and serve as a guide on how you might replicate certain GTM functionalities with pure vanilla Javascript.

Warning: Conflicting Terminology
Triggers:
- Page View
  - Page View
  - Dom Ready
  - Window Loaded
- Click
  - All Elements
  - Just Links
- User Engagement
  - Element Visibility
  - Form Submission
  - Scroll Depth
  - YouTube Video
- Other
  - Custom Event
  - History Change
  - JavaScript Error
  - Timer
  - Trigger Group
GTM Extra: "All ___" vs "Some ___" (conditional operator)
GTM Extra: "Wait for Tags"

WARNING: Conflicting terminology

First, something that has to be addressed right away; certain words carry a different meaning within the Javascript world that powers GTM, vs the marketing world that uses it. At the top of the list is the word "event".

For the duration of this post, I'll try to be very specific about which type of event I'm talking about. For Browser DOM Javascript Events, I'll say "JS Events" or just "Events". For marketing events, I'll always say which platform it goes with, such as "Google Analytics Events".

Marketing Events vs JS Events:

For marketers, an "event" firing usually means a Google Analytics event, Facebook Pixel event, or any other platform recording some sort of user action occurring. Usually these events are not triggered unless specifically set up, and in fact, that is a common use of GTM; use a trigger, such as a button click, to fire a Google Analytics Event.

However, in the Javascript web development world, "event" means something completely different. In JS, an event, or more accurately a "DOM event", is a signal that is emitted by the browser's JS APIs, which you can then "listen" for. A large amount of them are fired automatically - for example, if you submit a form, the form "submit" event is fired automatically. Browser DOM events are a vast and complicated topic, but the important things to remember for the context of this post is that:

DOM events happen whether or not you are listening for them
You cannot capture events that are outside the scope of where your code is running
A lot of GTM functionality is built upon JS events and listening for them

To visualize just how common and automatic JS events are, I've setup a small demo (opens in new tab). On the right side of the demo, you can see a stream of JS events as they happen in real time. On the left side, you can trigger these events to occur by moving your mouse, clicking on buttons, typing in fields, etc. For the full demo source, visit the codepen.

Triggers:

Page View:

Help page - link.

Trigger	What it Does	Javascript
Page View	Fires ASAP - basically as soon as GTM starts loading.	Since this fires ASAP, this would be the same as just putting JS code in the same place, or instead of your GTM embed snippet.
DOM Ready	Fires when the basic DOM structure of the page has been parsed. Images, styles, and other things could still be loading.	`window.addEventListener('DOMContentLoaded',function(evt){ /* Do Something / });` Or, since this won't fire if the page has already* hit DOM ready before this code ran, you can wrap it like this: `(function(){ function handleEvent(){ /* Do Something */ } if (document.readyState==="interactive"){ handleEvent(); } else { window.addEventListener('DOMContentLoaded',handleEvent); } })();`
Window Loaded	Fires when the page has been fully loaded (waits for images, scripts, etc). Can take a while!	`window.addEventListener('load',function(evt){ /* Do Something / });` Or, since this won't fire if the page has already* hit DOM load before this code ran, you can wrap it like this: `(function(){ function handleEvent(){ /* Do Something */ } if (document.readyState==="complete"){ handleEvent(); } else { window.addEventListener('load',handleEvent); } })();`

Click

Trigger What it Does Javascript

Trigger	What it Does	Javascript
All Elements	Listens for any click DOM event.	`document.addEventListener("click",function(evt){/* Do Something /});` Or, listen to a specific element `var elem = document.querySelector("#MyElement"); elem.addEventListener("click",function(evt){/ Do Something */});`
Just Links	Listens for click events, but "pre" filters to click events that have fired on links.	`var links = document.querySelectorAll("a"); for (var x=0; x<links.length; x++){ links[x].addEventListener("click",function(evt){/* Do Something */}); }`

All Elements

Listens for any click DOM event.


document.addEventListener("click",function(evt){/* Do Something */});

Or, listen to a specific element


var elem = document.querySelector("#MyElement");
elem.addEventListener("click",function(evt){/* Do Something */});

Just Links

Listens for click events, but "pre" filters to click events that have fired on links.


var links = document.querySelectorAll("a");
for (var x=0; x<links.length; x++){
    links[x].addEventListener("click",function(evt){/* Do Something */});
}

User Engagement

Trigger	What it Does	Javascript
Element Visibility	This is an advanced trigger that really shows off the power of GTM. There are a lot of different options with this trigger, but the basics of what it does is checks a few different things about the element to see if it is visible: getComputedStyle to see if the element has the display property set to hidden, and getBoundingClientRect to see if the boundaries of the element are within the current view of the user.	This is not trivial to code out by hand, in a way that works across browsers and reliably. One option is to use JQuery and check visiblity whenever the page is scrolled: `// Note - this fires whenever the element comes into view, regardless if already fired. (function(){ function testVisible(){ if($("#MyElement").is(":visible")){ /* Do something */ } } testVisible(); $(window).on("scroll",testVisible); })();` Or, you could follow one of these guides: here or here
Form Submission	Fires whenever a <form></form> is submitted. Listens for the "submit" JS event.	`document.addEventListener("submit",function(evt){/* Do Something /});` More robust option - listen for specific form, and don't submit it UNTIL the marketing tag has fired. `var myForm = document.querySelector("#ContactForm"); myForm.addEventListener("submit",function(evt){ // Prevent form from being submitted right away evt.preventDefault(); // Do something / sendGoogleAnalyticsEvent("Form Submitted"); */ // Use a delay to ensure the above marketing tag has finished setTimeout(function(){ // Submit the form myForm.submit(); },1000); });`
Scroll Depth	This allows you to configure tags to fire once the user has scrolled a certain amount down the page. Uses scrollTop and scrollLeft to determine offset from origin and whether or not the threshold has been reached.	A simple version that just waits until a threshold has been hit might look something like this: `(function(){ // 50% percent as float var threshold = 0.5; var fired = false; function checkVerticalScroll(){ if (!fired){ var totalHeight = parseInt(getComputedStyle(document.body).height.replace("px","").replace("%",""),10); if (document.body.scrollTop > (totalHeight * threshold)){ /* Do Something */ fired = true; } } } document.addEventListener("scroll",checkVerticalScroll); })();`
YouTube Video	This trigger checks the page for any embedded YouTube Videos, and if they are there, allows you to fire tags based on a video play event (like start, pause, etc). The way this works is a little complicated, since YouTube embeds are usually iFrames. First, it requires that all YouTube embed Iframes contain enablejsapi=1, which is why they have the checkbox "Add JavaScript API support to all YouTube videos", which automatically adds that string to all embeds. Once an embed has that flag, the video will start sending cross-origin messages (via window.postMessage()) with event data, which GTM is listening for. Finally, GTM checks the event that is passed via postMessage, and parses what type of YouTube event it maps to (play, pause, etc.) and determines if your tag should fire.	Here is some demo code that should work as long as your embeds have the enablejsapi=1 flag turned on. (function(){ // Change to match your YouTube Iframe embed var ytEmbedId = "myvideo"; if (document.querySelectorAll('script[src="https://www.youtube.com/iframe_api"]').length == 0){ var tag = document.createElement('script'); tag.src = "https://www.youtube.com/iframe_api"; document.body.appendChild(tag); } window.onYouTubeIframeAPIReady = function(){ console.log("YT Ready"); var player = new YT.Player(ytEmbedId); player.addEventListener("onStateChange",function(evt){ switch (evt.data) { case YT.PlayerState.UNSTARTED: // Video start /* Fire video start event / break; case 0: // Video Ended / Fire Video End Marketing Tag / break; case 1: // Video playing break; case 2: // Video paused / Fire Video Paused Marketing Tag */ break; default: break; } }); } `})();`

Other

Trigger	What it Does	Javascript
Custom Event	This trigger is very specific to GTM - it can be used to fire a tag when a specific named event is pushed to the GTM DataLayer - which is kind of like a shared stream of events that exists in the page. Often, this kind of trigger is used when multiple vendors are working together to implement something. So, for example, a shopping cart vendor can push an event to the DataLayer for when a user adds an item to the cart, and then the ad agency vendor can listen for that event with this custom event trigger and fire a GA conversion event when it is trigger.	If you are not using GTM in the first place, you wouldn't really need a JS alternative to this. Just use native custom JS events! For multiple vendors working together, one vendor can use dispatchEvent() to send the event, and another can use addEventListener to listen for it. However, if you do not have access to GTM, but another vendor does, and they want to share events with you by pushing to the dataLayer, you could still listen for their dataLayer despite not having a GTM login / access. You would basically replicate this trigger with some code like this: (function(){ // Hold ref to original push fn var originalPushFn = window.dataLayer.push; // Replace with custom push fn window.dataLayer.push = function(){ // Pass arguments to orginal function - VERY IMPORTANT originalPushFn.apply(this,arguments); // Our own processing for (var x=0; x<arguments.length; x++){ if (typeof(arguments[x])==="object"){ filterDataLayerPush(arguments[x]); } } } function filterDataLayerPush(pushObj){ // Handle // Example - waiting for event named "addToCart" if (pushObj["event"] === "addToCart"){ /* Fire GA Add To Cart Marketing Tag */ } } `})();`
History Change	Lets you fire a tag when the URL changes in a non-standard way, without the browser actually navigating pages. The first way is through a "URL fragment" change. Most people know this as what follows the "hash mark" - e.g. example.com/subpage#subsectionalpha - where #subsectionalpha could change to #subsectionbeta without the page reloading. This is common on single-page-applications (SPAs). The second thing it listens for is the URL being changed through the HTML5 pushstate API. This is something newer and less common, but becoming more common with the rise of PWA (progress-web-apps).	For URL fragment / hash changes, the code is pretty simple. There is built in window JS event you can listen for: `window.addEventListener("hashchange",function(evt){ if (location.hash === "#subsectionalpha"){ /* Do Something */ } });` Unfortunately, there is not yet a good built-in JS event to listen for as an indicator of a pushState happening. The easiest current solution is to basically replace the global history.pushState function with your own function that scans the event and then passes it to the original function. If you look at the source code for GTM, you can see that is exactly what Google wrote their GTM JS code to do!
JavaScript Error	This gets triggered whenever an uncaught Javascript error occurs. The main use of this trigger is to capture the error data, and relay it to another platform that consolidates JS errors into some sort of dashboard, such as Sentry.io or Airbrake	Most error collection dashboards probably have SDKs or plug-n-play libraries to make capturing client errors easy, so I doubt you would have to do much hand-coding in terms of catching and relaying JS errors. However, if you need to, there is a global window JS event that fires on uncaught JS errors for some browsers, which is aptly named "error". Or refer to this site on a cross-browser approach.
Timer	This is pretty simple - fires a tag after a set amount of time has passed. Or fires it every __x__ amount of time.	Here is a quick demo of how you could implement this in JS. `(function(){ // Configure time in MS and max fires var interval = 2000; var maxFires = 4; var timer; var fires = 0; timer = setInterval(function(){ if (fires < maxFires){ /* Do Something */ fires++; } else { clearInterval(timer); } },interval);` `})();`
Trigger Group	A trigger group is just a way in GTM to combine multiple triggers together into one rule. Normally, when you assign triggers to a tag, they function in an "OR" configuration, meaning that the tag will fire if ANY of the triggers get triggered. However, if you assign a trigger group, the triggers in that group function in an "AND" configuration, meaning that ALL of the triggers must have been triggered in order for the tag to fire. Example: You want to fire a "high value user" tag if a user scrolls below 50% of the page, AND spends at least 60 seconds on the site. You could enforce this by creating a trigger group that contains both a Timer Trigger as well as a Scroll Depth Trigger, and then assign that trigger group to the tag you want to fire.	This is specific to GTM, so there is no need for a JS equivalent. JS developers should already be familiar with basic logic, such as AND vs OR - a GTM trigger group is basically the same as doing IF(TRIGGER_A && TRIGGER_B && ...){/* Do Something */}

GTM Extra: All __ vs Some _ (conditional operator)

On most triggers, there is an extra option to control firing that enables you to filter to a subset of the trigger. For example, on the "Click" trigger, there is an option that lets you pick between "All Clicks" and "Some Clicks".

When you turn on the "Some" filter, you can narrow down by many built in variables and conditions. The list is pretty extensive (link to built-in variables), but here are some of the most useful ones and how they map to Javascript:

Condition Can be filtered on: What it Does Javascript

Condition	Can be filtered on:	What it Does	Javascript
matches CSS selector	Element (Click,Form)	A CSS selector is a way to target an element based upon its properties that are mappable to how CSS gets applied to the element. An analogy could be selecting people based upon a description - "Pick all men in the room with blonde hair, glasses, and red shoes". In the context of GTM, you might use this option to do something like filter link clicks to just click-to-call links, so you can fire a phone call marketing conversion tag. To do so, you could filter by "matches CSS selector", and then put the value as 'a[href^="tel:"]'	Most browsers support the .matches() method, so this is easy in JS: `// Attach to all links document.querySelectorAll("a").forEach(function(elem){elem.addEventListener("click",filterLinkClicks)}); // ...elsewhere in code function filterLinkClicks(evt){ console.log(evt); if(evt.target.matches('a[href^="tel:"]')){ /* Do Something */ // Fire click-to-call marketing conversion tag } }`
Contains / starts with / equals / etc.	Pretty much anything...	Since this can be used as an operator with pretty much any variable, this can mean a lot of different things. For example, when used with "Click Text" or "Form Text", GTM is basically comparing all the text contained within the HTML element to the value you enter to compare against.	Again, there are tons of different combinations that this filter allows for, but none of them are very complicated to implement in JS. For example, no matter how many elements are nested within another element, you can get all the text that is contained within and check if it contains a string using element.innerText: `var searchString = "Find Me!"; // Case sensitive “contains” if(myElement.innerText.includes(searchString)){ /* Do Something / } // Alternative case-sensitive approach if (myElement.innerText.indexOf(searchString) !== -1){ / Do Something / }` `// Case in-sensitive if (myElement.innerText.toLowerCase().indexOf(searchString.toLowerCase()) !== -1){ / Do Something */ }`

matches CSS selector

Element (Click,Form)

A CSS selector is a way to target an element based upon its properties that are mappable to how CSS gets applied to the element. An analogy could be selecting people based upon a description - "Pick all men in the room with blonde hair, glasses, and red shoes". In the context of GTM, you might use this option to do something like filter link clicks to just click-to-call links, so you can fire a phone call marketing conversion tag. To do so, you could filter by "matches CSS selector", and then put the value as 'a[href^="tel:"]'

Most browsers support the .matches() method, so this is easy in JS:


// Attach to all links
document.querySelectorAll("a").forEach(function(elem){elem.addEventListener("click",filterLinkClicks)});
// ...elsewhere in code
function filterLinkClicks(evt){
    console.log(evt);
    if(evt.target.matches('a[href^="tel:"]')){
        /* Do Something */
        // Fire click-to-call marketing conversion tag
    }
}

Contains / starts with / equals / etc.

Pretty much anything...

Since this can be used as an operator with pretty much any variable, this can mean a lot of different things. For example, when used with "Click Text" or "Form Text", GTM is basically comparing all the text contained within the HTML element to the value you enter to compare against.

Again, there are tons of different combinations that this filter allows for, but none of them are very complicated to implement in JS. For example, no matter how many elements are nested within another element, you can get all the text that is contained within and check if it contains a string using element.innerText:


var searchString = "Find Me!";
// Case sensitive “contains”
if(myElement.innerText.includes(searchString)){
/* Do Something */
}
// Alternative case-sensitive approach
if (myElement.innerText.indexOf(searchString) !== -1){
/* Do Something */
}
// Case in-sensitive
if (myElement.innerText.toLowerCase().indexOf(searchString.toLowerCase()) !== -1){
/* Do Something */
}

GTM Extra: Wait for Tags:

Another "extra" option that appears in multiple spots within GTM is the option to "wait for tags". This is useful because sometimes a trigger, such as a link click, will actually take the user to a different website, and if that happens before the tag has a chance to fire, then the marketing event never gets processed. By turning on "wait for tags", GTM will actually STOP the user-initiated event from happening (such as navigating to a different site via a link) UNTIL your tags have fired.

From a developer perspective, the way this is usually done is through event.preventDefault() and calling stopPropogation(). Then you do whatver you want (fire tags, log something), then resume the event by either re-emitting it without stopping it again, or calling the native handle (e.g. form.submit())

The risk, or downside to turning on this feature, is that interrupting the flow of JS events can cause issues with third-party libraries or your own codebase, depending on how various pieces of code are listening for events. Example.

Here is a demo of how you might fire off a tag when a user clicks a link to an external site, and stop them navigating to the new site until your tag has fired:


document.querySelectorAll("a[href]").forEach(function(link){
    // Only for external links
    if (link.hostname !== document.location.hostname){
        link.addEventListener("click",function(evt){
            // Only preventDefault on links opening in same tab
            if (link.getAttribute("target")!== "_blank"){
                // Stop browser from navigating to link
                evt.preventDefault();
                // Continue onwards to link destination after a delay of 0.25 seconds
                setTimeout(function(){
                    document.location.href = link.href;
                },250);
            }
            // If the link opens in the same tab, the code below has 250 ms to execute before it is too late (see above code)
            /* Do Something */
            // Fire Marketing Tag
        });
    }
});

Desktop Cloud Transform (DCT)

Joshua Tzucker — Wed, 24 Apr 2019 15:28:27 GMT

I created Desktop Cloud Transform, from scratch to fully functional prototype, in about a week (about 40-50 hours of logged time before feature complete). I used Qt/C++, Cloudinary's API, SQLite, Javascript, and other technologies. I decided to build it for my own needs, and also as an opportunity to get up to speed on QT and C++.

Github repo & readme: https://github.com/joshuatz/desktop-cloud-transform

What is Desktop Cloud Transform (DCT)?

Elevator pitch: Desktop Cloud Transform (or DCT) is a desktop application that allows you to upload and apply complex transformations to images in the cloud, simply by dragging and dropping. It uses the power of Cloudinary's transformation engine, which means you can apply almost any combination of alterations to your images just like you could in Photoshop, but since it is all done in the cloud, you don't need a powerful computer, expensive software, and more importantly, you don't need to know how to use complicated image editing software.

Here is a sample configuration - taking any local image file, and turning it into a fun Polaroid style image:

[video width="1920" height="1080" mp4="/media/Desktop-Cloud-Transform-DCT-Polaroid-Demo-John.mp4" loop="true" autoplay="true"][/video]

Here is a more practical business application - showing off UI mockups on actual physical hardware that would normally cumbersome to deploy to and take a picture of, such as a Tesla screen:

[video width="1920" height="1080" mp4="/media/Desktop-Cloud-Transform-DCT-Tesla-GUI-Demo.mp4"][/video]

User Interface

I used QML for creating the user interface, which I think turned out nicely. Here is the main UI / home screen:

And configuration editor :

What is supported? What are DCT "Configurations"?

DCT configurations are combinations of DCT settings (such as "overwrite existing file") and Cloudinary transformation settings (such as "crop to 200 x 200 pixels, blur faces, etc"). Once you save a configuration, you can drag and drop images onto its row and the image you dropped will be uploaded to Cloudinary, have the configuration applied, and then the result downloaded back to your local computer folder.

Cloudinary does all the heavy lifting for DCT - storing images, transforming, applying effects, etc. So pretty much any transformation that Cloudinary supports can be saved within a DCT "Configuration". If you are looking to create some complicated presets, I highly recommend taking a look at Cloudinary's massive "Transformation Reference" documentation. You can also take a look at my other Cloudinary project, which explored some unique ways to combine Cloudinary transformations.

Extra Feature: Macro support

An extra feature of DCT is the ability to use "macros" within transformation strings. For example, if you put "{UPLOADED}" inside an outgoing transformation string, my program will replace it with the correct string to represent an overlay layer that references the public ID of the image you dragged onto the configuration row. Some other currently supported macros are {filenameNoExt}, to insert the filename of your uploaded image, without the extension (e.g. "My File" instead of "My File.jpg").

For an up-to-date list of supported macros, make sure you have the most recent version of the software, and check the documentation here.

Cloudinary SDK for C++:

As a technical side note, if you are a fellow developer looking to integrate Cloudinary into your C++ based application, unfortunately I have to tell you that they do not have an existing C++ SDK. However, their API is just as powerful - most of their SDK libraries are basically just translating SDK calls into API calls. In addition, you might find some of the code I have written helpful in getting started; I have some helper functions to do things like generate the special secure API upload signature (see here, for example). I've used a decent amount of QT specific classes, but it should be moderately easy to convert to std libraries or something using boost.

Disclaimers:

I built this application as a learning experience to get up to speed on QT/C++ and solve some repetitive image manipulation needs of my own. Not a lot of development time has gone into it so far, so it still has some rough edges and hopefully will see improvements over time.

In addition, although I have a fondness for Cloudinary and this application is mostly a wrapper around their services, it is not officially endorsed, supported by, or associated with Cloudinary in any manner. Cloudinary is a registered trademark of Cloudinary Ltd.

QT - Code Snippet for QNetworkRequest Lambda Callback with Connect

Joshua Tzucker — Fri, 12 Apr 2019 01:00:09 GMT

This is not best practice, but I wanted to handle the result of a very simple web request (GET a URL) in as few lines of code as possible, primarily just as a test mechanism. Pretty much every example I could find out there either didn't work, or required connecting true signal and slots, where the class that contains the initiating code would need to be a QObject derived class so that it can function as the slot mechanism for "receiving" the update. What was frustrating to me about this was that I have spent most of my coding time working with Javascript and similar web-oriented languages, which always have tons of options for asynchronous operations and things like callbacks, futures, and async/await.

Finally, after a little bit of searching and trail and error, I found the solution with Lambda expressions and the new syntax available for QObject::connect (which does not require that the receiver be a QObject slot):

void Helpers::checkInternetConnection(){
qDebug() &lt;&lt; "Checking internet connection...";
QNetworkAccessManager *netManager = new QNetworkAccessManager();

// Use lambda as receiver with connect to get result of network finished event
QObject::connect(netManager,&amp;QNetworkAccessManager::finished,[=](QNetworkReply *reply) {
    qDebug() &lt;&lt; "In lambda!";
    if (reply-&gt;error()){
        qDebug() &lt;&lt; "Fail";
    }
    else {
        qDebug() &lt;&lt; "Pass!";
    }
});

// Start the network request
QNetworkReply *netReply = netManager-&gt;get(QNetworkRequest(QString("https://www.google.com/")));
}

You can can even manipulate the value of outside variables through lambdas by including them in the capture argument. Here I'm de-referencing a pointer to a boolean argument and filling in the value based on the result of the network request:

// In Helpers class:
void Helpers::checkInternetConnection(bool *res){
qDebug() &lt;&lt; "Checking internet connection...";
QNetworkAccessManager *netManager = new QNetworkAccessManager();

// Use lambda as receiver with connect to get result of network finished event
QObject::connect(netManager,&amp;QNetworkAccessManager::finished,[=](QNetworkReply *reply) {
    qDebug() &lt;&lt; "In lambda!";
    if (reply-&gt;error()){
        qDebug() &lt;&lt; "Fail";
        *res = false;
    }
    else {
        qDebug() &lt;&lt; "Pass!";
        *res = true;
    }
});

// Start the network request
QNetworkReply *netReply = netManager-&gt;get(QNetworkRequest(QString("https://www.google.com/")));
}
// Elsewhere:
bool hasInternet = false;
Helpers::checkInternetConnection(&hasInternet);
QTimer::singleShot(2000,&hasInternet{
qDebug () << “res = ” << hasInternet;
});

Although this is tempting to use for a one-off testing mechanism, once I stopped to think for a moment about what I needed to do, I realized that I was better off not fighting using QObject classes. If I'm trying to track whether or not my application has internet access, I'm much better off creating a App Status QObject class, which would have a "hasInternet" member, which I can then hook together with a signal of "internetConnectionChanged" and have "hasInternet" be the receiver slot for my checkInternetConnection function.

Frankly, thanks to QT's macro system, that actually isn't even all that much boilerplate code, and once I set that up, I can then easily expose my QObject app status class to the QML side, and do something neat like fade out and disable buttons that rely on internet connected features when the application detects a loss of network.

Cross-Platform Interacting with Windows, Titles, Keypresses, and More

Joshua Tzucker — Wed, 03 Apr 2019 13:40:27 GMT

One of the things that bothers me about development is when you come across a problem, and you think to yourself, "surely someone else has solved this before", but then you start searching and keep coming up empty. This is the issue I had when starting a recent project that needed to do things like see which Window (application) is open and active in a cross-platform (Windows, Linux, Mac) way.

In fact, I had such a hard time finding existing cross-platform libraries that I started writing one myself, starting with the Windows side of things, but when the complexity got really high, I revisited my search, and finally got some results. I wanted to share what I found, since as I said, these resources were not easy to track down.

Quick list of options:

So far I have only found three libraries that are cross-platform, although I'm still keeping an eye out. See the embedded spreadsheet below for a comparison of features, or click here to open in a new tab:

Most Robust: Robot (Robot/robot and Robot/robot-js)

I really wish I had found this library sooner. The Robot library is an incredibly thorough and well put-together library that can handle pretty much anything you would need in order to manipulate windows and send keypresses. What makes it even more impressive is that it appears to be written and maintained by just one very dedicated individual; David Krutsko. Kudos to him on building this amazing library.

Robot can be used as a C++ library, or a NodeJS library. I have just started playing around with it for C++, combined with QT, and I can share a tip related to getting it working. To get it to work for my project, I had to change a few things:

Changed compiler from MinGW to MSVC.
- Robot uses the <threads> library, which, at least for the version of MinGW shipped with QT, is not available. So trying to compile I got a bunch of errors about mutex and threads. Someone put together a Thread library for MinGW, but I couldn't get it to work, so I just gave up and switched over to MSVC, which basically works out of the box with this library.
Manually linked up wingdi.h / Gdi32.dll Windows GDI library
- For Windows, Robot uses some bitmap / drawing functions that rely on the Windows gdi32 library. When trying to compile without any special setup, I got a bunch of linker errors related to the use of this library - things like "Clipboard.obj:-1: error: LNK2019: unresolved external symbol __imp_GetDIBits referenced in function "public: static bool __cdecl Robot::Clipboard::GetImage(class Robot::Image &)" (?GetImage@Clipboard@Robot@@SA_NAEAVImage@2@@Z)"
  - If you are using QMake like I am, just add "LIBS += -lgdi32" to either your project file, or in a PRI that gets loaded for this library.
  - Also make sure that QMake can find the Windows Kit folder where the DLL and header files live. You can always add it manually through "INCLUDEPATH +="

Here is the PRI file that I created to pull in all the Robot source files into my QT project. Depending on how you have structured your project, you would need to change the ROBOT_LIB_SOURCEPATH variable.

# https://github.com/Robot/robot ROBOT_LIB_SOURCEPATH = $$PWD/../lib/robot/Source INCLUDEPATH += $$ROBOT_LIB_SOURCEPATH Rare use case will require uncommenting below - =1 forces new C++11 ABI #DEFINES += “_GLIBCXX_USE_CXX11_ABI=0” Robot needs some extra win libs - primarily for clipboard and bitmap operations INCLUDEPATH += “C:/Program Files (x86)/Windows Kits/8.1/Include” LIBS += -lgdi32 HEADERS += $$ROBOT_LIB_SOURCEPATH/Enum.h $$ROBOT_LIB_SOURCEPATH/Bounds.h $$ROBOT_LIB_SOURCEPATH/Clipboard.h $$ROBOT_LIB_SOURCEPATH/Color.h $$ROBOT_LIB_SOURCEPATH/Global.h $$ROBOT_LIB_SOURCEPATH/Hash.h $$ROBOT_LIB_SOURCEPATH/Image.h $$ROBOT_LIB_SOURCEPATH/Keyboard.h $$ROBOT_LIB_SOURCEPATH/Memory.h $$ROBOT_LIB_SOURCEPATH/Module.h $$ROBOT_LIB_SOURCEPATH/Mouse.h $$ROBOT_LIB_SOURCEPATH/Point.h $$ROBOT_LIB_SOURCEPATH/Process.h $$ROBOT_LIB_SOURCEPATH/Range.h $$ROBOT_LIB_SOURCEPATH/Robot.h $$ROBOT_LIB_SOURCEPATH/Screen.h $$ROBOT_LIB_SOURCEPATH/Size.h $$ROBOT_LIB_SOURCEPATH/Timer.h $$ROBOT_LIB_SOURCEPATH/Types.h $$ROBOT_LIB_SOURCEPATH/Window.h

SOURCES += $$ROBOT_LIB_SOURCEPATH/Bounds.cc $$ROBOT_LIB_SOURCEPATH/Clipboard.cc $$ROBOT_LIB_SOURCEPATH/Color.cc $$ROBOT_LIB_SOURCEPATH/Hash.cc $$ROBOT_LIB_SOURCEPATH/Image.cc $$ROBOT_LIB_SOURCEPATH/Keyboard.cc $$ROBOT_LIB_SOURCEPATH/Memory.cc $$ROBOT_LIB_SOURCEPATH/Module.cc $$ROBOT_LIB_SOURCEPATH/Mouse.cc $$ROBOT_LIB_SOURCEPATH/Point.cc $$ROBOT_LIB_SOURCEPATH/Process.cc $$ROBOT_LIB_SOURCEPATH/Range.cc $$ROBOT_LIB_SOURCEPATH/Screen.cc $$ROBOT_LIB_SOURCEPATH/Size.cc $$ROBOT_LIB_SOURCEPATH/Timer.cc $$ROBOT_LIB_SOURCEPATH/Window.cc

Getting Robotjs (octaImage/robotjs) working with C++ and QT

Robotjs (not to be confused with Robot-JS by Robot) is a NodeJS desktop manipulation library written by octaImage, aka Jason Stallings. One of the neat things about this library, besides it being written in C, is that Jason did a really nice job of keeping the C/C++ layer separate from the Javascript / NodeJS layer. This layer of separation allows us to actually use the native C code in its raw form, before it gets compiled by node-gyp into a Node addon.

I was able to do this surprisingly easily with QT - I just setup a PRI file to import all the raw header and source files, making sure not to pull in the files that wrap functions for node (primarily robotjs.cc). Here is my PRI file, but note that I'm not pulling in 100% of the files, since I only wanted to test part of the library:

NODE_MODULESPATH = $$PWD/../node_modules NODE_SOURCEPATH = $$PWD/../node_modules/robotjs/src INCLUDEPATH += $$NODE_SOURCEPATH Special - seems to be an issue with a windows constant - could also be related to _WIN32_WINNT issue on active-windows-lib DEFINES += “MOUSEEVENTF_HWHEEL=0x1000” HEADERS += windows.h $$NODE_SOURCEPATH/os.h $$NODE_SOURCEPATH/keycode.h $$NODE_SOURCEPATH/keypress.h $$NODE_SOURCEPATH/mouse.h $$NODE_SOURCEPATH/screen.h $$NODE_SOURCEPATH/types.h $$NODE_SOURCEPATH/deadbeef_rand.h

SOURCES += $$NODE_SOURCEPATH/keycode.c $$NODE_SOURCEPATH/keypress.c $$NODE_SOURCEPATH/mouse.c $$NODE_SOURCEPATH/screen.c $$NODE_SOURCEPATH/deadbeef_rand.c

I also had to set a constant, for MOUSEEVENTF_HWHEEL. I've moved on from using this library, so I can't recall exactly why I had to add it, but I'm guessing the compiler was complaining about not knowing the value for it, which probably has to do with a screwed up path import on my end.

Understanding the Inner Workings

If you want to roll your own solution, or just understand more about the inner workings of these libraries and how you use system API calls to interact with Windows and inputs, I do actually have some advice for you.

First, start with the source code for the active-win library. It is well-commented and is basically the textbook example for how these system API calls are meant to be used, for Windows, Mac, and Linux.

Next, if you are developing for Windows, be aware that most of their desktop APIs use strings in a... ahem... "interesting way". You will see strings passed around as "LPTSTR", which is basically a macro that expands to either LPWSTR (long pointer to wide string, which is composed of wchar_t) or LPSTR (composed of char), based on if your project uses Unicode. This stack overflow answer has a good breakdown of some unique types that the Windows API uses.

For keypresses, be aware that although we have things like the Unicode standard so that characters and strings are encoded the same across platforms, actual keypresses are usually represented by enums/constants that are unique per platform. For example, Windows represents keypresses through "Virtual-Key Codes", the constants for which can be found here. Meanwhile, Unix uses X11 and defines it own set of constants, which can be found here among other places. I started compiling a database of these different constants across platforms, which can be found here.

Finally, when looking for existing source code to reference and see how this problem has been tackled by others, don't forget some interesting places to look. Software that is likely to use system APIs to interact with Windowed applications and keypresses is:

Automated testing software
"Mocking" software
Password database apps
- For example, you can see how KeePass uses some native Windows API calls to see which window is open and send passwords to the right one.
Screenshot apps
- You can see here how Greenshot uses Dapplo.Windows to register system level keypress hooks - so they can listen for the Print Screen Key to be pressed, as well as custom combos.
Productivity trackers (ones that automatically record which applications you are spending time in)
- Good example: ActivityWatch - Python source code
Browsers
- Chromium source

Know of something better?

If you feel like I missed something big, feel free to leave a comment below. It is always interesting to see how different solutions are engineered around a shared problem. I'm still looking for a cross-platform solution for key "hooks" - where your code can listen for a keypress regardless of whether or not it is headed to your application, and you can interrupt the press and prevent it from propagating. For example, KeePass, in part, manually implements Windows BlockInput, which is part of the standard User32 lib.

Excluding Employee Traffic from Google Analytics, Pixels, and More

Joshua Tzucker — Mon, 01 Apr 2019 13:52:01 GMT

A question that I keep seeing pop up on digital marketing forums, and have had to tackle myself in the past, is how to exclude "internal traffic" from showing up in Google Analytics (e.g. when your own employees are browsing your own website). This post will cover a few different solutions, some of which can work for many different platforms other than just Google Analytics, such as preventing pixel fires coming from employees. There is a lot of misinformation when it comes to marketing technology, so please approach this topic with a critical eye and keep in mind that technology is also subject to change.

This is a very long and comprehensive guide to this issue, so you might want to jump to the section that interests you using the table of contents below:

Why Does This Matter?
- Google Analytics and Performance Metrics
- Digital Advertising and Spend
Intro to IP Address Exclusions and Cookie Based Exclusions
- Comparison of IPs vs Cookies
- Downsides to Geographic Based Exclusions
How to Implement IP Address Exclusions
How to Implement Cookie Based Exclusions
Summary, Conclusion, and Final Note

Quick Reminder: Why does this even matter?

If someone else told you to implement this, you might be wondering why it is even worth the time and effort to set up. Especially if your site gets a bunch of traffic already, what difference does it make if a few extra page views show up from internal traffic? Or employees occasionally see your own ads?

Why internal filtering is important for Google Analytics and other analytics or performance metrics platforms:

Although internal traffic might have a small impact on metrics like total page views, it is important to recognize it can greatly affect aggregate metrics like "avg session duration", "time on page", "pageviews per session", and many more.

Here is a practical example. Let's say that your company is working on developing a new landing page and wants to evaluate results. So far, 50 actual users have visited the page and each of them has only spent 10 seconds looking at the page before moving on to a different part of the site. There is a team of 2 employees at your company that worked on building the landing page and continue to tweak it, so each of their "time on page" is 30 minutes. If you look at the average time on page for that landing page in GA, you would see 79 seconds as the average time on page, which doesn't look that bad, despite the majority of users only spending a measly 10 seconds on the page! This can make it very hard to evaluate performance based on site changes, especially since in order to make site changes, more employees have to be interacting with the site.

Why internal exclusions are important for digital advertising (showing ads to employees):

The issue with showing ads to employees is mostly one that is tied to scale; if you are at a small business, this is much less of a concern. However, if you are a large business, displaying ads to employees can actually have a surprising impact on both budget and performance metrics.

Ad Spend: For display advertising, since impressions can be very inexpensive, it can be easy to forget how spend can add up. For example, lets say your average CPM for a remarketing campaign is $5. That's just $0.005 per impression. This is an extreme example, but if you have 200 employees at your workplace, your campaign is frequency capped at 60 impressions an hour, and your employees have visited your website but are not being excluded from targeting, assuming an 8 hour work day, you could potentially be wasting $480/day on internal impressions (((200 * 60 * 8)/1000)*5). That could be upwards of 14 grand a month in wasted budget! For PPC advertising, costs can skyrocket if your employees are seeing and clicking your own ads, especially if you are in an industry where clicks are more expensive, such as law firms.
Digital marketing metrics: Most people immediately worry about wasted spend when considering impressions wasted on employees, but the impact on metrics is important to consider as well. Employees are not likely to click on your own ads (and for good reason; one would hope they already are familiar with the place they work every day!) which will drive down your CTR for display campaigns. For paid search campaigns, such as AdWords, the negative effect on CTR is worse, since CTR and expected CTR is a component of quality score and ad rank, both of which can affect your final CPC and ad positions!
If your own employees are trying to find your company website (to log into a internal portal, read documentation, etc.), every time they search your company name on Google, if you have a brand campaign without employee exclusions, one of two things is likely to happen:
- A) They see your ad, click on it, and you have now just paid for your own employee to visit your own site. Or...
- B) They see your ad, they know it costs you money if they click on it, so they ignore it to scroll further down the page and click on the regular search result for your site. But, now Google has seen that user ignore your ad, which means your CTR will decrease and potentially your expected CTR metrics as well!

Introduction to IP Address Exclusion and Cookie Based Exclusion

The two main overall methods to block or exclude internal traffic is IP based filtering and cookie based exclusions. Here is a brief generic description of both:

IP address exclusion: Your "public" IP address is a special address, kind of like your postal address, that is provided by your internet service provider (e.g. Comcast, Frontier, etc.) and is unique to your network (the internet connection that is shared across devices at your work), but not unique per device on the same network. It is automatically sent by your computer as part of every web requests, such as loading a web page. Google Analytics does not collect IP addresses (at least not in a way you can view as a user), but allows you to exclude traffic based on it.
Cookie based exclusion: A cookie is a very small digital file that can contain arbitrary text, and websites can "set" cookies on your computer just by you visiting them. The power of cookies is that they flow in both directions, from the server to your computer, and then back again from your computer to the server. This is how websites know that you are a logged in user, and why if you clear your cookies, you suddenly need to log back into all your password protected websites. Using the power of cookies, it is possible to block internal traffic for a variety of purposes and across networks.

Comparison of IP Address vs Cookie Based Exclusion

First, here is a quick comparison of the two main methods that can be used to differentiate employee traffic from non-employee traffic - IP based exclusion vs cookie based exclusion:

IP Based Exclusion	Cookie Based Exclusion
No action required from employees to deploy	Employees have to either visit a special page or your business needs to already have some sort of internal site in use
Can be used to block: GA recording traffic Showing ads to your own employees* Can't be used to block: Most pixel fires FB pixel * = Depends on ad platform, more on this later. ** = Not without special setup, see section on advanced IP exclusion.	Can be used to block: GA recording traffic Most pixel fires FB pixel showing ads to your own employees * * = Depends on ad platform, more on this later.
Works across devices (as long as they are using the same network). If 200 employees are all sharing the same network, they can be blocked by excluding a single IP address.	Each device needs to have its own cookie set. Per device, each browser also needs its own cookie set (e.g. Chrome vs Firefox)
Each network (E.g. building A vs building B) would need an IP address filter set up for it. IP addresses can change, so lists would need to be kept up to date. Can't be used to block traffic if employees take laptops home, to conferences, travel, use VPNs or proxies, etc. Can't be (easily) used with dynamic IPs	Cookies persist, no matter how many times a device switches networks, even if it is a mobile device. Employees can manually clear cookies, which would require them to be re-registered with the blocking cookie.
One time setup is easy, if you just want to exclude traffic from Google Analytics or block paying to show ads to your own employees. However, requires more upkeep if IP addresses change, employees use VPNs or Proxies, etc.	One time setup is more complicated, but should stay working for a long time. Furthermore, a more advanced setup will allow you to block internal traffic for more than just GA.
You cannot (at least not easily) temporarily or permanently turn off exclusions for a single employee that uses a shared IP address you are blocking. Example: You have 200 employees you want to avoid showing your own ads to, but the CEO of the company likes seeing his own ads. This kind of setup is not really possible with IP based exclusions.	You can very easily disable exclusions for certain employees, or in a rush, simply use an incognito browser.
Testing exclusions is hard, or impossible	Testing exclusions is easy
No changes necessary to website	Changes have to be either made directly to website, or through tag manager like Google Tag Manager (GTM)

Why not use geographic based exclusion?

If you live in the digital marketing world, you might be familiar with location targeting and location exclusions available in most digital advertising platforms (AdWords, Facebook, DSPs / RTB systems). As such, you might also be wondering why I don't simply advise that everyone exclude the location of their office building from targeting and Google Analytics monitoring. The short answer is that, depending on the scenario, I might actually advise that you use both, or multiple combinations of exclusions and targeting (see summary at bottom of this post). The longer answer is a little complicated.

You see, the term "location" has become a very loose term in the digital marketing space, and depending on the technology stack you are using (Google Analytics, Facebook, etc.) it can mean very different things. On many platforms, location is actually determined by... IP address! In fact, Google Analytics currently uses IP-to-geography technology for its location metrics, as mentioned on their official help page. The problem with this, as they mention, is that mapping IP addresses to physical locations is "approximate", and from personal experience, not very reliable. IP-to-Geo works by looking up an IP address to a table (think Excel or Google Sheets) of known physical locations associate with that IP. These tables are usually bought and sold (examples: MaxMind, Neustar, etc.), and are usually created by other companies brokering data that they acquired through other companies that... well you get the picture. The end result is that IP-to-geo technology is notoriously variable in nature, and Google Analytics might report a location for a user that is 10-100 miles off from the location reported by AdWords / Google Ads for the same exact user!

If location is being determined by IP address, this also means that VPNs will introduce very inaccurate reports, as the location of the VPN provider will be recorded, not the user.

The good news is that although Google Analytics uses IP-to-geo, most ad platforms, including Google Ads, rely on newer and more reliable technology (**when it is available) to determine user location, like GPS, logged-in user PII, and even SSID (router names) scanning. So excluding your office building from targeting in your Google Ads campaign is actually not a bad idea. The only caveat here is the same as IP blocking; you can't easily turn it off for a single employee, so if your boss really wants to see their own ads, location exclusion is not going to work well for you, and you are much better off using cookies.

How to Implement IP Address Exclusions

As mentioned above, IP addresses can be used to exclude internal traffic from showing up in Google Analytics, as well as some other purposes. Here are instructions for each goal:

How to exclude employee visits, traffic, and events from showing up in Google Analytics by excluding IP addresses:

To exclude traffic in GA based on IP address, the steps are pretty simple. You can follow my steps below, or visit the official guide from Google.

First, gather the IP addresses that you will need to block. As a reminder, if you are trying to implement this for a large company, there might be more than one IP address being used at a single geographic location; each building might have its own IP address, or even each floor of a building. It all depends on how your IT department has set up networking. Make sure you are recording the "public" IP address, which is the one that is shared across employees. To find it, you can simply Google "what is my ip" on the network you want to get the IP of, or visit a site like this one.
Once you have the IP addresses you want to exclude traffic from, you can go ahead and setup a filter in the admin area of Google Analytics to exclude traffic from them. However, you need to pick which view to apply the filter to. I highly recommend having at least one "view" in your account that is unfiltered, so you might need to create a brand new view if you only have 1 view in your account at the moment.
Navigate to: Google Analytics -> Admin -> Property -> View (the view you want to add the IP filter on) -> Filters -> + Add Filter.
Give your filter a name, leave filter type as "predefined", select "exclude" as filter type, and then under "Select source or destination" select "traffic from the IP addresses", and for "select expression" use "that are equal to". Then paste in an IP address into the field below, and hit save!
If you have multiple IP addresses that you want to block traffic from, you will have to create a filter for each one using the standard method above. However, you can use Regular Expressions (aka Regex) to create a single filter that can block dozens of IP addresses at once. See this page for details, and/or use my custom built tool to quickly generate a pattern for you!

How to use IP exclusions for digital advertising (AdWords, Display Ads, etc.)

There are so many advertising platforms out there, so my main recommendation here is to simply search the help guides/ documentation for the platform you are using, or reach out to an account rep and ask if IP exclusions are possible for your account. To help expedite this, here is a table of guides on IP excluding per platform, or a note if it is not possible:

AdWords / Google Ads	Link
DoubleClick Bid Manager / Display & Video 360	Carrier and ISP Targeting [IP exclusion might or might not be possible natively]
Facebook Ads	Not possible natively
Bing Paid Search	Link
AdRoll	Link

Advanced: Blocking remarketing tags/pixels from firing based on IP address:

If the ad platform you are using does not support IP based exclusions, and/or you want to be able to block remarketing/retargeting tags from even firing in the first place based on IP, that is actually possible.

Without Google Tag Manager:

First, here is a developer oriented solution that does not require Google Tag Manager, or any special JS libraries like jQuery. You can conditionally load marketing tags either server-side (PHP, NodeJS, etc.) or client-side (Javascript), with server-side being the recommended method.

Here is client side first, as a vanilla JS solution (un-minified source):

!function(){window.conditionallyLoadTags=function(t,n,o){n="function"==typeof n?n:function(){},o="function"==typeof o?o:function(){};var e,i,a=!1,c=!1;if("object"==typeof t&&null!==t&&("object"==typeof t.cookie&&(e=t.cookie.name,((i=document.cookie.match("(^|;) ?"+e+"=([^;]*)(;|$)"))?i[2]:null)==t.cookie.value.toString()&&(a=!0)),!a&&"object"==typeof t.ipAddresses)){c=!0;var f=t.ipAddresses;!function(url,method,callback,OPT_failCallback){method="string"==typeof method&&""!==method?method:"GET",callback="function"==typeof callback?callback:function(){},failCallback="function"==typeof OPT_failCallback?OPT_failCallback:function(){console.warn("httpFetchVanilla failed")};var t=new XMLHttpRequest;t.onreadystatechange=function(){4==t.readyState&&(200==t.status?callback(t.responseText):failCallback())},t.open(method,url,!0),t.send()}("https://api.ipify.org/?format=json","GET",function(t){try{var e=!1,i=JSON.parse(t).ip;Array.isArray(f)?-1!==f.indexOf(i)&&(e=!0):f.test(i)&&(e=!0),!0===e?o():n()}catch(t){n()}},function(){n()})}c||(!0===a?o():n())}}();
conditionallyLoadTags({
‘ipAddresses’ : [‘216.3.128.12’,‘19.117.63.126’]
},function(){
console.log(‘You are not an employee. Firing tags.’);
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag(‘js’, new Date());
gtag(‘config’,‘UA-12345678-01’);
},function(){
console.log(‘EMPLOYEE RECOGNIZED!!! Not firing tags.’);
});

For a server side solution, it can vary greatly based on what technology you are using, so here is a generic solution that would work across most PHP based sites:

Using Google Tag Manager

Using IP addresses with Google Tag Manager (GTM) introduces some additional complexity, but allows for more reusable code, since once it is set up, you can easily assign an IP address variable as a blocking trigger for multiple marketing and collection tags.

I'll defer to Simo Ahava's guide on implementing this, as it does an adequate job and he is kind of the king of GTM stuff.

Once this is setup, you can set up all kinds of conditional tags that fire or don't fire for internal employees. You can block FB pixels from firing, remarketing tags, conversion pixels, and more.

How to implement cookie based exclusions:

This is where I see a lot of guides fall short, despite cookies being a great way to differentiate employees vs non-employees, especially if you have internal webpages that only employees visit (e.g. employeeportal.example.com, example.com/internal/wiki-page, etc.).

Without GTM:

The summarized version of how this works, is that whenever an employee visits a page that only an employee could see (Wordpress Admin section, internal wiki page, task management page) we use Javascript to set a cookie that basically says "Hey, server! I'm an employee!". Then in our website server code, we check for the existence and value of that cookie - if we see it, we don't return tracking code that we return for all other users.

I actually use this exact method on this website; I don't like logging my own hits to GA, so whenever I am logged into Wordpress and browsing my own site, I make sure a cookie is set marking myself as an internal user, which doesn't expire for a full year. You can see this actual code on the Github repo for this site.

If you are looking for some generic code, you can drop this Javascript on any internal page where you want to set the "employee marking cookie":

function setCookie(name, value, days){
	var d = new Date;
	d.setTime(d.getTime() + 24*60*60*1000*days);
	document.cookie = name + "=" + value + ";path=/;expires=" + d.toGMTString();
}
setCookie('isInternalEmployee','true',365);

And then for conditionally loading tags based on that cookie, you can either implement code server side like this:

<?php if($_COOKIE['isInternalEmployee']!=='true'): ?>
    <!-- Put all the tags you want to conditionally load here. GA example below: -->
&lt;!-- Global Site Tag (gtag.js) - Google Analytics --&gt;
&lt;script async src="https://www.googletagmanager.com/gtag/js?id=GA_MEASUREMENT_ID"&gt;&lt;/script&gt;
&lt;script&gt;
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'GA_MEASUREMENT_ID');
&lt;/script&gt;
<?php endif; ?>

Or client side, with Javascript, like this (un-minified source):

!function(){window.conditionallyLoadTags=function(t,n,o){n="function"==typeof n?n:function(){},o="function"==typeof o?o:function(){};var e,i,a=!1,c=!1;if("object"==typeof t&&null!==t&&("object"==typeof t.cookie&&(e=t.cookie.name,((i=document.cookie.match("(^|;) ?"+e+"=([^;]*)(;|$)"))?i[2]:null)==t.cookie.value.toString()&&(a=!0)),!a&&"object"==typeof t.ipAddresses)){c=!0;var f=t.ipAddresses;!function(url,method,callback,OPT_failCallback){method="string"==typeof method&&""!==method?method:"GET",callback="function"==typeof callback?callback:function(){},failCallback="function"==typeof OPT_failCallback?OPT_failCallback:function(){console.warn("httpFetchVanilla failed")};var t=new XMLHttpRequest;t.onreadystatechange=function(){4==t.readyState&&(200==t.status?callback(t.responseText):failCallback())},t.open(method,url,!0),t.send()}("https://api.ipify.org/?format=json","GET",function(t){try{var e=!1,i=JSON.parse(t).ip;Array.isArray(f)?-1!==f.indexOf(i)&&(e=!0):f.test(i)&&(e=!0),!0===e?o():n()}catch(t){n()}},function(){n()})}c||(!0===a?o():n())}}();
conditionallyLoadTags({
‘cookie’ : {
‘name’ : ‘isInternalEmployee’,
‘value’ : ‘true’
}
},function(){
console.log(‘You are not an employee. Firing tags.’);
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag(‘js’, new Date());
gtag(‘config’,‘UA-12345678-01’);
},function(){
console.log(‘EMPLOYEE RECOGNIZED!!! Not firing tags.’);
});

dasf

Using Cookies with GTM to conditionally fire tags:

The nice thing about GTM, is you can set up all of the above without needing to actually touch server-side or front-end code - you can built it all entirely within GTM. Here is how I would approach that.

Add the tag that drops the employee cookie:

First, we are going to built a tag that sets the employee cookie. That way we can easily assign that tag and cookie to hundreds of internal employee pages, as long as they have GTM installed. We can reuse the code above, but wrap it in a script tag:

<script>
function setCookie(name, value, days){
	var d = new Date;
	d.setTime(d.getTime() + 24*60*60*1000*days);
	document.cookie = name + "=" + value + ";path=/;expires=" + d.toGMTString();
}
setCookie('isInternalEmployee','true',365);
</script>

and put it in a script tag, like this - go to "Tags -> New -> Choose Tag Type -> Custom HTML":

Now, you need to tell GTM which pages your employees are visiting and on which it should drop this cookie. If you have already set up a trigger for internal pages, you can assign it, otherwise you will need to create one from scratch. You can use simple URL pattern rules to target pages like example.com/internal/wiki or dev.example.com, but if you have a more advanced internal system or CRM, more setup might be necessary. You assign the pages as a "Page View" trigger to the cookie-setting tag, which you can see assigned above as "Internal Employee Pages".

Configure GTM to check for the value of that cookie and store as a variable

Now, we need a way to access this cookie inside of GTM and check if it is there or not. We can use a GTM "variable" to do this. Go to "Variables -> User-Defined Variables -> New -> Variable Configuration -> 1st Party Cookie". Now use the same name of the cookie that we are setting above as the cookie to retrieve - if you haven't changed anything, it should be "isInternalEmployee", like this:

Now, lets set up a trigger based on the value of this cookie, so we can either exclude or target employees with any given tag. Go to "Triggers -> New -> Choose trigger type -> Page View". Change "This trigger fires on" from "All Page Views" to "Some Page Views", and then select the cookie we created as the condition to test against, and then "equals" or "contains" the string we set.

OK, now we can use this trigger as a "blocking" trigger on any tag! Google Analytics, Google Ads Remarketing, Facebook, Custom HTML, you name it! On any tag, once you have defined what triggers it, in the same "Triggering" section, just hit the "Add Exception" button, then select the "Employee View" trigger that we created to have as an exception - this will prevent the tag from firing if someone has the employee cookie!

Further note: Instead of excluding employees, you can also target them

There is probably not much use for this, but still something cool to think about: you can use the strategies above to target employees for campaigns instead of excluding them. What sort of campaign would need that? Well, for a large company, for example McDonald's, maybe they want to promote an employee benefit, like paid managerial classes or an internal job opening. Using a cookie based approach, you could create a remarketing campaign targeting just your own employees for this goal. Or maybe you want to deploy special tracking just for employees; if you have a public-facing wiki, you might care more about how your employees use it than the general public, so you can know which topics need more training materials.

What about Local Storage (localStorage)?

Local storage (aka localStorage aka Web Storage) is a browser feature similar to that of cookies. It lets websites save and retrieve small amounts of text data to your browser, which is persisted even if you close and open your browser back up. Like cookies, localStorage is "scoped" to the originating site, so that example.com can't access the localStorage set by joshuatz.com.

You could easily swap out most of my cookie code for code that would store the internal user flag in localStorage. See this page for an example.

The benefit to localStorage over cookies is that localStorage is kept even if a user clears their cookies. However, this is only true if a user individually clears cookies; using Chrome or Firefox's "clear browsing data" to erase everything will also clear out localStorage (this is something that a lot of people miss when recommending localStorage over cookies for internal traffic blocking).

The immediate downsides to localStorage over cookies are simply that it is slightly more complicated and less supported (cookies have been around for longer and act pretty much exactly the same on any given browser). However, there is also a less obvious downside: certain browsers (e.g. Safari) will not persist localStorage values in an incognito browser window, even in the same tab. This seems unlikely to have a significant impact, but my point is really that there is not a great reason to pick localStorage over cookies, at least for the time being.

A further disclaimer here is that there continues to be pushback against the use of cookies and web tracking technology, both from consumers as well as developers. As such, browsers continue to change how they let sites interact with browser technologies, a recent example being Safari choosing to block third party cookies by default. It could be that in a year from now, my advice here will be less valid as first party cookies come under scrutiny. Or maybe localStorage read/writes get limited because of some security exploit, so cookies remain the winner. Who knows what the future holds.

Summary, Conclusion, and Notes:

As you might have already guessed, my preference is for cookie based exclusions over IP address or geography based exclusions. As someone who understands the "behind the scenes" technology, it seems clear to me that where cookies are reliable, technically sound, and configurable, things like IP address and geographic targeting can be "black boxes" of data that have passed through hundreds of brokers before reaching you, and you are not always getting what is promised. However,there really is no harm in combining many of these strategies together as you see fit. For example, maybe for a branded PPC campaign that has an incredibly high CPC, you use all three exclusions to prevent any employees from clicking on your own ads, but then on a general low-cost remarketing display campaign, you just set up a simple cookie exclusion in GTM, that way some employees can still see the remarketing ads if they want to.

If you disagree with anything here, feel free to chime in. I am happy to make adjustments to the article if I find your reasoning sound and based in facts. In general, these kinds of topics could use a lot more discussion in the digital marketing world :)

IP Address Regular Expressions Generator for Google Analytics

Joshua Tzucker — Sun, 31 Mar 2019 11:51:54 GMT

I threw together this tool (took about 15 minutes to make) for a related post about filtering traffic by IP address in Google Analytics views. Excluding multiple IP addresses at the same time with a single filter in GA is only possible by using a custom IP address filter pattern, which uses regular expression pattern matching. This tool is for those that want to quickly generate a regex pattern to use based on a list of IP addresses.

Simply paste your list of addresses into the left box, and than copy the generated pattern from the right box and paste right into your new Google Analytics filter!

Click the "Demo" button to see an example. You can validate results using this tool.

See the Pen <a href="https://codepen.io/joshuatz/pen/MRWboq/">IP Address Regex Filter Generator</a> by Joshua T (<a href="https://codepen.io/joshuatz">@joshuatz</a>) on <a href="https://codepen.io">CodePen</a>.

The code behind this tool can be found here.

NodeJS - Interacting on CMD Line / Shell - WITHOUT Creating a JS File

Joshua Tzucker — Tue, 19 Mar 2019 00:51:36 GMT

I was having trouble finding this info in one spot, so I made this post as "refresher" to myself on some easy ways to interact with NodeJS from the command line, and without needing to have an actual JS file as the input. The main reason why I wanted to figure out how to do this, was so I could check my NodeJS ABI version with a single command from my CLI

As usual, one of the best sources of info is the official docs. Here is the documentation page for the Node command line interface (CLI).

The built in "REPL"

REPL, which is an acronym for "Read-Eval-Print-Loop", is a feature baked into the Node.js standard installation. From your command line, simply type "node" and press enter without anything after it. You will enter the REPL, in which you can type out JavaScript and instantly execute it with results. By default, Node will even detect when you are trying to write a multi-line statement (for example, a for loop), and will let you write it out before executing.

Alternative, you can also type .editor and press enter, after entering the REPL, to enter the "editor mode". This will let you write out a longer script in the console, including line breaks, and only executing it once you are ready.

For more details and a better guide, check out Flavio's guide here - https://flaviocopes.com/node-repl/

Using "print" (node -p)

The -p (or -print) argument basically evals the string following it and prints the result. For example, to get the ABI #, we can use:

node -p "process.versions.modules"

Using "eval" (node -e)

The "eval" option takes a string input and "evals" it, which is to execute it as though it were raw code being passed in. Since we are running in the command line interface and node will exit immediately after eval'ing, if we want to see the result of the eval option, you either need to use "node -p" instead, or in your code, make sure output is going to be passed to the console. For example, this results in the same result as my node -p example:

node -e "console.log(process.versions.modules)"

Don't forget about piping!

Just like with almost anything on the CLI, you can pipe data to node, by using the pipe operator ("|"). For example, to print "hello world" to the console:

echo "Hello World" | node -p

... Or, pipe the output of Node to something else! For example, pipe it to grep to search for a pattern:

node -p "process.versions" | grep "openssl:\s*'(?:[^\'])+" -P

Or send version info to a text file:

I got what I wanted:

Here is a single line command that will spit out the node version and ABI # to command prompt, with some line breaks around it so it is easy to read (this is the same command shown in the thumbnail for this post):

node -p "'\n\n\n\n   Node Version = ' + process.versions.node +'  ||  ' + 'Node Version ABI # = ' + process.versions.modules"

Additional Info:

Getting installation paths:

Get the install path of NPM by using:

npm config get prefix

Or by using:

which npm

Getting QT Creator Working for Android Development - First Steps

Joshua Tzucker — Tue, 12 Mar 2019 00:44:48 GMT

This post was written as I worked through trying to get QT Creator to be able to, at the very least, build and run an example project for Android and see it on an emulator.

Errors encountered:

"Android build SDK not defined"
- Under "Projects > Active Project > Build" you need to specify which Android Build APK to target / build for

- - If this dropdown is empty, it probably means that your SDK configuration is not set up properly. See below, and if you are using Android Studio, you might need to try manually downloading and installing the Android SDK separately on your hard drive

For "Select Android Device" when trying to run app, AVD emulator shows under "Incompatible Devices", with "API Level of device is: -1"
- I think that, at least in my situation, this was due to a XML mismatch between what avdmanager generates and what QTCreator is expecting. It looks like for older versions of the Android API, the AVD XML config file does not have the "AvdId" property, whereas newer ones do (or maybe it was because I created my new AVD from the command line avdmanager instead of Android Studio). Either way, it looks like QT needs "AvdId" to be in the XML file for it to parse it correctly.

- - You can find your generated AVD config XML file under C:\Users\[USERNAME]\.android\avd\[AVD_ID]
  - I simply added the missing AvdId property to my XML file that matched the ID and the folder name
  - Afterward, the AVD instance started showing under compatible devices, although in the Devices panel it still shows with "AVD Target: API -1" which is not correct.

Under "Build Settings > Build Steps > Build Android APK", only SDK version 25 ("android-25") appears selectable in the "Android Build SDK Dropdown"
- After installing a new SDK, either through the QT GUI or the sdkmanager, I had to reset my SDK path field to get QT Creator to re-scan the directory, then completely close and open QT creator back up again
- Also, make sure you have installed both the emulator image, as well as the "SDK Platform" under the android version you want
When trying to run on an emulator generated by Android Studio, you get this "AVD Start Error" : "emulator: ERROR: unknown skin name 'nexus_5x'
- Similar to another issue, this seems to be something to do with how QT parses android config files vs how Android Studio does. You need to find the "config.ini" file that goes with your emulator instance. For example, if you are trying to get the default Android Studio x86 emulator running, it might be at something like "C:\Users\Joshua\.android\avd\Nexus_5X_API_28_x86.avd\config.ini"

- - Open it, and look for the line with the keypair of "skin.path". It probably looks like "skin.path=skins\nexus_5x". You will want to change this to point to the actual full path of the skin, like "skin.path=C:\Users\Joshua\AppData\Local\Android\Sdk\skins\nexus_5x"

When trying to run on my actual Android phone, not the emulator, my phone shows up under "incompatible devices", with a message about "ABI is incompatible"
- In my case, I had selected the kit to target Android x86, but my phone (a OnePlus 2) has an ARM 64 processor. I just had to switch my kit before running and then it worked!

When trying to use any QtQuick submodules (e.g. qtquick.controls) I would get an error in my QML like "QML module not found (QtQuick.Controls)

This is probably an indicator that my kits are set up incorrectly, as QT *should* be able to automatically resolve QML dependencies for Android vs Desktop, but for whatever reason, in my case, it is not. I had to add conditional sections to my project/qmake file that modifies the QML_IMPORT_PATH variable to pull in the right QML sources based on Android version (based on tip here). See below:

# Android specific build settings
android {
    log("ANDROID_TARGET_ARCH=")
    log($${ANDROID_TARGET_ARCH})
    equals(ANDROID_TARGET_ARCH, armeabi-v7a) {
        #[doAndroidArmStuff]
        QML_IMPORT_PATH += C:/Qt/5.12.1/android_armv7/qml
        QML_IMPORT_PATH += C:/Qt/5.12.1/android_armv7/qml/QtQuick
    }
    equals(ANDROID_TARGET_ARCH, arm64-v8a) {
        #[doAndroidArmStuff]
        QML_IMPORT_PATH += C:/Qt/5.12.1/android_arm64_v8a/qml
        QML_IMPORT_PATH += C:/Qt/5.12.1/android_arm64_v8a/qml/QtQuick
    }
    equals(ANDROID_TARGET_ARCH, armeabi) {
        #[doAndroidArmeabiStuff]
    }
    equals(ANDROID_TARGET_ARCH, x86)  {
        #[doAndroidx86Stuff]
        QML_IMPORT_PATH += C:/Qt/5.12.1/android_x86/qml
        QML_IMPORT_PATH += C:/Qt/5.12.1/android_x86/qml/QtQuick
    }
}

Pre-requirements

Download and install Android Studio, since that will auto-install a bunch of helpful tools and emulators
Download and Install the Java JDK
- Right now, version 8 works, whereas >8 breaks
- Make sure to set windows user var JAVA_HOME to this path
Create a root level folder for easy access - C:/Android
- Manually download the Android NDK and install there, like "C:\Android\ndk\android-ndk-r19b"
- If you want to manually install Android SDK here, create C:\Android\sdk
  - Now, download and extract the command line SDK tools download from Google (the one that contains the sdkmanager)
  - Navigate to sdkmanager (should be under c:\Android\sdk\tools\bin after you extracted) and in cmd prompt, run "sdkmanager "platforms;android-25" or whatever version of the SDK you want to install

Setup

Now, go to Qt Creator > Tools > Options > Devices, and under "Android", plug in the necessary paths (Java JDK, Android NDK & SDK)

QT should automatically create kits for you and detect installed Android Platforms / Emulators
If you are using a manual SDK path outside of Android Studio, QT will also prompt you to auto-download and install extra tools it needs
- However, it won't install emulator stuff automatically…
- It won't auto download the emulator process, so make sure you check the install box for "Android Emulator"
- it won't auto download an emulator image, so you need to go to the "SDK Manager" tab and install at least one system image to use as an emulator (with AVD Manager). You usually want "Google APIs Intel x86 Atom System Image"
  - For example, here I have download an Android 7.1 image, and then created an AVD for it

- If the QT avd manager is not working (it wasn't for me) you can manually provision with the cmd tool for it. This command is functionally equivalent to the result of the dialog above
  - avdmanager create avd -n "Android_a25_s7.1_x86" -k "system-images;android-25;google_apis;x86" -c 400M

Nested Opacity Inheritance in QML and CSS - How does this work again?

Joshua Tzucker — Fri, 22 Feb 2019 21:29:28 GMT

The Problem:

I'm brand new to QML, and so when I started on my first project and couldn't get nested opacity inheritance to work like I was expecting it to, I immediately blamed QML and thought "this would be so much easier with CSS". However, after reflecting for a second and firing up a Codepen to test, I realized I was wrong, and I had incorrectly remembered how CSS handles the opacity property in nested elements (hint, it is the same as QML in this case).

The problem is pretty basic; in both CSS and QML, if you nest a component (CSS people, think "element"), the child element ends up somewhat inheriting the opacity of the parent, to the effect that the child can not have an opacity that is more than the parents opacity. In practice, this means that by only manipulating the opacity property, a nested component can be less opaque than the parent, but not more opaque. Here is a super simple example:

Works as expected: Nested element opacity = 0.5, parent element opacity = 1.0

Doesn't work as expected: Nested element opacity = 1.0, parent element opacity = 0.5

You can see in the example above where the nested component has an opacity of 1, you might think that the child should appear with completely solid background, while the parent has a transparent red background. However, due to the way that both CSS and QML handle blending, relative values, rendering nodes, and other technicalities beyond the scope of this post, the practical result is that, in actuality, the child ends up with an opacity capped at that of the parent, so both appear as 0.5.

Here is a fun little demo I set up to show how this works with CSS. Use the slider to change the opacity, and the button to swap which elements are opaque.

Another way to think of it that might make more sense is by referring to opacity as a relative percentage - in this case, if the parent is set to 50% opacity, and the child is set to 100%, 100% of the parent (50%) is still just 50%, so both end up with the same value.

You could also think of a piece of paper with two squares printed on it, one is half as "bright" as the other. If you submerge the whole paper in a solution that fades it to be half as visible, now both squares are half as visible as they were before (50%) and one square is still half as "bright" relative to the other, which means it really is 25% visible (0.5 * 0.5).

The Solution:

One of the easiest solutions, in both CSS and QT / QML, is to just avoid nesting when you can. Here is a comparison showing how you can have a component appear within another with a higher opacity when it is really a sibling, vs when it is a child:

But what if, understandably, you really want to leave you components in a nested structure, but still get nested opacity to work? Well, there is a few options, but the easiest, and my favorite, is one that works for both CSS and QML - simply use RGBA (RGB-ALPHA) instead of opacity! This is actually the solution that I use throughout this website, and I use it so often that I forgot I do it, and is the reason why when I started using opacity instead of RGBA in QT I had to remind myself of why I'm always using it.

Here is how you might use RGBA to display the solid green rectangle inside the transparent red one, like above, with CSS (codepen link):

Or with QML (full code repo here):

A cool thing in QML - Reusing colors with RGBA

One thing that is very neat about QML that you can't really do in CSS (at least not in pure CSS, I'm not talking about Sass or JSX) is referencing values from other components. A cool application of this in combination with RGBA is telling QML to use the same color as another component, but fade it a bit. For example, this is a great way to achieve a nice glow effect around an element without having to keep hardcoding in colors. This also allows you to keep the same opacity in a nested component while allowing for a dynamic color (e.g. if the component color is bound to a C++ value).

Further options in QML:

QML has some additional options for altering how inherited opacity works, such as the "layer.enabled" property, or shader effects. However, at least in my view, these are all much more complicated and with potential side effects that far eclipse the simple implementation of RGBA.

Another quick solution similar to RGBA though, is use a quad color code.

QT - FontAwesome, Import, and Using Files Outside Source Directory

Joshua Tzucker — Fri, 15 Feb 2019 02:39:01 GMT

Disclaimer: This was written while working on my very first QT and C++ project. Take with a very large grain of salt.

Oh boy. Whenever you think something is going to take 5 seconds and be super simple, that is when it takes all day and makes you want to smash your keyboard into bits. Enter the world of trying to use external resources in QT, relative paths, QRC files, and .PRO files. Not fun.

I wanted to use FontAwesome (an icon set) with QML, but I didn't want to add the FontAwesome files into my actual QT workspace, for several reasons. One of which is that this is something I would prefer to use a package manager, like NPM, for.

So, in this case, I set up a simple package.json a level up from my QT project directory, npm installed FontAwesome, and then attempted to use the files in QML. My file structure looked something like this:

package.json
node_modules
lib
Actual_QT_Workspace
- my-project.pro
- main.cpp
- qml.qrc
- ...

However, I quickly ran into an issue...

Issue 1: How to include files outside the Source directory in resources, so you can use "import" with them?

The very first thing I tried (man can you imagine if this simply worked) is using a typical relative path syntax in my QML file pointing to the path of the file in node_modules, like this:

import "../node_modules/@fortawesome/fontawesome-free/js/all.js" as FontAwesome

I thought it might work at first, since I am new to QT and didn't know any better, and also that the editor autocompleted the path, showing that it recognized the file as existing. However, trying to build resulted in this kind of error:

qrc:/main.qml:5 Script qrc:/node_modules/@fortawesome/fontawesome-free/js/all.js unavailable
qrc:/node_modules/@fortawesome/fontawesome-free/js/all.js:-1 No such file or directory

This is because, and I'm grossly over-simplifying here, but QT uses QRC (QT Resource Collection) files to bundle resources together that get passed to the compiler. In order for my application to use a file off the disk, the easiest way to access it is by specifying the file in a QRC that is included as part of the build process.

Using a relative path in a .qrc file

So the next thing I tried was adding this local file to the default QRC file, qml.qrc. I tried using the built in QRC editor panel in QT Creator (right click *.qrc, hit "Open in Editor", then use the "Add > Add Files"), but when trying to add the file, I got this error:

The file C:\Users... is not in a subdirectory of the resource file. You now have the option to copy this file to a valid location.

No! I don't want to copy the file - that is the whole point of this - I want it left in node_modules. This actually seems like a bug, because you can easily get around this by right clicking on the *.qrc and instead of using the editor, hit "Add existing files...", then select the file you want to add. This actually worked! However, it was a little silly, because all it does is use the relative path in the <file></file> element in the QRC doc - I could have easily have tried that by hand instead of using the menu. The resulting QRC ended up looking something like this:

<RCC>
    <qresource prefix="/">
        <file>main.qml</file>
        <file>../node_modules/@fortawesome/fontawesome-free/js/all.js</file>
    </qresource>
</RCC>

Another option is to create a brand new QRC file, leave it in the parent directory / outside directory, and include the QRC in your .PRO file as part of RESOURCES. Then in your new QRC file, you don't always have to keep traversing up a directory (or multiple, depending on your structure). For example:

.pro file:

# ...
RESOURCES += qml.qrc
RESOURCES += ../external-assets.qrc
# ...

And then in external-assets.qrc :

<RCC>
    <qresource prefix="/">
        <file>node_modules/@fortawesome/fontawesome-free/js/all.js</file>
    </qresource>
</RCC>

Issue 2: You can't use FontAwesome like that Joshua!

Bah! I should have known that would be too easy to work! Although QT ships with a JS engine, and you can definitely use JS with it, you have to remember that there are many many caveats to that. In short, you can't just import FontAwesome's all.js into your QML and have it load everything; for one, all.js loads a bunch of CSS files, of which QML does not natively parse / apply. In addition, you'll get errors about setTimeout, and other stuff like that where all.js is trying to use normal window JS globals, but of course QML!==window DOM.

So... how do we get around this? Well, unfortunately it is a little complicated. Lucky for me, I found this great in-depth guide by Martin Höher that lays out how to use FontAwesome files with QML, step by step. Although Font-Awesome recently bumped up versions from 4.x to 5.x, his Python script for generating the Icons.qml Singleton still works perfectly! However, currently Font-Awesome is excluding icons.json from their NPM package, which is what his script uses as the input. You can still find the icons.json if you manually go to their website and download the zip, which is what I am doing in the meantime to avoid the headache of having to parse their icons without a .json index.

I'm still keeping the font-awesome files outside my QT directory, although there is not much reason to at this point while I currently can't really use the NPM version of FA. I just extract the four files I need from the downloaded zip and place them in one flat directory under /lib/.

Here is close to what my files look like now:

qmldir:

singleton CustomFonts 1.0 CustomFonts.qml
singleton Icons 1.0 Icons.qml

CustomFonts.qml:

pragma Singleton
import QtQuick 2.0
Item {
id : customFonts
readonly property FontLoader fontAwesomeRegular: FontLoader {
source: ”../lib/fontawesome-free-5.7.2-web/fa-regular-400.ttf”
}
readonly property FontLoader fontAwesomeSolid: FontLoader {
source: ”../lib/fontawesome-free-5.7.2-web/fa-solid-900.ttf”
}
readonly property FontLoader fontAwesomeBrands: FontLoader {
source: ”../lib/fontawesome-free-5.7.2-web/fa-brands-400.ttf”
}
readonly property var names : {
    "fontAwesomeRegular" : customFonts.fontAwesomeRegular.name,
    "fontAwesomeSolid" : customFonts.fontAwesomeSolid.name,
    "fontAwesomeBrands" : customFonts.fontAwesomeBrands.name
}
}

main.qml

import QtQuick 2.7
import QtQuick.Controls 2.3
import QtQuick.Layouts 1.0
import QtQuick.Controls.Material 2.3
// Singleton import
import "."
ApplicationWindow {
id : root
visible: true
// ...

// Chrome icon
Text {
    color: "black"
    font.pixelSize: 40
    text: Icons.faChrome
    font.family: CustomFonts.names.fontAwesomeRegular
}

// ...
}

qml.qrc

<RCC>
    <qresource prefix="/">
        <!-- ... -->
        <file>../lib/fontawesome-free-5.7.2-web/fa-regular-400.ttf</file>
        <file>../lib/fontawesome-free-5.7.2-web/fa-solid-900.ttf</file>
        <file>../lib/fontawesome-free-5.7.2-web/fa-brands-400.ttf</file>
        <file>../lib/fontawesome-free-5.7.2-web/icons.json</file>
        <!-- ... -->
    </qresource>
</RCC>

To-Do:

There are lots of ways I could optimize how this is loaded, especially if I want to be considerate of how others might have different build systems. If I find time, I might revisit this post, especially as I learn more about QT and its internals.

wow.js - force trigger after display visibility change

Joshua Tzucker — Thu, 14 Feb 2019 07:17:14 GMT

Wow.js is a very light-weight Javascript library for automatically showing simple animations on page scroll.

Most of the time it works perfectly without issue, but today I ran into a curious issue where I couldn't get it to trigger when an element that should have animated was initially hidden and then became visible or "revealed" (via it's parent element having display:none or something similar and then becoming visible). In fact, where the element should have shown up and animated, there was just a white empty space.

There are a few threads related to this problem, such as this open issue, but I couldn't quite get any of the solutions listed to work for me, and was having trouble finding docs on the internals of wow.js.

However, after just a little digging, I was able to find some easy ways around this. For pretty much all of these solutions, make sure that you have a window level variable pointing to your WOW instance, like this:

// wow.js init
window.wow = new WOW({
    // settings here
});
wow.init();

Easiest solution:

The easiest solution is to simply call "wow.show(element)", where element is the element that needs to be animated.

wow.show(element);

Here is an example where .card-reveal is the parent element that is changing visibility, and I’m forcing WOW to animate when someone clicks the button that changes that visibility:

// Get all expandable cards
$('div.card > .card-reveal').parent().each(function(index){
    var card = this;
    // Attach listener to all activator triggers
    $(card).find('.activator').on('click',function(){
        $(card).find('.card-reveal .wow').each(function(){
            wow.show(this);
        });
    });
});

Force re-animation regardless of scroll:

Finally, if you need to "reanimate" an element, regardless of scroll position, you will want to use this code, since by default WOW will not trigger an animation more times than once, and not more iterations than specified by "data-wow-iteration".

function forceWowReanimation(element){
	element.classList.remove('animated');
	element.style.removeProperty('animation-iteration-count');
	element.style.removeProperty('animation-delay');
	element.style.removeProperty('animation-iteration-count');
	element.style.removeProperty('animation-name');
	wow.applyStyle(element,true);
	setTimeout(function(){
		wow.show(element);
	},10);
}

The setTimeout in the above code is normally something I would try to avoid, but in this case it is necessary because wow.show sets visibility to "visible" regardless of arguments, so if I don't give a delay between applyStyle (which sets visible to "hidden") and .show(), the animation will not trigger.

Key Codes Reference Spreadsheet - ASCII, QT, X11, Etc.

Joshua Tzucker — Wed, 13 Feb 2019 12:24:48 GMT

Links:

I've just started working on a project that involves keypresses and character codes, and just skimming the surface of what it entails, I can tell this is going to be a headache to implement. From a cursory glance, it seems like a lot of this information is spread out all over the place, from OS to OS, to framework to framework.

I'm trying to use this post as a way to compile some of the more crucial information, mostly for myself, but I also thought others might find it useful as well.

Update 4/3/2019 - Added Mac Keycodes to master Google Sheet. Mac/OSX uses CGKeyCode provided through the macOS SDK.

Cloudinary WYSIWYG Visual Editor for Transformations

Joshua Tzucker — Sat, 09 Feb 2019 15:39:59 GMT

If you are looking to jump directly to the tool, click here to open it (in a new tab)

Source Code: https://github.com/joshuatz/cloudinary-wysiwyg

6 Minute Intro Video

This video covers most of the content on this page, so feel free to give it a view instead of, or in addition to the content below.

Spreadsheet used in Demo Video with samples and formulae:

Link: Here

What is Cloudinary?

Normally, when you combine images, text, and other elements into a new compiled image, perhaps with Photoshop, in order to share that resulting image with other people, you would need to save it and then upload it somewhere. Cloudinary changes the game; upload a "base" image, and then you can apply as many transformations, effects, or overlays to that image simply by manipulating a shareable URL. This is easier to show than it is to explain, so here is an image that shows 4 different URLs, all generated from one original image (the top left image):

What is the Cloudinary-WYSIWYG Editor Tool?

I found the concept of Cloudinary fascinating, and I realized that their transformations (crop, rotate, skew, etc) could be combined in ways that pushed the limits of the platforms, but a major limiting factor is how confusing they can be to manually join together and keep track of. To illustrate that, here is a URL that was generated by this editor tool, but would have been practically impossible to parse out by hand:

https://res.cloudinary.com/ACCOUNT_NAME/image/upload/c_fill,h_500,o_40,w_500/h_224,l_pixel,w_224/co_rgb:070707,e_colorize,fl_layer_apply,g_north_west,x_28,y_75/h_395,l_pixel,w_454/co_rgb:2744a6,e_colorize,fl_layer_apply,g_north_west,x_23,y_35/h_210,l_pixel,w_210/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_34,y_82/c_scale,h_188,l_profilepicture,w_215/fl_layer_apply,g_north_west,x_34,y_107/h_209,l_pixel,w_208/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_254,y_82/l_text:Roboto_37_normal:Stats:/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_316,y_93/l_text:Roboto_19_normal:Flowers%20Pollinated%20:%2020/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_266,y_141/l_text:Roboto_19_normal:People%20Chased%20:%2025/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_266,y_167/l_text:Roboto_55_normal:Buzzy%20McBuzzer/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_52,y_314/l_text:Roboto_31_normal:Rank%20:%2010/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_170,y_380/h_119,l_pixel,w_119/co_rgb:c8c61f,e_colorize,fl_layer_apply,g_north_west,r_59,x_295,y_194/l_text:Roboto_42_normal:%3C10%3E/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_308,y_237/sample

This editor let me build that URL by dragging and dropping shapes, images, and text, all without ever needing to manually edit a URL or understand how editing steps convert to Cloudinary transformation parameters. However, if you are a developer that is looking to learn more about Cloudinary, this is also a great learning tool because it lets you instantly see how a visual change (resize, move, color, etc.) is reflected by a change in the Cloudinary transformation "chain".

But what is it for??? / Practical Applications:

Beyond serving as a learning tool for Cloudinary novices, calling it an editor might lead to some misgivings. This tool is really not meant to be a replacement for Photoshop, or even Paint.net for that matter. If you just need a one-off image, this editor is going to leave you disappointed. What this tool was meant for is when you need to generate 1,000+ unique images, all based on a set of dynamic inputs. This is usually needed when you are developing an application or something that wants to be able to "share" rich media updates to Facebook, Instagram, Twitter, Slack, etc. For example, let say you are making a social gaming application, and you want users to be able to share an achievement with their friends (something like "Reached Level 10!"). You could simply have them share a block of text, but that is not very attractive, and if you are looking to grow your fanbase, you might want them to be able to share a flashy image that might attract the attention of their friends. Again, you could just create a single image that everyone shares when they reach level 10, but again, not that impressive. What if you could have them share an image that features the achievement text, their profile picture, and other dynamic details? You can with Cloudinary, and you can use this tool to build the template! In fact, that super long URL in the above section was created with my tool, and generates this image based on user profile data:

Another example is generating images where the base template, e.g. the background of the image, does not change, but the overlaying text changes frequently and is unique per user. The example I provide in my demo video is for a theoretical SlackBot that messages Jira users with a daily task summary:

In this Jira example, I create the background template outside of my editor tool (but could have actually used it if I wanted to) and then used it as the "base layer" in my editor, while positioning text on top. Then, using JS/NodeJS/PHP/Google Sheets/Whatever I want to use, I can dynamically generate thousands of these status updates, completely unique per user with their stats.

Some other practical applications:

E-commerce sites
- Overlay product details in text over product image
- Embed templated images into your product feed for use with Google Shopping, Facebook Product Catalogs, etc.
Cloudinary learning tool
- Mess around with different combinations to see how different objects get converted into chained transformations, and subsequently into a URL string
Dynamic image generation for Google Sheets or any other spreadsheet program
- You can use this tool to create the template, then use spreadsheet functions like CONCAT() to create the URL dynamically given values in other columns
Dynamic image generation for sites that can't use server-side image manipulation - this way you can use Javascript, plus the export from this tool, to generate an <img> tag src completely 100% client side.
Meme generators / image macro sites
Creating image embeds for Slack Apps / Slack bots / FB Messenger Bots

Behind the Scenes / Tech Used:

The true originating force behind the creation of this tool was rather simple; I wanted to learn the basics of ReactJS and build something cool as my first project using it, then thought of how, a bit earlier, I had been wishing that something like this tool existed. Although I feel like the end product is pretty neat, I still have mixed feeling about React even after getting my feet wet. My next project will probably use VueJS, which although similar, has some key differences that I think my suit my development habits.

A library / framework that I used extensively on this tool, and can talk about much more positively than React, is Fabric.js. Don't let their somewhat cheesy homepage fool you; this is an amazing library for manipulating Canvas elements with Javascript. Like, seriously amazing. Every time I thought I was going to have to write my own code to handle something I assumed they wouldn't cover - bam - they would have support for it out of the box. For example, if you want to get the coordinates of an imaginary "bounding box" that contains a canvas object of interest, you can do so with just one line of code.

Obviously, kudos to Cloudinary as well, for providing a very impressive service, comprehensive documentation, and quickly responding to my GitHub issue :)

Brainstorming - Checking for Remote Power Failure Over the Internet

Joshua Tzucker — Fri, 08 Feb 2019 16:39:00 GMT

I live in Washington state, which, due to the amount of trees and their proximity to power lines, often means that I might lose power in my apartment, while just blocks away, everyone else is powered up. I also often visit my parents, and this has led to multiple instances where I am torn between spending the night at their house, where they still have power, or starting the drive back without knowing whether or not I will have hot water and my favorite electronics ready for use.

At these times, I have often wished for an easy way to check remotely, over the internet, whether or not my home has power. Building and setting up a home server is something on my to-do list (and has been on it for a while) and it would easily solve this issue, but with a potential snowstorm on the way, and a visit to my parents planned, I wanted to set something up in a jiffy. I was also thinking - given the plethora of internet-connected devices the average consumer, and I myself own, surely there must be some way to accomplish this already?

To be clear - the easiest solution is to simply check with the power utility company. However, since during the last outage their reporting system went down with the power (LOL) I was annoyed enough to look into my own method for checking.

Here are some of the ideas I explored. My quick solution for today is at the bottom (or click here to jump to it).

Idea: Leave a spare Laptop / PC on and running TeamViewer (remote control software for screen sharing). This would allow me to log in remotely and even check out the webcam remotely. If it fails to connect, I know my power has gone out.
- Failure: Most computers, including the ones I own, do not automatically start up after power has been lost and then restored. Some BIOS configurations will allow for enabling a setting which turns this feature on, but it is not often found on laptops or tablets, which are the only spare devices I own and would be comfortable leaving TeamViewer installed as a server on.
Idea: Use an old Android phone as a remote security camera. If I can't connect, I know the power has gone off. Or I could install some simple server software on it.
- There are a bunch of free applications that cover these needs
  - Remote security camera:
    - Alfred Home Security Camera: Play Store Link
  - Simple home server
    - LAMP like stack: "HTTP Server Powered by Apache" - Play Store Link
- ISSUE: Same as the Laptop / PC issue. Many Android phones do not automatically boot themselves back up after losing and regaining power. HOWEVER, this is still one of the best solutions, for multiple reasons:
  - If the screen is off, even old phones should hold a charge for a decent amount of time before depleting the battery. Hopefully your power comes back on before then.
  - There is a way to force your phone to auto boot back up after power has been restored. However, it requires root - instructions here

At this point, I started thinking "what devices do most people own, that are always plugged in, and already feature auto-restart / auto-boot on power failure and subsequent restoration?"...

Idea: Roku / Amazon Fire Stick / Smart TV
- This led me down a very fascinating rabbit hole of technology, which I could read about for days
  - Both Roku and Amazon FireTV/Fire Stick have network endpoints which you can make requests to and get data back. Both support something called DIAL, which is sort a protocol for 1st-screen (TV) discovery and control by 2nd-screen (smartphone, tablet) devices.
    - - Roku is much more developer friendly, and has published docs on using DIAL. In addition, they have their own API they have developed, which is surprisingly in-depth and powerful. A little scary actually, in terms of what can be controlled without authentication...
        
        For example, as long as I'm on the same network, I can actually "press" keys on the Roku remote just by issuing network requests. For example, making a POST request to this URL presses the "Home" key on my Roku!
        
        http://[Roku_Local_IP]:8060/keypress/Home
      - Amazon is much more secretive and harder to find information on. They also support DIAL, but it is less clear on what functionality is supported. However, by poking around, it looks like they support the general DIAL syntax. For example, if Netflix is installed, a GET request to this URL will return XML information about the application.
```
http://[FireStick_Local_IP:8009/apps/Netflix
```

Idea (and SOLUTION, at least for today): ROUTERS!!!
- If you are already using DD-WRT (open source firmware for routers, great stuff), you have tons of options at your fingertips, especially since it supports scripting.
- In general, this is a risky option, since the last thing you want to do with a router is purposefully allow outside access pointing in.
  - This is basically the whole point of firewalls, which are built into most routers.
    - If you try to make a GET/POST/etc HTTP request to your Public IP address, that is very unlikely to get routed anywhere.
- Works for me - Pinging the router! - For my quick, no fuss setup, I saw that my ISP's router had "Drop Incoming ICMP Echo Requested to Device WAN address" actually turned to off by default - 2 seconds later, I had fired up my terminal and a simple "ping [MY_PUBLIC_IP]" later... low and behold, the router responded and no packets dropped!
  - I should have checked this first, but had assumed that this would be locked down. To be fair, many routers might have this locked down, so you might have to explore the other options above if this is not working for you.
  - Since I have an Android phone, I can use Android's terminal to ping from my parents house and see if my router is up and running.
    - There are also dedicated apps for checking on ping status - such as this one.
  - Please note that an ICMP ECHO packet is different from HTTP - see wiki here - but the reason I mention this is because you can't use typical website uptime monitoring tools to check if your router is up or down. If you want to use an automated alert site, look for one that specifically supports "ping", such as UptimeRobot.

Strange Tip - Quickly Generating Typewriter Effect Videos (MP4, GIF, etc.)

Joshua Tzucker — Thu, 07 Feb 2019 12:50:57 GMT

File this under "shouldn't work, but does".

I was in need of a tool today to quickly create a "typewriter effect" video - you know, a video where text appears letter by letter, as if someone was typing it out while being recorded. I rarely ever make videos, so I don't have the best software (I'm using Magix Movie Edit) and it does not have a typewriter effect animation built in.

There are solutions out there online for generating GIFs, but they the quality seemed off and there was not much configuration available in terms of font and sizing. Commercial solutions are in the hundreds of dollars, which is way out of budget for a one-off project like the one I was working on.

After a little searching, I stumbled across an interesting response to a related question on Quora that suggested using Microsoft PowerPoint.

Hmm. Could that work?... IT DOES! This feels like such a "dirty" solution, but this was seriously so easy - I just created a slide in Microsoft Powerpoint with dimensions of 1920x1080 to match my video, added the text, set the animation effect to "Appear" and "Animate Text" to "By letter", and then ran the animation while running screen capture software. Here is a sample output (not bad, right?!!):

If you are looking for a step by step process for PowerPoint, here is a video off YouTube that covers how to add it:

Google Sheets - Quick Start with Base64 - With and Without Scripting

Joshua Tzucker — Fri, 01 Feb 2019 06:53:18 GMT

Sample Spreadsheet showing both solutions: Link

I'm used to Google Sheets having built-in formulae for practically everything, so I was surprised when I needed to Base64 encode a string today and could not find a function to do so. The easiest solution is to use a "Custom Function", which allows you write your own named function in JavaScript, and then call that function from anywhere in your Google Sheets file, just as if it was a built-in function. That is Solution A, below. Skip to Solution B if you want to see how I created a custom formula that only uses native Google Sheets functions!

Solution A - Using a Google Sheets Script "Custom Function":

This is super easy. Just add the following code to the scripts file attached to your Google Sheet, by going to "Menu > Tools > Script Editor"


/**
 * Base64 Encode Input
 * @param {any | Array<any[]>} input - Input cell, or range of cells
 * @param {boolean} [OPT_webSafe=true] - If should use websafe variant of base64
 * @param {boolean} [OPT_plainText=false] - If should treat input as plaintext instead of UTF-8
 */
function base64Encode(input, OPT_webSafe, OPT_plainText) {
  if (!input) return input;
  const charSet = OPT_plainText ? Utilities.Charset.US_ASCII : Utilities.Charset.UTF_8;
  const useWebSafe = OPT_webSafe !== false;
  const encoder = useWebSafe ? Utilities.base64EncodeWebSafe : Utilities.base64Encode;
  if (Array.isArray(input)) {
    return input.map(t => base64Encode(t, OPT_webSafe, OPT_plainText));
  }
return encoder(input, charSet);
}
/**

Base64 Decode Input
@param {any | Array<any[]>} input - Input cell, or range of cells
@param {boolean} [OPT_webSafe=true] - If should use websafe variant of base64
@param {boolean} [OPT_plainText=false] - If should treat input as plaintext instead of UTF-8
*/
function base64Decode(input, OPT_webSafe, OPT_plainText) {
if (!input) return input;
const charSet = OPT_plainText ? Utilities.Charset.US_ASCII : Utilities.Charset.UTF_8;
const useWebSafe = OPT_webSafe !== false;
const decoder = useWebSafe ? Utilities.base64DecodeWebSafe : Utilities.base64Decode;
if (Array.isArray(input)) {
return input.map(t => base64Decode(t, OPT_webSafe, OPT_plainText));
}

return Utilities.newBlob(decoder(input, charSet)).getDataAsString();
}

Once you have pasted and saved that code into the script editor, switch back to your spreadsheet, and reload the page. You should now be able to use the custom function very easily. 99% of the time, you should be able to use it with default settings, like so:

You can even use the formula with an array of cells as the input,and it will automatically fill down!

Breaking down the custom function:

This is basically the same code as above, but broken down into separate functions, so you can see how it works.


function base64Encode(input) {
  return Utilities.base64Encode(input,Utilities.Charset.UTF_8);
}
function base64EncodeWebSafe(input) {
  return Utilities.base64EncodeWebSafe(input,Utilities.Charset.UTF_8);
}
function base64Decode(input) {
  return Utilities.newBlob(Utilities.base64Decode(input,Utilities.Charset.UTF_8)).getDataAsString();
}
function base64DecodeWebSafe(input) {
  return Utilities.newBlob(Utilities.base64DecodeWebSafe(input,Utilities.Charset.UTF_8)).getDataAsString();
}

The main difference between WebSafe (aka "base64url") and non-WebSafe is that "/" is replaced with "_" for WebSafe, since slashes are reserved in URLs for path separators. For more details see RFC-4648.

Notice that the "could not decode string" error in the screenshot above is because you should not use base64DecodeWebSafe to decode something that was originally encoded with the WebSafe variant of Base64. In this case, it doesn't like the "_" character, which is specific to the WebSafe variant. In general, you are probably safer just using the WebSafe version most of the time, if you want to avoid these kinds of errors.

EDIT: Thank you to Maxime for pointing out an error in my original code; I did not account for the fact that Utilities.base64Decode & Utilities.base64DecodeWebSafe both return byte-arrays, not strings. For some reason I only tested my encode functions and did not notice this. This post has been updated, and for further reference, see Maxime's comments at the bottom, and this Apps Script doc page.

EDIT: I added the UTF-8 argument to all the functions, in case anyone is using this with non ASCII text (for example, a foreign language that uses accented or non Latin characters). If you are only using ASCII, you could leave off this argument, and Sheets will default. However, as ASCII is part of UTF-8, it doesn't hurt to leave it.

Solution B: "Pure Formula" function - No Google Apps Scripting required!

Custom functions through Google Sheets App Scripting is fast and simple. However, this is not always the most "performant" solution, and I felt like taking on a challenge today, so I actually came up with a formula that uses only native Google Sheets functions to Base64 encode a string - no scripting! This is also helpful if you are not allowed to use Scripts within your Google Sheets file due to restrictions placed by your employer, or you are just looking for the quickest copy and paste solution.

Here is the full (insanely long and complex) Google Sheets formula that converts text to Base64, nothing else required. Just replace "A2" with the cell you want to convert the text from:


=CONCATENATE(JOIN("",ARRAYFORMULA(SWITCH(BIN2DEC(TEXT(SPLIT(REGEXREPLACE(CONCATENATE(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))),REPT("0",(FLOOR((LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))))+(6-1))/6)*6)-LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8)))))),"(.{6})","$1/"),"/"),"000000")),0,"A",1,"B",2,"C",3,"D",4,"E",5,"F",6,"G",7,"H",8,"I",9,"J",10,"K",11,"L",12,"M",13,"N",14,"O",15,"P",16,"Q",17,"R",18,"S",19,"T",20,"U",21,"V",22,"W",23,"X",24,"Y",25,"Z",26,"a",27,"b",28,"c",29,"d",30,"e",31,"f",32,"g",33,"h",34,"i",35,"j",36,"k",37,"l",38,"m",39,"n",40,"o",41,"p",42,"q",43,"r",44,"s",45,"t",46,"u",47,"v",48,"w",49,"x",50,"y",51,"z",52,"0",53,"1",54,"2",55,"3",56,"4",57,"5",58,"6",59,"7",60,"8",61,"9",62,"-",63,"_",64,"="))),REPT("=",(FLOOR((LEN(JOIN("",ARRAYFORMULA(SWITCH(BIN2DEC(TEXT(SPLIT(REGEXREPLACE(CONCATENATE(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))),REPT("0",(FLOOR((LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))))+(6-1))/6)*6)-LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8)))))),"(.{6})","$1/"),"/"),"000000")),0,"A",1,"B",2,"C",3,"D",4,"E",5,"F",6,"G",7,"H",8,"I",9,"J",10,"K",11,"L",12,"M",13,"N",14,"O",15,"P",16,"Q",17,"R",18,"S",19,"T",20,"U",21,"V",22,"W",23,"X",24,"Y",25,"Z",26,"a",27,"b",28,"c",29,"d",30,"e",31,"f",32,"g",33,"h",34,"i",35,"j",36,"k",37,"l",38,"m",39,"n",40,"o",41,"p",42,"q",43,"r",44,"s",45,"t",46,"u",47,"v",48,"w",49,"x",50,"y",51,"z",52,"0",53,"1",54,"2",55,"3",56,"4",57,"5",58,"6",59,"7",60,"8",61,"9",62,"-",63,"_",64,"="))))+(4-1))/4)*4)-LEN(JOIN("",ARRAYFORMULA(SWITCH(BIN2DEC(TEXT(SPLIT(REGEXREPLACE(CONCATENATE(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))),REPT("0",(FLOOR((LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))))+(6-1))/6)*6)-LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8)))))),"(.{6})","$1/"),"/"),"000000")),0,"A",1,"B",2,"C",3,"D",4,"E",5,"F",6,"G",7,"H",8,"I",9,"J",10,"K",11,"L",12,"M",13,"N",14,"O",15,"P",16,"Q",17,"R",18,"S",19,"T",20,"U",21,"V",22,"W",23,"X",24,"Y",25,"Z",26,"a",27,"b",28,"c",29,"d",30,"e",31,"f",32,"g",33,"h",34,"i",35,"j",36,"k",37,"l",38,"m",39,"n",40,"o",41,"p",42,"q",43,"r",44,"s",45,"t",46,"u",47,"v",48,"w",49,"x",50,"y",51,"z",52,"0",53,"1",54,"2",55,"3",56,"4",57,"5",58,"6",59,"7",60,"8",61,"9",62,"-",63,"_",64,"="))))))

Here is a slightly shorter version of that formula, but requires having a Base64 lookup sheet as part of your Google Sheet file. See sample sheet here.


=CONCATENATE(JOIN("",ARRAYFORMULA(VLOOKUP(BIN2DEC(TEXT(SPLIT(REGEXREPLACE(CONCATENATE(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))),REPT("0",(FLOOR((LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))))+(6-1))/6)*6)-LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8)))))),"(.{6})","$1/"),"/"),"000000")),Base64Lookup!$A$1:$B$65,2,FALSE))),REPT("=",(FLOOR((LEN(JOIN("",ARRAYFORMULA(VLOOKUP(BIN2DEC(TEXT(SPLIT(REGEXREPLACE(CONCATENATE(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))),REPT("0",(FLOOR((LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))))+(6-1))/6)*6)-LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8)))))),"(.{6})","$1/"),"/"),"000000")),Base64Lookup!$A$1:$B$65,2,FALSE))))+(4-1))/4)*4)-LEN(JOIN("",ARRAYFORMULA(VLOOKUP(BIN2DEC(TEXT(SPLIT(REGEXREPLACE(CONCATENATE(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))),REPT("0",(FLOOR((LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8))))+(6-1))/6)*6)-LEN(JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))),2,8)))))),"(.{6})","$1/"),"/"),"000000")),Base64Lookup!$A$1:$B$65,2,FALSE))))))

Additional background on Solution B and how I came up with the formula:

I started off trying to understand how strings are converted to Base64 representations, period. I found this page, which although written about NodeJS, actually does a good job of covering the basics of how ASCII is converted to Base64. I then went, step by step, through the process of manually converting a string inside Google Sheets, first into Unicode, then into binary, then into 6 bit chunks, then into decimal, and finally, into base64 by using a decimal-to-base64 lookup table. Some of these steps had built in Google Sheets functions to accomplish them, such as getting the Unicode value of a character (that is simply "UNICODE(A2)"), but some other steps required crafting my own complicated formula. Some examples:

Converting ASCII (Text) to Binary (base2)


=JOIN("",ARRAYFORMULA(BASE(UNICODE(SPLIT(REGEXREPLACE(A2&"","(.{1})","~$1"),"~")),2,8)))

Let's break this down step by step.

SPLIT(REGEXREPLACE(A2&"","(.{1})","~$1"),"~")

This splits apart a string into individual characters, by inserting "~" before each character, then splitting by "~". This returns an array, which is why ARRAYFORMULA is required.

BASE(UNICODE(...),2,8)

This converts each character into its unicode value, then converts it into base2, which is binary, and pads it to 8 bits in length (a byte)

=JOIN("",ARRAYFORMULA(...))

Finally, this joins all the binary bytes into one cell. If you want to, you could use JOIN(" ",...) to visualize the binary value as byte-separated.

RIGHT-Padding a string to a desired length (adding trailing characters)

There is tons of information out there on keeping leading characters in Google Sheets, or left-padding, but almost nothing on right-padding, or adding trailing characters to get to a desired length. Furthermore, I wanted something even more specific; to right-pad a string until the entire string was evenly divisible by a specific integer. This is important because Base64 uses padding on both the input and the output - it breaks input into chunks of 6 bits before converting, and output needs to be divisible by 4. I came up with two separate formula that both produce the same result, but one uses an IF statement, which can get messy in Google Sheets if you have a lot of nested IFs already. Replace A2 with the cell you want to pad, replace "MIN_DIVISIBLE" with the integer you want A2 to become divisible by, and replace the equals in "=" with the character you want to pad to the right with.

Without IF() statement:

=CONCATENATE(A2,REPT("=",(FLOOR((LEN(A2)+(MIN_DIVISIBLE-1))/MIN_DIVISIBLE)*MIN_DIVISIBLE)-LEN(A2)))

With IF() statement:

=IF(MOD(LEN(A2),MIN_DIVISIBLE),CONCATENATE(A2,REPT("=",MIN_DIVISIBLE-MOD(LEN(A2),MIN_DIVISIBLE))),A2)

Here is an example showing padding binary to 6 bits, with trailing zeros.

=CONCATENATE(A2,REPT("0",(FLOOR((LEN(A2)+(6-1))/6)*6)-LEN(A2)))

Which results in "11" becoming "110000":

Non-ASCII characters - beware binary padding!

It was brought to my attention that my original "pure" formula was not handling foreign characters correctly. This wasn't too surprising, but after I started digging into my own work, I was wondering why it shouldn't work as-is; after all, the Google Sheets Unicode() function should be able to handle all non-ASCII chars, and the lookup tables should stay the same (standard ASCII) regardless of input charset. After breaking down my own function into pieces, I found the culprit - the BASE() function.

I was using BASE() to turn each Unicode (UTF-8 presumably) code point into binary, then pad to 8 digits long. Can you guess the problem? I'll give you a hint. The standard+extended ASCII table ranges from decimal 0 to 255. If you want to use the full UTF-8 range, and map other characters, you need to go above. For example, this character - "ă", or "latin small letter a with breve" is mapped to Unicode 259. Found the issue yet? Once you go past 255, converting from decimal (base10) to binary (base2) gives you an output that exceeds 8 bits (a byte). 259 in binary is "100000011", which is 9 bits long. I was telling BASE() to pad all outputs to 8 bits, but when the output was already longer, instead of padding to a multiple of it, it just ignored it and left it at 9 bits long. This was breaking the rest of my formula, which relied on byte chunking (groups of 8 bits)!

Solution: There could be a solution to this, although it would be rather complicated since multiple spots in the formula rely on chunking and an expected 8 bit starting chunk.

More resources:

I also frequently used base64 online converters, to ensure my formula work was correct, such as this instant online converter and this one.

How about you? Do you have any tips or feedback related to Base64 conversion inside Google Sheets? I didn't cover Excel much, since I rarely ever use it, and it would take a while to come up with an Excel equivalent of my custom formula, since Excel doesn't have functions such as REGEXREPLACE(). However, if you are interested in an Excel solution, it is probably possible to achieve using VBA - I might start with something like this as a starting point.

Updates:

1/27/2021: “Pure formula” fixes

1/27/2021: Summary

Fixed issue with the “pure formula” approach. It wasn’t working with input strings that contained line breaks (e.g. \n), as it would trim off the first character following the line break character. This was traced back to the RegEx pattern that splits the input into individual characters - I had forgotten the flag to let the dot pattern (.) match all characters including line breaks!. I also took this opportunity to fix two other bugs - the handling of ' (single quotes) and ~ (used in my formula as a delimiter)

1/27/2021: Detailed Notes

The fix to match . not matching newlines was simply to add the appropriate dotall flag for RE2 (the RegEx flavor that Google Sheets uses). In effect:
- Replaced: REGEXREPLACE(A2&"","(.{1})","$1~")
- With: REGEXREPLACE(A2&"","(?s)(.{1})","$1~")
In fixing the above, I noticed another issue: input strings with ' (single quotations) were not working, period (throwing fatal error Function UNICODE parameter 1 value should be non-empty.)
- The fix for this was to start double escaping single quotes
- Replace: REGEXREPLACE(A2&"","(?s)(.{1})","$1~")
- With: REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1~"),"'","''")
  - NOTE: Order of operations is important here! Prefixing each character with the delimiter needs to come first, or else the double escaped string ('') will be treated prematurely as two characters, instead of an escaped single char.
Another issue, which had actually been bothering me from the start, is that the pure formula cannot handle input strings with ~, since internally that is used within the formula as a delimiter.
- I finally found a solution; I was able to find a non-printable ASCII character code to use a delimiter, which should (probably) never show up in input strings!
- Replace: SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1~"),"'","''"),"~")
- With: SPLIT(REGEXREPLACE(REGEXREPLACE(A2&"","(?s)(.{1})","$1"&CHAR(127)),"'","''"),CHAR(127))

Cloudinary - Adding Solid Shape Overlays (Squares, Circles, Etc.)

Joshua Tzucker — Wed, 23 Jan 2019 07:08:44 GMT

Disclaimer: For anyone wondering, this post was not sponsored by Cloudinary or is in any way officially affiliated, endorsed, or supported by them. I am just a big fan of them, and enjoy seeing how I can use all their features in unusual combinations.

Background Info:

This task, overlaying solid shapes on Cloudinary, is simple (somewhat), yet might be missed by many as even being possible. It uses a concept that I am fascinated by, that what is displayed on our computer screens is not a single image, but the result of many transformations and calculations. For example, it is impossible (in practical methods) to display a true curved line (or anything other than a perfectly horizontal or vertical line) on a pixel based display; pixels are squares, so the best you can do is push more and more pixels together until, when viewed from a distance, squares that are placed in a non-linear (e.g. polynomial) arrangement start to appear curved. This is why curves in really old video games, that are low resolution, look blocky and jagged.

For those interested, the process of transforming non-pixel-based shapes, such as a vector representation of a circle, into a pixel-based form (which is what you are looking at right now), is called "rasterisation".

[caption id=“attachment_185” align=“aligncenter” width=“300”] Wojciech Muła [GFDL (http://www.gnu.org/copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/) or CC BY-SA 2.5 (https://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons[/caption]

Anyway, I digress. The reason why I brought up transformations is because they are part of the core of Cloudinary, and what allows me to overlay custom shapes on top of images. The fact that Cloudinary lets users "chain" transformations together in an unlimited fashion, means that almost any shape can be dynamically rendered with a Cloudinary URL, given the right combination of pixel transformations.

Note:

Throughout this article, ACCOUNT_NAME stands in for the name of your Cloudinary cloud account, which should be a short string of characters and numbers.

Adding Squares:

Here is a practical example: A simple blue square added to the default Cloudinary sample image. Here is the sample image without any transformations, and the URL used to fetch it.

https://res.cloudinary.com/ACCOUNT_NAME/image/upload/sample

And here is the same image, with a blue square rendered over it:

https://res.cloudinary.com/ACCOUNT_NAME/image/upload/l_pixel,w_200,h_200/co_rgb:4a90e2,e_colorize,fl_layer_apply,g_north_west,x_270,y_100/sample

The trick here is that "l_pixel" maps to an image I've uploaded to my account, with the public ID of "pixel", and is a 1x1 solid colored pixel saved as an image file. The transformations applied above blow it up to 200 x 200 pixels, position it at x=270 and y=100, and "colorize" it with an RGB / Hex string corresponding to blue. You might think this is very basic, but the fact that you can chain these together means you can do crazy things like this:

https://res.cloudinary.com/ACCOUNT_NAME/image/upload/c_fill,w_500,h_500,o_40/h_302,l_pixel,w_17,x_146,y_48/co_rgb:4a90e2,e_colorize,fl_layer_apply,g_north_west,x_146,y_48/h_21,l_pixel,w_100,x_63,y_332/co_rgb:4a90e2,e_colorize,fl_layer_apply,g_north_west,x_63,y_332/h_97,l_pixel,w_20,x_63,y_254/co_rgb:4a90e2,e_colorize,fl_layer_apply,g_north_west,x_63,y_254/h_20,l_pixel,w_153,x_70,y_47/co_rgb:4a90e2,e_colorize,fl_layer_apply,g_north_west,x_70,y_47/h_100,l_pixel,w_100,x_166,y_70/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_166,y_70/h_15,l_pixel,w_85,x_246,y_50/co_rgb:6283aa,e_colorize,fl_layer_apply,g_north_west,x_246,y_50/h_83,l_pixel,w_15,x_246,y_51/co_rgb:6283aa,e_colorize,fl_layer_apply,g_north_west,x_246,y_51/h_20,l_pixel,w_89,x_246,y_117/co_rgb:6283aa,e_colorize,fl_layer_apply,g_north_west,x_246,y_117/h_76,l_pixel,w_17,x_321,y_117/co_rgb:6283aa,e_colorize,fl_layer_apply,g_north_west,x_321,y_117/h_22,l_pixel,w_100,x_240,y_173/co_rgb:6283aa,e_colorize,fl_layer_apply,g_north_west,x_240,y_173/h_138,l_pixel,w_19,x_327,y_173/co_rgb:27a456,e_colorize,fl_layer_apply,g_north_west,x_327,y_173/h_21,l_pixel,w_79,x_331,y_235/co_rgb:27a456,e_colorize,fl_layer_apply,g_north_west,x_331,y_235/h_100,l_pixel,w_23,x_386,y_238/co_rgb:27a456,e_colorize,fl_layer_apply,g_north_west,x_386,y_238/sample

Adding Circles:

Again, this is a bit of a "cheat"; there is not built-in way to overlay a solid circle with Cloudinary. However... it is possible to create a circle by first creating a square, and then rounding the corners with a radius. Check it out!

https://res.cloudinary.com/ACCOUNT_NAME/image/upload/c_fill,h_500,w_500/g_north_west,l_pixel,r_150,w_300,h_300/b_rgb:4a90e2,co_rgb:4a90e2,e_colorize,fl_layer_apply,g_north_west,x_90,y_90/sample

Let's Kick It Up a Notch:

OK, so far this might not seem all that impressive. What is the point? Well, think about how this could be applied to situations where you need to dynamically generate an image overlay based on a set of criteria. For example, if you make an online game and need to generate an achievement badge that can be shared to Facebook as an image, but the colors are based on the user's profile settings. You can generate that image with Cloudinary, just by passing transformations in the image path, which means the image URL can even be generated completely client-side. Check out this doozy of a URL:

https://res.cloudinary.com/ACCOUNT_NAME/image/upload/c_fill,h_500,o_40,w_500/h_224,l_pixel,w_224/co_rgb:070707,e_colorize,fl_layer_apply,g_north_west,x_28,y_75/h_395,l_pixel,w_454/co_rgb:2744a6,e_colorize,fl_layer_apply,g_north_west,x_23,y_35/h_210,l_pixel,w_210/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_34,y_82/c_scale,h_188,l_profilepicture,w_215/fl_layer_apply,g_north_west,x_34,y_107/h_209,l_pixel,w_208/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_254,y_82/l_text:Roboto_37_normal:Stats:/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_316,y_93/l_text:Roboto_19_normal:Flowers%20Pollinated%20:%2020/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_266,y_141/l_text:Roboto_19_normal:People%20Chased%20:%2025/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_266,y_167/l_text:Roboto_55_normal:Buzzy%20McBuzzer/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_52,y_314/l_text:Roboto_31_normal:Rank%20:%2010/co_rgb:ffffff,e_colorize,fl_layer_apply,g_north_west,x_170,y_380/h_119,l_pixel,w_119/co_rgb:c8c61f,e_colorize,fl_layer_apply,g_north_west,r_59,x_295,y_194/l_text:Roboto_42_normal:%3C10%3E/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_308,y_237/sample

Crazy, right? However, once you come up with a set of transformations you want to use together, you could either save it as a saved transformation, or dynamically generate the URL with just a little bit of code, for example, like this:


function getGamerBadge(gamerProfile){
	return 'https://res.cloudinary.com/ACCOUNT_NAME/image/upload/c_crop,h_500,o_40,w_500/h_224,l_pixel,w_224/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_28,y_75/h_395,l_pixel,w_454/co_rgb:' + gamerProfile.colors.dark + ',e_colorize,fl_layer_apply,g_north_west,x_23,y_35/h_210,l_pixel,w_210/co_rgb:' + gamerProfile.colors.light + ',e_colorize,fl_layer_apply,g_north_west,x_34,y_82/c_scale,h_188,l_' + gamerProfile.profileImageId + ',w_215/fl_layer_apply,g_north_west,x_34,y_107/h_209,l_pixel,w_208/co_rgb:' + gamerProfile.colors.light + ',e_colorize,fl_layer_apply,g_north_west,x_254,y_82/l_text:Roboto_37_normal:Stats:/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_316,y_93/l_text:Roboto_19_normal:Flowers%20Pollinated%20:%20' + gamerProfile.stats.pollinated + '/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_266,y_141/l_text:Roboto_19_normal:People%20Chased%20:%20' + gamerProfile.stats.chased + '/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_266,y_167/l_text:Roboto_55_normal:' + encodeURIComponent(gamerProfile.name) + '/co_rgb:' + gamerProfile.colors.light + ',e_colorize,fl_layer_apply,g_north_west,x_52,y_314/l_text:Roboto_31_normal:Rank%20:%20' + gamerProfile.rank + '/co_rgb:' + gamerProfile.colors.light + ',e_colorize,fl_layer_apply,g_north_west,x_170,y_380/h_119,l_pixel,w_119/co_rgb:' + gamerProfile.colors.badge +  ',e_colorize,fl_layer_apply,g_north_west,r_59,x_295,y_194/l_text:Roboto_42_normal:%3C' + gamerProfile.rank + '%3E/co_rgb:000000,e_colorize,fl_layer_apply,g_north_west,x_308,y_237/sample';
}
var buzzyGamerProfile = {
	name : 'Buzzy McBuzzer',
	rank : 10,
	stats : {
		pollinated : 20,
		chased : 25
	},
	profileImageId : 'profilepicture',
	colors : {
		dark : '2744A6',
		light : 'ffffff',
		badge : 'c8c61f'
	}
}
var buzzyGamerBadgeImageSrc = getGamerBadge(buzzyGamerProfile);

More information:

I am currently working on a tool related to all of these features, but in the mean time, here are some additional resources to check out if you are interested in the power-user features of Cloudinary transformations:

Cloudinary Transformation Reference / Cheat Sheet
Transformation "Cookbook" - shows examples of different effects and transformation chains, with URL and code examples.
Cloudinary JavaScript library source code - useful for looking at class definitions of different Cloudinary layers and how they work together. As mentioned in another post, this is where I tracked down a bug I was running into related to remote fetch overlay transformations.

No Fuss Prism Toolbar (J-Prism-Toolbar)

Joshua Tzucker — Sat, 12 Jan 2019 19:04:11 GMT

Quick JS plugin for PrismJS (code embed syntax highlighter) that I put together, and use throughout this site . Typical "windows" style toolbar, with minimize/expand, fullscreen, and copy-to-clipboard buttons.

CSS to JS Converter / Escaper

Joshua Tzucker — Wed, 09 Jan 2019 18:23:22 GMT

In general, you probably want to avoid putting CSS directly into your Javascript. However, there are times when it is unavoidable, or simply not worth the time. I made this tool for myself, for those times when I simply need to paste in a block of CSS into JS code as a string literal, while preserving (or removing) formatting.

Just paste your CSS into the left panel, and then copy the generated JS from the right side.

See the Pen <a href='https://codepen.io/joshuatz/pen/yGqWyy/'>CSS to JS Converter Tool</a> by Joshua T (<a href='https://codepen.io/joshuatz'>@joshuatz</a>) on <a href='https://codepen.io'>CodePen</a>.

Frustrating Day with Postman - Problems Log - 1/4/2019

Joshua Tzucker — Fri, 04 Jan 2019 16:05:25 GMT

Postman is my go-to tool for testing a new API, replaying HTTP requests, and general mucking about with network requests. They call their tool a "API Development Environment", and if you deal at all with APIs and/or endpoints like webhooks, you should check them out. In general I am a fan, and have been for some time now, but today I have been running into a seemingly endless supply of roadblocks trying to get something simple working. I thought I would jot down what I ran into, some of which is my own fault, and some is partly due to how Postman is designed.

In no particular order:

Postman refuses to load on Windows / crashes / GUI screen flashes white and black / glitches out

The problem:

This problem actually started a while ago for me (at least how I remember it), but seems to have gotten progressively worse. At first Postman would just take an incredibly long time to launch, and would flash a bunch and hang for a while until it finally loaded. However, today it started its normal glitching out when trying to launch, but then seemed to never load and basically crashed.

The Fix:

I assumed this would be complicated to fix, but actually it is very simple. This is a known issue with how Postman uses certain GPUs, and you just need to set a global system environmental variable that tells Postman to avoid using the GPU. See screenshot below (click for larger version):

Alternatively, if you are feeling lazy and want an even faster route, launch a Windows Command Line window with Administrator rights and run this command:

setx -m POSTMAN_DISABLE_GPU "true"

A POST request was working with the body set as "raw", but not with the body set to "x-www-form-urlencoded" or anything else

The problem:

Pretty much exactly what the heading says. I was trying to get a simple POST request to work, which I had imported from Chrome. It would work if I used "raw" as the body content, but whenever I tried to send the exact same body content but as form-data or x-www-form-urlencoded instead, I would get unexpected results.

More details:

This was actually my bad (well mostly). I had created my request in Postman by using Chrome's "copy as cURL (bash)" and then using the import feature in Postman. However, by default this only copies the body as raw, and Postman doesn't seem to be able to auto-parse the raw into form-data, so I had manually copied and pasted from the "Form Data" panel in Chrome network requests to the "Bulk Edit" field editor in Postman -> x-www-form-urlencoded. What I missed is that Chrome includes a space between key and value pairs in the dev panel, which got pasted into Postman. Since Postman doesn't use trim() in parsing the Bulk Edit input, the value got passes as a literal space before each value. So what should have been sent as something like:

myProp=myVal

... was getting sent as:

myProp= myVal

The fix:

The easy fix here was simply to find and replace ": " with ":" before pasting from Chrome into Bulk Edit. However, this could maybe be fixed by using Postman Pre-Request Scripts.

Postman console not matching actual requests

The problem:

This was a strange one. I was trying to figure out why a request wasn't working, and I wanted to see the actual request once it left Postman, so I opened up the Postman Console, which conveniently logs requests and lets you inspect them after you hit "Send".

I was dumbstruck when I noticed that, according to the console, Postman was sending key-value pairs in my request body that I had very specifically DISABLED / unchecked in the Postman GUI. What the heck? I tried a few different things (making sure I didn't have duplicate key/value pairs, trying different variations), but I kept seeing unchecked and disabled value pairs showing up in the Postman console.

I finally got frustrated and wanted to verify that the console itself was making sense. So I changed the endpoint I was working on to a RequestBin endpoint (a service that gives you a URL you can make requests to and it logs them online for you to inspect). I then made the same request and compared the Postman console with the actual request that came through to RequestBin. They didn't match - Postman's Console did not match the ACTUAL request sent and received!!! RequestBin showed what I wanted, that Postman indeed was not sending values I had unchecked, but that still didn't explain why the console was incorrectly showing those being sent. ARGH!

The fix:

As much as my gut reaction is to blame myself (PEBKAC, blah blah), this appears to either be an actual legitimate bug, or poorly documented functionality that I would argue goes against user expectation. As of right now, I see multiple bug reports (here, here, and here) on Github that are about this exact issue, which reassures me that I'm not crazy (at least in this instance).

The quick and dirty fix, if you simply want to see EXACTLY what Postman actually sent as a request, is to add a "Pre-Request Script" and/or "Test Script" that includes the following:

console.log(request);

Whereas the default request logging that Postman does seems to suffer from this bug, the "request" variable actually does represent the real request that gets sent, and does not show things like unchecked variables. I would recommend putting the console.log(request) in the "Test" script panel, since that one runs after the request executes, so it should be the most accurate.

Revisiting Chrome Extension Development - Build System Tips

Joshua Tzucker — Thu, 03 Jan 2019 07:28:48 GMT

Yesterday, I realized something not very good. There was a chrome extension I developed years back (this one) that depended on a service outside my control (TimeClockWizard). When that service changed a bunch of its code a year or so ago, it broke all functionality in my extension. When I found out, I posted a notice on my site that the extension was "deprecated", but forgot to pull the actual extension from the Chrome Webstore. Or maybe I had just assumed that people would stop trying to use and download it when it had stopped working. However, when I was browsing my Google Analytics account yesterday, I saw some pings from the extension and realized that people were still actively trying to install and/or use it. It made me feel bad, so I thought I would give fixing the extension a crack. It actually went really well, and I'm probably going to publish a fix to the webstore soon that should fix everything in the extension for everyone.

However, what I want to talk about in this post is how certain decisions I made earlier, in terms of tooling, made coming back to work on the extension, even years later, a lot easier. I also made some further improvements as I worked on the repairs yesterday and today. Here is some of what was the most helpful to have setup in my dev pipeline:

Auto ZIP Archive Creator for Webstore Upload / Releases

If you want to upload your Chrome extension to the Chrome Webstore, you need to zip it up in order to upload it. Plus, ZIP archives are always decent to have if you want to use them as a way to distribute "releases". Since the Chrome Webstore requires you to provide a version number in your manifest that gets bumped up on new releases, I setup a script that bundles up all my required assets and archives them into a ZIP that has a filename corresponding to current version # in my manifest.json file. It will also overwrite the zip file every time it runs, so you can keep running it until you have released a new version, than bump your version number up, so the next time it runs it will create a new archive file.

I have this run after the code has been minified, and just the files I want to upload have been moved to "/dist" (see section on package.json further below). This is super simple - basically just a slight modification of the demo code from the npm "archiver" package:


/**
 * @file Archive_Creator.js
 */
var fs = require("fs");
var archiver = require("archiver");
// Get version info
var manifestJSON = require(”../manifest.json”);
var versionString = manifestJSON.version.toString();
var output = fs.createWriteStream(”./builds/build_“+versionString+“.zip”);
var archive = archiver(“zip”, {
zlib : {level : 6} // compression level
});
// listen for all archive data to be written
output.on(“close”, function() {
console.log(archive.pointer() + ” total bytes”);
console.log(“archiver has been finalized and the output file descriptor has closed.”);
});
// good practice to catch this error explicitly
archive.on(“error”, function(err) {
throw err;
});
// pipe archive data to the file
archive.pipe(output);
// append files from a directory
archive.directory(”./dist/”,"");
// finalize the archive (ie we are done appending files but streams have to finish yet)
archive.finalize();

Prepping files for zipping / minifying / copying

It is a good idea when creating a zip to upload to the Chrome webstore to only upload A) what is absolutely necessary to run your extension and B) minified versions of your files. This is as a courtesy to users to keep file sizes down, and in case Google ever starts getting restrictive about the maximum size of extension ZIPs.

I have my system setup like this: minify files ("Minifier.js")-> copy minified files and assets files (PNG, etc.) to /dist ("Prepare_Dist.js") -> take entire /dist folder and archive it, sending archive to /builds/build_[version].zip ("Archive_Creator.js"). You can see this pipeline reflected in my package.json script section below:


"scripts": {
	"prepare-dist": "node ./npm/Prepare_Dist.js",
	"minify": "node ./npm/Minifier.js",
	"zip": "node ./npm/Archive_Creator.js",
	"stage": "npm run minify && npm run prepare-dist",
	"build": "npm run minify && npm run prepare-dist && npm run zip"
}

Good tip on debugging popup window

This seems like a no-brainer kind of thing, but while working on repairs, I realized that I needed to be able to debug some of my popup's JavaScript, but the code I needed to debug was running before I could even open Chrome Dev tools after opening the popup. I could use SetTimeout as a quick hack, but there is a much better way - simply right click on the button for your Chrome Extension in the toolbar and click "Inspect Popup"! Thank you Google, and thank you StackOverflow.

localStorage is pretty cool for persisting data

If you need to persist user data across sessions, the built in browser localStorage API is pretty nifty and convenient. It is incredibly simple to use (values are stored as key-pairs) and access is automatically handled by the browser and scoped to the origin of the initiating code (i.e. my extension should not be able to access the localStorage set by another extension or website).

Advice I wish I had followed from the beginning - multiple files are OK

Since so much of a Chrome extension is centered around distribution and bundling of assets, I was tempted from the beginning to keep my source code to a minimal number of files. For example, if you want to give certain script files different privileges (such as running in the background) you have to specify them by filename in your manifest.json, which could be messy if you have dozens of files. This led to some annoyances where it would have been a lot cleaner and easier to look through my own code if I had split certain functional components into their own files.

What is especially silly about this is that I could have just as easily added a step to my build process that concatenates multiple files into one - so the final zip archive file would have stayed around the same size, and my manifest.json could also have stayed (relatively) the same.

Lesson Learned - Cloudinary Transformations - Remote Fetch as Overlay

Joshua Tzucker — Fri, 28 Dec 2018 13:19:35 GMT

I'm working on a random project at the moment - I won't go into details at the moment - but the basic building blocks are ReactJS + Cloudinary + JQuery + some other stuff. Everything was going smoothly until I got to implementing a feature that is integral to the project - "chaining" the output of a Cloudinary remote fetch as a transformation overlay on top of another image. In particular, I wanted to use the core Javascript library functions to do this, as opposed to their React SDK (I'm not even sure if it would be possible with just the React SDK) If you have come to this post through a Google search, chances are that you are looking for this kind of info, and already know what all of this means, but if you didn't, here is the quick and dirty of what I was trying to do (if you don't care about this, click here to skip to the solution).

What I was trying to do:

Cloudinary is a cloud-based SaaS product that can serve as both a CDN and image manipulation platform. You most likely have used their services before without knowing it; thousands of sites use their platform as a backbone for fast image serving and manipulation. At the moment, my project involves the image processing part of Cloudinary. One feature that they offer is remote fetch - you give Cloudinary an already hosted image URL, and they mirror the image to your account and provide a CDN link. I wanted to go a step further and overlay a remote URL onto a Cloudinary image, but I ran into a bunch of issues.

Solution:

The crux of the solution is that you need to pass an object to the "overlay" method on a transformation object - like this:

let cloudinaryImageTag = cloudinaryInstance.imageTag(YOUR_IMAGE_BASE);
let tr = cloudinary.Transformation.new();
tr.overlay({
  resourceType : 'fetch',
  url : remoteSrc
});
cloudinaryImageTag.transformation(tr);

Failed Approaches / Lesson Learned:

The reason why I didn't arrive at this solution faster is two-fold. One, Cloudinary, although powerful, has very fragmented documentation covering dozens of libraries and the distinction between them is not always clear. Two, and this was a learning lesson for me, I should have gone to check the actual source code of the libraries a lot sooner. I've always been a RTFM kind of guy, but sometimes even the manual doesn't line up with what *actually* is going on. In this case, the documentation suggested a few conflicting things:

It might not even be possible
- Pretty much every sample code snippet and documentation page referencing the overlay transformation parameter shows the way to do it is to pass in the publicId of the image you want to overlay like this:
- ```
let tr = cloudinary.Transformation.new();
tr.overlay(new cloudinary.Layer().publicId('my_cloudinary_publicId'));
```
  - Of course, remote URLs don't have a public ID - that is the whole point! I need Cloudinary to fetch the remote asset and clone it.
- This started to lead me to believe that what I wanted to do wasn't even possible, and I would need to implement some logic to upload the remote image first and retrieve a publicId before trying to overlay...
  - FALSE: After a bit of digging, I finally stumbled across this page, which suggested that maybe Cloudinary didn't support this until recently (2017), but at least this was a sign I was still going in the right direction and they definitely do support it now
That based on the above, I should base64 encode my remote URL and pass it as part of the publicId.
- The syntax for overlay is l_{publicId}, and since their example uses l_fetch:{base64_OF_IMAGE_URL}, I I tried this with some variation of code like this:

let tr = cloudinary.Transformation.new();
let publicId = 'fetch:' + base64.encode(remoteSrc);
tr.overlay(new cloudinary.Layer().publicId(publicId));

This did not work...

Some of the (auto-generated?) documentation pages started pointing me towards FetchLayer, which extends the Layer class. This seemed like it made sense - FetchLayer is specifically a Layer that is supposed to be referencing a "fetched" asset, so it doesn't have a PublicId
- This is where I started actually looking at the source code for the core Cloudinary JS library. If you look at the source, you can see that I was actually somewhat on the right path with my attempt #2, but me base64 encoding the URL was redundant since FetchLayer already does this.
- This is actually where I got really frustrated. Based on the source and docs, ANY of the following *should* have worked, but didn't:
  - ```
  tr.overlay(new cloudinary.FetchLayer().url(remoteSrc));
```
- ```
tr.overlay(new cloudinary.FetchLayer(remoteSrc));
```
  - ```
  tr.overlay(new cloudinary.FetchLayer({
      url : remoteSrc,
      resource_type : 'fetch'
  }));
```
- All of the above threw the same error:

Uncaught TypeError: layerOptions.substr is not a function

FINALLY - I found two things, I think at the same time. Both brought me to the working solution
1. There is an actual bug with FetchLayer - methods that rely on turning options into a string (.serialize(), .toHtml()) will all fail with the error outlined in attempt #3, because of this line of code, here, and pasted below:
  - ```
  } else if (/^fetch:.+/.test(layerOptions)) {
      result = new FetchLayer(layerOptions.substr(6)).toString();
  }
```
- There is a pull request that specifically addresses and fixes this bug, which is partly how I found out about it in the first place, but it hasn't been merged in yet
2. If you pass a plain object into the transformation object constructor, the first if statement that checks for a plain object evaluates to true, and bypasses the faulty chunk of code entirely! This lets all the regular methods, such as .serialize(), work! The solution is up at the top of the page, or click here.

Amazon Add to Wishlist Bookmarklet

Joshua Tzucker — Tue, 25 Dec 2018 10:43:44 GMT

7/12/2019 Update - Project Status

Unfortunately, this project should now be considered "dead" - Amazon has removed and/or locked down all the services and endpoints that this tool relies on to work, and I could not find suitable replacements, even after a thorough search. Details on this can be found on this Github issue.

Github page: here.

Intro:

The short explanation for what this is: a "bookmarklet" - which is a bookmark that executes code when clicked - that lets you add a product from almost any website to your Amazon wishlist (aka an Amazon Universal Wishlist). Here is a demo showing me adding an item to my wishlist from Target:

Amazon used to offer a bookmarklet that functioned very similarly to what I built, usually referred to as the "Amazon Universal Wishlist Button" or the "Amazon Add to Wishlist Bookmarklet". However, they retired it in 2018 and have been only allowing users to add non-Amazon hosted products to their wishlist if you use their new Chrome Extension. I got annoyed by this decision, because the bookmarklet was perfect for what was needed and was very unobtrusive, whereas the new Chrome Extension runs in the background, is way overkill for just adding things to your wishlist, and has the potential to collect data on your web browsing activities as well as alter pages you look at.

I created this bookmarklet mostly because I was curious if it was even possible to make something that would bypass the need for the Chrome Extension, and furthermore, I wanted to use the bookmarklet myself (and am now doing so).

Installation:

If you want the hard way, you can clone my Github repo, and create a local installer page. If you want the easy way, just use the installer page embedded below:

** Installation embed removed due to project status **

How do I use this?

To use this tool, simply click the bookmark you dragged into your toolbar whenever you are on a product page that you want to add to your wishlist - the popup should appear almost instantly after you click it. You do not have to share any logins or information with me, however, you have to not be logged out of Amazon in the browser you are trying to use the bookmarklet in. Amazon keeps you logged in for a very long time, so unless you logged yourself out on purpose, or are trying to use the bookmarklet in an incognito window, this should not be an issue.

Something is not working!

I created this tool on a whim, mostly because I wanted to see if it was possible to make in the first place, and then also be able to use it for myself. As such, I'm likely not going to devote much time to improving it, or fixing it if Amazon changes something that breaks its functionality. However, feel free to contact me anyways if you have a concern - use the contact link at the top of this page.

Flic + Automagic + Toggl API = Easy Time Tracker Button

Joshua Tzucker — Wed, 19 Dec 2018 09:01:05 GMT

Background:

At my last job, I was given a Flic button to use as I saw fit - at first, after looking up what they cost (around $30) I was skeptical if I could get that kind of value of what was basically just a remote button for my smartphone. However, after playing around with it for a bit, I quickly realized that the real value in the Flic button was not the built-in integrations, but the fact that they allow extending its functionality by the use of Android "Intents", which are a way for different Android Apps and services to communicate across the system. For those technically inclined, being able to use Android Intents as an action when the Flic Button is pressed means that you can basically tie the Flic button to any other Android App, service, or HTTP request, through the use of an automation app like Tasker or Automagic. For me, this opened a world of opportunities.

Flic + Automagic + Toggl

I was so impressed with Flic and their service, that I ended up purchasing a Flic button for my home. This post is about my favorite use of it - as a start/stop/resume button for the time tracker I use, Toggl. I did this by creating a very custom flow in an Android automation app called Automagic. Through the flow I set up, if I press the Toggl button my Automagic flow and code will execute, which ends up doing a few different things depending on the state of my Toggl time tracker:

If...
- Case A: I have an existing time entry already started / in progress
  - The Flic button acts as a "stop timer" button and stops the current timer and time entry
- Case B: My Toggl timer is not running, but my flow is set to "resume" mode and there is a recent time entry
  - The Flic button "resumes" my last time entry. Toggl doesn't really have a true "resume" function - this basically starts a duplicate entry, which Toggl is smart enough to combine with the previous in reporting
- Case C: My Toggl timer is not running, but my flow is set to "new" mode and/or there is no recent time entry
  - The Flic button "starts" a brand new timer going, with the description set to generic "Working Time"
For the creation of new time entries or resumes, my flow passes additional attributes, such as a tag of "API", and "created_with" equal to "Flic + Automagic + API" - this way I can easily identify later on which timer entries were created with the button and which were created manually

Edit 2/26/2019: I've updated my flow and the code in my Github gist to also correctly use the same project ID that was used in the last time entry if resuming an entry. Toggl is not great at grouping entries together unless you explicitly tell it they had the same project ID.

Demonstration Video:

Setting up Flic to broadcast a "global intent"

In order for Automagic to be able to know about the button click from Flic, we need to need to somehow pass an even from Flic to the Automagic app. The Android OS actually has something baked into it for this type of cross-app communication - an "intent".

Basically, you can set the Flic app to emit a global message that the Automagic app can listen in on, by having it emit an intent with the "Send Intent" action. You can set this up in Flic by doing the following:

Open the flic app and tap the button you want to add the intent to
Remove any existing action attached to the trigger you want to use ("Click", "Double Click", etc.).
Tap the trigger name to start selecting an action
Now, in the actions menu, find "Tools" in the bottom of the menu, click it, and then find "Send Intent" as the action
Now you can create a completely custom intent that Flic will emit when it is triggered. It is up to you what values you use; just make sure they match what you are listening for / filtering on in the Automagic app or wherever you are listening for the intent
Make sure to hit "Save Action"!

I created a video that walks through these steps, creating a custom intent with an action string of "globalflicintent-home", which corresponds with my Automagic code. You can see it below:

The Automagic Flow

If you want to use my flow, you will have to make one minor modification. I have redacted my Toggl authentication token (for obvious reasons), so you will need to generate your own token and paste it as the string that will become the global variable called "global_togglAuth" in "set togglAuth to Home". Note that you are not pasting your API token, you should be pasting the Base64 encoded string of either "[YOUR_API_TOKEN]:api_token" or "[USERNAME]:[PASSWORD]". See this page for details. Optionally:

Change the string on the first step to match whatever the Android Intent is that you are sending from Flic.
Change step 3, toggl_action='resume' to toggl_action='new' to have the flow create a brand new time entry if the timer is stopped, instead of resuming the most recent previous time entry

I have uploaded the XML export of my Automagic flow and posted it as a Github gist here.

Here is the visual representation of the flow (click the image for the full size zoom-able version):

Reminders / "Gotchas" about Automagic Scripting

I really like Automagic as an automation app, particularly because it supports custom scripting. However, although the scripting language looks and functions a lot like JavaScript, it is not, and there are some mistakes that JS programmers are like to make that tripped me up a little and are worth mentioning:

Checking if a variable is set / defined / not null - with getValue();
- Let say I want to check if a variable is defined in order to determine how to continue the flow. The easy way to this is with getValue() which is described the docs as follows:
  - ```
  Object getValue(String name, Object default)
  Returns the value of the variable named name or returns default when the variable is undefined or null.
```
- It is easy to glance over this and miss something important... I'll show you the mistake I made:
  - ```
  myVariable = 'Hello World';
  isDefined = getValue(myVariable,false);
  if (isDefined!=false){
     // Do something
  }
```
- In the above code, isDefined actually evaluates to false and the IF statement will never evaluate to true! The key here is that getValue expects a string as the first parameter, which should be the name of the variable, not the variable itself. Here is the correct code:
  - ```
  myVariable = 'Hello World';
  isDefined = getValue('myVariable',false);
  if (isDefined!=false){
     // Do something
  }
```
You can't initialize an object variable the normal way
- If I want to create a new variable that is equal to an object, my instinct is to do something like this:
  - ```
  myObj = {"Lorem" : "Ipsum"};
```
- This is wrong! This will not work with Automagic. Here is how to construct this in Automagic
  - ```
  myObj = fromJSON('{}');
  myObj['Lorem'] = 'Ipsum';
```
- Using the above syntax, you can even create nested objects, and then later, use toJSON() to convert to a JSON string to use in a POST HTTP request.
If you want to check the length of an array, use length(arrVar), not arrVar.length
Use bracket notation to retrieve object values, not dot notation - e.g. myObj['myKey'] not myObj.myKey.
To use variables in non-script actions (e.g. an HTTP request) wrap the variable name in {} - this is actually easy to remember since this is how many templating engines and languages handle combining strings with variables.

Reminder to self: How to fix super slow typing when using USB OTG plugged into smartphone

I was editing this Automagic script on my smartphone, and plugged in a USB keyboard (via OTG) to make editing easier. However, every single keypress took forever and lagged terribly. The problem was instantly fixed when I changed the active Android Keyboard from "Gboard" (Google's distributed keyboard app) to the default "Android Keyboard (AOSP)". My guess is that Google's fancy keyboard app was trying to spellcheck the entire script field and was not detecting a plugged in USB keyboard, whereas the stock keyboard maybe has lower level access and knew to butt out of the "conversation". Regardless, it was an easy fix!

Materialize CSS Styled Business Card

Joshua Tzucker — Wed, 12 Dec 2018 23:33:06 GMT

A quick MaterializeCSS styled business card I threw together for this portfolio site.

Word Permutation Builder

Joshua Tzucker — Thu, 22 Nov 2018 09:08:28 GMT

Weekend Project - OkCupid Messaging Improvements - UI Mockup and Demo

Joshua Tzucker — Mon, 27 Nov 2017 06:00:00 GMT

Weekend Project - OkCupid Messaging Improvements - UI Mockup and Demo

Published: 11/26/2017

Background (Click to Hide/Show)

A common complaint for female users of almost any dating site is the constant barrage of messages from men that are unsolicited, inappropriate, and show that the senders didn’t even skim their profile information. I have heard this from others, and it also seems especially prevalent where I live, where the ratio of (single) females to males seems skewed so that there are more single males. This issue also works both ways, and I’m sure is not limited to the binary representation of gender either.

Obviously, it would be best if society could fix the root cause, and everyone could agree to stop being rude and stop sending messages that are just the word “Hi” or “Ur 2 Htt, Msg Me Rt now”, without even reading someone’s profile, but I don’t have high hopes for that. However, I think that dating sites could do a lot to address this, and I’m a little surprised that OkCupid, which often is ahead of the trend, has yet to do something like this (at least that I’m aware of).

Basics:

I came up with two simple solutions that they (OkCupid or another dating site) could implement in their User Interface (UI) to help reduce the quantity of low quality messages that certain users like to send out frequently and without discrimination. These would serve as "challenges" to make it more difficult for users to send generic messages without reading someones profile.

Both of these solutions prevent a user from sending their first message to another user without having spent some time looking at the profile before initiating. Once the user completes one of the challenges, or if a message has been exchanged between the two users prior, these features/challenges should disable themselves.

Demo for each solution was created using Javascript, and a fake profile page with user information replaced with dummy text.

Solution A: Timer

This is the easiest solution to implement, and to me, is a no-brainer. Requiring users to spend a minimum amount of time (in my example, 30 seconds) looking at a profile before sending a message seems like a completely reasonable thing to do, and I would be flabbergasted if anyone could come up with a solid argument *against* it. If you can't spend at least 30 seconds reading what someone thoughtfully wrote in their profile, why should they spend 1+ minutes reading, thinking, and responding to your message?

Demo Video - Click Below to Play

Sorry, your browser does not seem to support the HTML5 Video Element

Note that the developers console on the right is shown for demonstration purposes.

In this example, the timer stops counting if something is hiding the profile from view. This stops users from simply leaving the error message about the timer up until the time passes, while looking at content elsewhere. This feature could be fleshed out further, and more requirements could be added, such as scrolling to the very bottom of the profile, expanding sections, etc.

Solution B: Custom Question

This solution is more complicated to implement than the first (simple timer). In this feature, a user (the receiver) can set a custom challenge and expected response. The user (sender) trying to send their first message to the receiver will be prompted to answer the custom question, and if their answer matches the one saved by the receiver, they will be allowed to send their message.

Demo Video - Click Below to Play

Sorry, your browser does not seem to support the HTML5 Video Element

Note that the developers console on the right is shown for demonstration purposes.

These questions could theoretically be automatically generated by the system - for example, if the user has indicated in their profile settings that they own dogs (a simple checkbox in OkC’s UI) the system could generate a corresponding question, “Does user [username here] have dogs?, Yes or No”.

The complication with this feature is that it would require creating and exposing a new internal API endpoint to validate answers, since having the answer matching logic purely in client side code would make this easily defeat-able to any tech-savy users, and someone could easily release a chrome extension or bookmarklet so that any user could circumnavigate the custom questions.

Recap and Further Thoughts:

Although this is a working bare-bones demo/mockup, there is a lot of wiggle room on how this could be implemented. For example, instead of enabling this feature across all users, OkCupid could only trigger it for "suspicious" users. For example, if a user is sending a large volume of messages per day in comparison to the average, they could start seeing the 30 second timer while others do not. This would be similar to how ISPs might throttle the connection speed for users abusing bandwidth, or how Google ReCaptcha v2 can be set up to only show when suspicious user activity is detected.

In addition, controls over the functionality of these features could be extended to the receiving users themselves - or adapted automatically. For example, the system could automatically turn on the feature for users that receive a higher amount of messages per day than others, but all users could get a toggle switch to turn it on if they want to.

Marketing Dashboards

Joshua Tzucker — Sun, 19 Nov 2017 06:18:13 GMT

It seems like every day there is a new dashboard / BI (Business Intelligence) / reporting / widget platform arriving, and it can be daunting to keep track of and compare them. Recently, I started organizing a spreadsheet to keep track of the major reporting dashboard offerings, and this is the result of that work.

I’ve embedded the entire Google Sheets comparison doc below, but the formatting is a bit funky with embeds, so if you want a better view, just visit the actual doc here.

With the shift from traditional media marketing (newspapers, print, etc) to digital marketing (especially data-driving digital marketing), more and more data has been flowing out of campaigns, but not necessarily into the proper funnels. Data about performance, data about spend, and even data about operational procedures that were required to launch the campaign all get shot out of various APIs, manual CSV exports, and aggregators. Finding a platform that can ingest this data to produce pleasant looking client facing automated reports, and/or insights for internal operations can be a daunting task. Hopefully this page can help.

By no means is it 100% comprehensive or representative of everything that is out there, but it covers most of the big ones (Google Data Studio, Power BI, Tableau), as well as some of the small(er) guys like Cyfe, Looker, and ReportGarden.

If anything, hopefully this can help marketers as a starting point for quickly identifying which dashboard platforms are outside the scope of their needs, and which ones they should invest more time in researching and scheduling demos with.

If you are upset that your favorite reporting dashboard / BI tool is not on this list, feel free to drop me a line, and I’ll take a look and add it if I think it meets the basic criteria. You can even use this easy to fill out form to submit a suggestion.

Google Chrome Extension - for TimeClockWizard

Joshua Tzucker — Sat, 06 May 2017 05:00:00 GMT

1/4/2019 - UPDATE: Extension Brought Back from the Dead!

I stopped using TimeClockWizard personally a while ago, so when they changed their website sometime in 2017 or 2018 and broke this extension, I don't think I even noticed (or at least not for a while). However, I had some spare time recently, so I decided to rebuild this extension to get it working again. It actually only took about a day, and now is functional again. If TCW changes their system again, and the extension once again breaks, feel free to let me know by commenting on this page. However, I can't guarantee that I will continue supporting this extension indefinitely.

You can get the extension here!

For work, I've been using a site called "Time Clock Wizard". From a managerial side of things, it is pretty good, since the cost is low and it fulfils all the basic bookkeeping requirements. However, from the user perspective, the site leaves a lot to be desired; it constantly logs everyone out, it displays accumulated hours per pay period instead of the normal Monday-Friday (which means we constantly have to pull custom reports to see how close we are to hitting overtime), and it is slow to navigate.

Since I've been toying around with Chrome Extensions lately, I figured it would be fun to try and make one so that I could more easily use the TimeClockWizard service - so that is exactly what I did - I coded up an entire chrome extension over a single weekend.

Features:

Only have to sign in once!
Clock In and Out all from the extension popup page
See your last punch time at a glance - answer "What time did I clock out for lunch?"
Check your weekly hours - avoid accidentally going over

See it in action!

Clock In:

Clock Out:

Resyncing Full Time Report:

Sometimes, if you are clocking in both through the website and through this extension, the estimated total weekly hours can start to go adrift. You can use the button provided to force a full resync of total weekly hours.

Technical Stuff!

After a weekend project I like to sum up cool things I used / learned / experienced - either for the first time, or in some way that felt transformative. Here are those things:

Chrome Extensions (duh!)
Local Storage (aka localStorage)
Date formatting
- Unlike other aspects of code that I start to like the more I use them, the more I work with date formatting (converting, offsetting, etc.) the more I HATE it.
XMLHttpRequest(s) - simple GET and POSTS can be so powerful :)
Async stuff (callbacks, promises)
Regex (is there ever a project that doesn't use rexeg? Don't answer, that was rhetorical).

If you are interested enough to be reading this section, you might already know, this but TimeClockWizard does not have a published API. Thus, the most challenging part of this project was not the UI or related code; it was trying to figure out how TCW handles network requests - spoiler: it is not as straightforward as one might think.

Source code might be published at some point...

Google Analytics Tester Bookmarklet

Joshua Tzucker — Mon, 07 Mar 2016 06:42:48 GMT

Grab the Google Analytics Bookmarklet!

To use the bookmarklet, simply drag the following link to your toolbar, then click on the bookmark whenever you want to test a page! - Drag this link: Google Analytics Tool

Warning: This tool is in beta! Released 3/6/2016. Page will be revised if updates are made.

What does this tool do?

I created this bookmarklet primarily for my own uses, but seeing as there is nothing out there like it, I felt it was worth sharing. Here is what this Google Analytics checker and event testing bookmarklet does:

Google Analytics Code Checker: Checks for ALL Google Analytics tracking codes that are on a page (regardless of what version of GA they are using - e.g. newer analytics.js vs older ga.js)
Google Analytics Accound ID Report Generator: Spits out a report of all the google analytics account IDs that are found on the page, and what type of analytics library they are using.
Google Analytics Event Tester: Verify that your Google Analytics is receiving data by clicking a button and triggering an event to be recorded. My bookmarklet will send the event to all the analytics accounts that are properly tracking the page.
Retest the page after DOM has changed: You can temporarily close the floating bookmarklet overlay panel by pressing on the x, manipulate the page and trigger changes, then reopen by clicking on the eye icon in the upper left, and continue to re-test Google Analytics or trigger more events!

Video Demo of Tool in Use

GIF Showing Hiding and Re-Opening of Floating Report Window

This tool does NOT inject any external script sources into the page. Although doing so would allow me to update the bookmarklet without requiring it be removed and re-added to users' browsers, it also would mean that the bookmarklet wouldn't work on sites that use the newer Content Security Policy (CSP), which I wanted to avoid as CSP gains momentum. The downside means that you might want to check this page every once in a while if you care about if the tool gets updated.

This bookmarklet is composed of a few major components:

Code that detects GA scripts and what type they are - this is trivial to do, and you can do so simply by regex testing against all scripts and their sources on the page (see here)
Code that handles creating events for both versions of the Google Analytics JS library and sending them to ALL attached accounts - this took me a while to make, and ended up being a project on its own. I'll post about it soon and the reusable JS you can use to easily trigger Google Analytics events on button clicks, hovers, and other event listeners.
An injected UI widget - this is the floating div that appears on the page when you use the bookmarklet. I am severely graphically untalented, so CSS continues to be a struggle for me. I think I did alright with this though.

Wijmo Combobox with an AJAX Data Source (Wikipedia API)

Joshua Tzucker — Sun, 28 Feb 2016 05:44:33 GMT

Want to skip right to the working demo? Click here.

Intro:

For a recent project, I needed to implement a ComoBox that a user could type into, and they would be offered a dropdown list of matching selections based on a third-party API (think of how Google Auto-suggest acts). This is difficult for a number of reasons, which I list further down. However, an easy workaround/hack I found was to use JSONP requests (courtesy of this answer on Stack Overflow), combined with the Wijmo 3 UI Widget Framework. The Wijmo ComboBox widget can be a little confusing to implement though, especially with a dynamic data source, which is the main reason why I came up with a demo and wrote this post (as a weekend project).

Building a Wijmo ComboBox with an AJAX Data Source

Working Example:

Since this is probably what most people are interested in seeing first, here is a working example. I am using the (unofficial) Wikipedia API as my source, so as you type in the ComboBox, you will see entries from Wikipedia matching your query. Pretty cool, right?

See the Pen <a href='http://codepen.io/joshuatz/pen/MyWOXG/'>Wijmo ComboBox with Cross-Domain AJAX Data</a> by Joshua T (<a href='http://codepen.io/joshuatz'>@joshuatz</a>) on <a href='http://codepen.io'>CodePen</a>.

Step 1 - Load Wijmo resources, widgets, and set initial options

The first step is to load all the necessary Wijmo resources (JS, CSS, etc). It is best to do this in the head of your document. Here is how I did it:

<head>
<!--         Wijmo Dependencies       -->
<!--Theme-->
<link href="http://cdn.wijmo.com/themes/aristo/jquery-wijmo.css" rel="stylesheet" type="text/css" />
<!—Wijmo Widgets CSS—>
<link href=“http://cdn.wijmo.com/jquery.wijmo-pro.all.3.20153.83.min.css” rel=“stylesheet” type=“text/css” />
<!—RequireJs—>
<script type=“text/javascript” src=“http://cdn.wijmo.com/external/require.js”></script>
<!—Config Jquery for Wijmo—>
<script type=“text/javascript”>
requirejs.config({
baseUrl: “http://cdn.wijmo.com/amd-js/3.20153.83”,
paths: {
“jquery”: “jquery-1.11.1.min”,
“jquery-ui”: “jquery-ui-1.11.0.custom.min”,
“jquery.ui”: “jquery-ui”,
“jquery.mousewheel”: “jquery.mousewheel.min”,
“globalize”: “globalize.min”
}
});
</script>
<!—         End Wijmo Dependencies       —>
<!—		 Setup Wijmo ComboBox		  —>
<script>
require([“wijmo.wijcombobox”], function () {
$(document).ready(function () {
$(“#Wikipedia_ComboBox”).wijcombobox({
data: [{
label: ‘Please start typing to get suggestions.’,
value: ‘null’
}]
});
$(“#Wikipedia_ComboBox”).wijcombobox({
select : function(){afterSelection()}
});
$(“#Wikipedia_ComboBox”).wijcombobox(“option”,“autoComplete”,false); //turn off auto-complete
$(“#Wikipedia_ComboBox”).wijcombobox(“option”,“forceSelectionText”,false);
});
});
</script>
<!—			End WijmoComboBox Setup				—>
</head>

You don't really need to worry about the above HTML, just swap "Wikipedia_ComboBox" for the ID of your own ComboBox. The only thing to note is that the

$("#Wikipedia_ComboBox").wijcombobox({select : function(){afterSelection()}});

code is a cool trick - Wijmo is letting me define a custom function I want to have called after a user makes a selection within the ComboBox, which in this case, is my afterSelection function (see more later).

Step 2: Write out your HTML

Obviously, this step is going to be very different for each person and their needs. In my case, I'm just creating a demo, so my HTML is very simple - a single input that will get turned into a combobox, and an extra div where the selected Wikipedia link will be displayed.

<body>
<input id="Wikipedia_ComboBox" placeholder="Type to Search and Use Dropdown" class="wijmo-wijcombobox wijcombobox" style="width:500px; height:50px;"/>
<div id=“Entry_Selection”>
<div>Here is the link of your selected Wikipedia entry: </div>
<div id=“Wikipedia_Link_Area”><a href=“null” target=“_blank” id=“Wikipedia_Link”>Entry Link</a></div>
</div>
</body>

Step 3: Write out the Javascript!

Here is the basic flow of my Javascript:

User enters a character into the combobox, which triggers a new query
I use the contents of the ComboBox as the input for my Wikipedia search (the search parameter). I use the JSONP-hack function to make the request with my query and send the result to my callback function
After a response is received from the Wikipedia API, my custom callback function is triggered. It parses the return data (which is JSON formatted) into an array, which is then passed to the Wijmo combobox via the data option. The standard format for the Wijmo data option is [{label: label, value: value}].
The user selects a Wikipedia entry from the dropdown auto-suggest list
Wijmo remembers I gave it a function to call if a selection is made, so it triggers that function. My function kicks in and updates the Wikipedia Link section with a link to the selected Wikipedia entry.

And, here is all the Javascript:

var dynamicData = [];
function jsonp(url, callback) {
var callbackName = ‘jsonp_callback_’ + Math.round(100000 * Math.random());
window[callbackName] = function(data) {
delete window[callbackName];
document.body.removeChild(script);
callback(data);
};
<span style="color: #008800; font-weight: bold">var</span> script <span style="color: #333333">=</span> <span style="color: #007020">document</span>.createElement(<span style="background-color: #fff0f0">&#39;script&#39;</span>);
script.src <span style="color: #333333">=</span> url <span style="color: #333333">+</span> (url.indexOf(<span style="background-color: #fff0f0">&#39;?&#39;</span>) <span style="color: #333333">&gt;=</span> <span style="color: #0000DD; font-weight: bold">0</span> <span style="color: #333333">?</span> <span style="background-color: #fff0f0">&#39;&amp;&#39;</span> <span style="color: #333333">:</span> <span style="background-color: #fff0f0">&#39;?&#39;</span>) <span style="color: #333333">+</span> <span style="background-color: #fff0f0">&#39;callback=&#39;</span> <span style="color: #333333">+</span> callbackName;
<span style="color: #007020">document</span>.body.appendChild(script);
}
function handleAjaxResponse(input){
var temp = document.getElementById(“Wikipedia_ComboBox”).value;
dynamicData = []; // reset our data source before filling it with resopnse
for (x = 0; x < input[1].length; x++){
dynamicData.push({“label”: input[1][x].toString(), “value”: input[3][x].toString()});
}
$(“#Wikipedia_ComboBox”).wijcombobox({
data: dynamicData
});
document.getElementById(“Wikipedia_ComboBox”).value = temp; // Necessary for a stupid reason; manipulating the data for the combobox seems to reset the input field, which we don’t want
}
function afterSelection(){
setTimeout(function(){ // setTimeout is necessary to avoid strange results due to wijmo/ajax delay 
console.log(“An entry has been selected”);
var Wiki_Link = document.getElementById(“Wikipedia_Link”);
Wiki_Link.href = $(“#Wikipedia_ComboBox”).wijcombobox(“option”,“selectedValue”);
Wiki_Link.innerHTML =  document.getElementById(“Wikipedia_ComboBox”).value;
},500);
}
document.getElementById(“Wikipedia_ComboBox”).onkeyup = function (){
var query = this.value;
var URL = “https://en.wikipedia.org/w/api.php?format=json&amp;action=opensearch&amp;limit=10&amp;search=“+encodeURI(query);
jsonp(URL,function(data){handleAjaxResponse(data);});
}

ComboBox Issues

ComboBox is not a native HTML input!

A Combobox, as its name might imply, is really a mashup of separate entities; the standard HTML input tag with a "text" type, and the HTML select tag (drop-down selector). Thus, the only real way that a combobox can be created is through Javascript. Luckily, there are several JS UI frameworks that have pre-built comboboxes to use. Here are some nice ones:

HTML has a lot of Cross-Domain Restrictions!

Restrictions on the ability of Javascript and HTML to make network requests and access cross-domain content is very good for the sake of the user; this prevents embedded banner ads from executing malicious attacks and other various attack vectors. Unfortunately, it also makes integrating APIs and third-party tools into a first-party page very tricky.

For example, many sites will let you embed a generated <form> into your page, but only through an iframe. The embedded iframe keeps the source of the original site, which means it has a different domain than your own site that you are embedding it into it. Now, due to the same-origin policy, you can't interact with it via Javascript. Bummer.

OK, so forms are out as a way to interact with third party content. How about actual HTTP requests? The XMLHttpRequest API is a super easy way to send and receive request via vanilla Javascript and even allows us to do advanced things like specify the request method and whether we are making an asynchronous request or not. Oh wait... the XMLHttpRequest method is also subject to the browser's cross-domain policy! So, you can make requests to your own domain, but requests to other domains are out, unless they have CORS enabled for your specific domain (which 99% of the time means you are SOL).

Argh... OK, so how about a compromise? What if we use server-side PHP with cURL to make and receive the request, while client side Javascript talks to that PHP page? Success! That will work! Of course, we are basically making our own API on our own domain, which is fine if we are building a first-party system, but for accessing a third-party API/datasource means double the work with double the latency. Another issue is that now instead of request originating from the user's IP, 100% of request are originating from our server, which without IP rotation, means all from the same IP. Thus, very easy to get either banned or rate-limited access to a third-party API. Not to mention increased use of our bandwidth and server processing resources. However, this is an OK tool for quick and dirty iframe-busting. Otherwise, not a great solution for integrating third-party APIs. Bummer.

Comboboxes and AJAX - More Resources:

Summary of JSON-P and cross-domain AJAX issues
Summary of cross-domain AJAX request workaround methods
Postman - a very useful tool for playing around with HTTP requests, especially if you are trying to understand third-party APIs

Google Scripts - ExaVault Direct Link Generator

Joshua Tzucker — Mon, 11 Jan 2016 06:00:00 GMT

ExaVault is essentially a service that provides file hosting with a simple directory interface, but has the distinct feature of offering FTP access and advanced permissions (which is why its user base is mainly businesses / enterprise). To make part of my job easier at work, I wanted to integrate our ExaVault access into a Google Scripts project I was working on. I wanted to have my Google Script retrieve a CSV from our ExaVault folder and parse and process it into a specific format. However, I quickly ran into an issue.

ExaVault, by default, provides links to files through the FTP protocol. The links look like “ftp://links:pass@test.exavault.com/transfer.pdf”. Unfortunately, the UrlFetchApp, which is a service Google Scripts uses to handle web requests, only works with HTTP/HTTPS requests/responses, and not the FTP protocol. So, I needed to figure out how get a direct link to my CSV in ExaVault, instead of a FTP link. The solution? Use the ExaVault API to generate a temporary direct link. It ended up being much easier than it sounds - ExaVault’s API is extremely easy to use.

Here is my reusable code (below) - the return of my function is the direct download link to your file, which you can then pass to the URLFetchApp. Feel free to modify/reuse/do whatever you want with it (attribution not required, but is always appreciated). Username, password, and filepath are all mandatory arguments for the function, whereas opt_downloadname is optional. Make sure “api_key” is declared and set to your api key value (you will need to do a one-time generation of an API key here: https://clients.exavault.com/clientarea.php?action=products).

var api_key = "PUT_YOUR_API_KEY_HERE"; // DO NOT SHARE WITH ANYONE, PRIVATE API KEY FOR EXAVAULT
var access_token;
function Exavault_URL_Fetcher(username, password, filepath, opt_downloadname) {
var done = false;
var downloadname = (opt_downloadname || “downloadfile”); // use the optional download file name if provided, otherwise default to “downloadfile”
while (done == false){
if (access_token == undefined){
Logger.log(“Retreiving access token using user+pass”);
var payload = {
“username” : username,
“password” : password,
“api_key” : api_key
};
var options = {
“method” : “post”,
“payload”: payload
};
var response = UrlFetchApp.fetch(“https://api.exavault.com:443/v1/authenticateUser”, options)
var parsed = JSON.parse(response);
var token = parsed.results.accessToken;
access_token = token;
}
else {
Logger.log(“Retreiving Download URL using token+key”);
var payload = {
“access_token” : access_token,
“filePaths” : filepath,
“downloadName”: opt_downloadname,
“api_key”: api_key
};
var options = {
“method”: “post”,
“payload”: payload
};
var response = UrlFetchApp.fetch(“https://api.exavault.com:443/v1/getDownloadFileUrl”, options)
var parsed = JSON.parse(response);
var downloadURL = parsed.results.url;
return downloadURL;
done = true;
}
}
}

Photography Gallery

Joshua Tzucker — Wed, 29 Apr 2015 11:21:46 GMT

Here are some samples of my basic photography skills

Note:

If you want to see a larger version of any the pictures below, click the thumbnail for an enlarged view.

Gallery:

Stukent Online Marketing Simulation (Competition)

Joshua Tzucker — Sun, 26 Apr 2015 05:00:00 GMT

Here is an overview of my work on a marketing simulation (called "Stukent")

Overview:

In December of 2014, I had the opportunity to use the world's first online marketing simulation, named "Stukent", in a class I was taking on electronic marketing. In a class of 43 students, we competed against each other in small groups over the course of the quarter, with a total of 6 rounds. The goal was to generate as many sales as possible of various tablets, and with the highest return on investment. I was selected team leader and ended up making 95% of the decisions over the course of the simulation, which meant setting up landing pages, creating email campaigns, managing SEM (both organic and paid), selecting keywords and adjusting bid amounts, and creating ad copy. I kept track of all metrics and decisions in Excel, and by the end of the course, was running 23 ad campaigns, 19 email campaigns (with multiple A/B tests), and 21 landing pages (with an average score of 91/100).

Results:

In one sentence: We placed in the two top for every single round, and were the #1 team at the end of the simulation, placing first in ROAS/ROI, traffic, market share… essentially every single metric.

Here is a breakdown of the improvement in our metrics by round:

Round	Impressions	Clicks	CTR	CPC	Conversion Ratio	ROI	CPA
Round 1	67,384	1,685	2.5%	$0.81	7.6%	2.99	$10.71
Round 2	75,049	1,812	2.41%	$0.80	7.51%	2.99	$10.72
Round 3	65,567	1,548	2.36%	$0.65	9.56%	4.46	$6.83
Round 4	107,525	2,598	2.42%	$0.50	10.66%	6.43	$4.66
Round 5	180,670	4,521	2.5%	$0.55	12.12%	6.51	$4.56
Round 6	247,848	6,647	2.68%	$0.56	13.54%	7.42	$4.14

To summarize the improvements, we more than doubled our return on investment by the end, generating $7.42 per every dollar invested. Our cost per action decreased to less than 50% of its original value, as a result of our CPC falling even as click-through-rate, impressions, and conversion ratio all rose.

Even if you average our scores across the entire simulation, we still came out as the top ranking team:

School Traffic Light: Timer and Noise Detector

Joshua Tzucker — Sun, 12 Apr 2015 00:00:48 GMT

I made a video to demonstrate the build process and features of this project - it is the fastest way to get an overview of what this is all about

All content on this page is protected under copyright, except for the images, which I will allow anyone to use as long as you follow Creative Commons 3.0 Attribution License (basically, just link to this webpage).

Inspiration:

This project came about thanks from a suggestion from my mother, who is an elementary school teacher. I was looking for a fun new electronics project to work on, and she mentioned that she would like a way to visually let students know how much time they have left on an assignment, or to adjust their voices to match the desired volume of the room. She said she had tried existing products (such as the Time Tracker), but they were too small and complicated. She had heard of the Yacker Tracker, but had not heard great things about it, and it was rather expensive. I came up with the idea to combine the two requirements (noise and time tracking) into one device, which I could build using a microcontroller (Arduino platform) and simple electronics (transistors, buzzers, capacitors, etc).

Creation

Rather than completely fabricate the body of a traffic light from scratch, I decided to start with a premade body. I looked around online, and my best option seemed to be a randomly blinking traffic light, which are sold as novelty items for dorm rooms / bachelor pads / etc. I found one on eBay that fit my size and price requirements, and bought it (see below).

I took the traffic light apart, and found that the bulbs were powered directly from AC, but were too dim to be visible from across the room. I ended up completely removing all the internals in the enclosure and fabricating my own bulbs from SMD LEDs (See below picture of a custom LED bulb I created).

They are very bright, safe to use, and consume very little power. As shown in the image below, all three SMD LED panels operating at the same time only consumes a total of slightly over 200mA, low enough that a standard laptop can power the device.

A lot went into this project, but here are a few major components.

A small circuit (based on this design) to take the output from a standard microphone and amplify it so the Arduino can measure the sound level in the room.
A small circuit (based on this design) to take the output from a matrix keypad and send it to a single analog input pin on the Arduino.
The actual code for the microcontroller to control everything and process user input - around 1,200 lines of C in total.
Most of my time was spent trying to make the device as user-friendly as possible. This meant creating prompts that show up on the LCD screen to instruct the user how to input choices, logic implemented in code that prevents users from making choices that don't make sense, and controls that are easy to use and clearly indicated.

I'm sure I left out some other aspects of what went into the build process, but it would take me forever to write up documentation on every step of the project.

Finished Product

Here are some pictures of the final result of all my hard work!

I also created an instruction sheet for my mom to have as a quick reference guide to using the traffic light:

Things I Could Have Done Differently:

Used pre-made SMD bulbs (such as those sold for auto use) instead of taking the time to solder my own on perfboard.
Purchased a pre-made traffic light that uses LED bulbs, instead of the one that I bought (that I had to strip and ended up being built with cheap materials).

I actually think this turned out great, considering the cost of the individual parts and the effort that went into it. Hopefully it will survive the classroom...

Thanks for checking out my project page!

Mini Bookstore Project

Joshua Tzucker — Mon, 23 Jun 2014 03:59:06 GMT

Just thought I would share a small project I just finished - a mini model bookstore for my mom (Robin) as a gift. I started with a design in Sketchup (a 3D-modeling program), then flattened all the individual components into 2d objects, printed them out, cut and glued them onto cardboard, cut the shapes out, and glued everything together. It turned out much better than I expected, and I was even able to add some electronics to the project (a flickering fireplace and overhead lights) and with an auto-shutoff powered by a 555-timer circuit.

Here is an animated GIF showing the 3D-modeling process in Sketchup:

Here is an animated GIF showing the actual build process:

Here are some pictures showing what the finished product looks like:

The awesome flickering Fireplace:

For those that want to know more about how this was created, here are the tools that you need:

Sketchup 2014 (formerly owned by Google, no longer)
The "Flattery" plugin for sketchup. This will allow you to "unfold" models so they become 2D. Official site is no longer good, use this page with updated plugin for 2014.
Export images to 2D using built in export option in Sketchup. This site has more info.
To make sure images were printed to scale, I added markers in sketchup with dimensions before printing, then in Microsoft Publisher, scaled the images until the dimensions matched the print size. You can use a multitude of image editors to accomplish this though.
Printer (any kind will do)
Lots of cardboard! I recommend recycling old shipping boxes if you order a lot from Amazon and other online retailers.
Exacto or other sharp knife to cut out cardboard (hobby knife recommended)
Hot glue gun and lots of hot glue sticks

Wireless Water Feature Meter

Joshua Tzucker — Mon, 13 Jan 2014 01:00:07 GMT

Video Overview:

This is was a long and complex project, so a video is going to be the easiest way for me to give a very quick overview of this project. Please watch it if you have the time.

All content on this page is protected under full copyright (all rights reserved), except for the images, which I will allow anyone to use as long as you follow Creative Commons 3.0 Attribution License (basically, just link to this webpage).

Note: Click on (almost) any image on this page to see it full-sized!

Table of Contents:

The “Why?”
The End Product
Bill of Materials
Beginning of Technical Section
Source Code
How It Works
My Background
How It Started: The Idea
Procedure

Prototyping
Building
Programming

The Issues
Things I Could Have Done Differently
Thanks

The “Why?” I’m a frequent reader of technology news sites and blogs, so I have been hearing about Arduino and microcontrollers for a while now. However, I have never had a class on circuits, or on programming, nor did I see a use for them for myself, so I didn’t see much reason to start using them. However, I decided that at some point I would give them a try, but only if I could find a project that would actually be beneficially and serve some actual purpose, rather than creating code that simply flashes lights and looks pretty (a common first project for those learning to use microcontrollers).

Then, in July of 2013, roughly half a year ago, my parents (mostly my dad) built a running stream in the back of their yard.

The Stream (from above)

The water flows down a long bed of rocks, and at the end, flows into a large 3-4 feet deep hole. At the bottom of that hole is an electric pump, which pulls the water up and out of the hole (which I will now refer to as the “tank”), through a hose, and back the top of the stream, where it starts the cycle all over again with a very pretty waterfall. The risk in how this system works is that if the water level got low enough, to where the pump at the bottom of the tank was pulling in air instead of water, the motor would rather quickly burn out, and pumps are not cheap. As it was the summertime, my dad said that this was worrying him, as water was evaporating rather quickly from the large exposed surface of the stream. To make matters worse, the hole at the end of the stream is covered with flat rocks, which although makes it pretty to look at, makes it near impossible to check the water level of the stream at a glance.

The end of the stream

It was out of this worry and inconvenience that I decided to build something to help them. I decided to create a device that would let them check the exact (or as close to exact) water level of the stream, from the comfort of inside their house.

Summer classes got in the way of getting work done on the project immediately, so there was some delay, but eventually I started putting time towards it. I first brainstormed different ways I could read the water level of return tank at the bottom of the stream. I did some research online and dismissed many ways that were possible, but I felt weren’t the best. If you want to see my brainstorming and build process more and detail, skip down below to the more technical section, or keep reading and you will get there eventually. Anyways, I eventually landed on the somewhat unique idea of using a long pole with a common anode running the entire length of the pole, with 8 nodes (screws in the pole) that would be connected to 8 analog inputs on an Arduino board.

A very bad illustration of my idea

When the pole gets submerged in water, any node that is underwater is going to be connected to the common anode, since water conducts electricity, and will read a positive voltage (my setup is somewhat similar to this instructables). The way it works is a bit more complicated than that, but again, if you want a more in depth explanation of the project, it will come later, in the bottom section.

Originally, I just planned to only measure the water level of the stream, wirelessly transmit it from the outside unit to a box sitting inside, and then have the inside box display the water level by way of 8 little LED lights that would either be on or off, depending on the level of the water. However, as I started to design my project, I started to realize that there were more and more features I could easily add with just a few lines of code, and I also found out how incredibly cheap electronics are direct from China using eBay. At a certain point I should have stopped myself and resisted the slippery slope of “feature creep”, but I didn’t, so the end product is way more advanced than I originally planned.

The End Product In terms of the programming that is in the end product, I won’t go into great detail, but will say that I wrote over a thousand lines of C for the program that runs on the microcontroller (if I were to print it out on paper it would use roughly 24 pages) and there are currently no (that’s right, zero) bugs to report.

Time: About 4 days to solder everything and get the enclosures completed, 2 hours to program the external unit, a few days to program the internal unit, and then a few more days to debug all the code. Probably somewhere around 1.5 weeks total of actual work.

What I like about this project, and why I made it, is that there is no commercial product (at least not that I could find) that even comes close to comparison. Most water alarms either use one point of electrical contact to sense whether water is present or not, but not the amount present (such as a buzzer that you might put in your basement to warn of flooding) or something called a “float”, which usually only measure to 2 levels of accuracy – whether the tank is full or near empty. Pretty much all of those commercial products also don’t work wirelessly – the alarm has to be where the water is. Finally, even units that only measure full or empty wirelessly are very expensive – this one is not even wireless, but uses a tethered alarm to alert to a tank being full or empty, but it starts at around 400 dollars .

My end product

measures how full the tank is in accuracy of 1/8ths (12.5%)
communicates over 2.4Ghz radio signals (can communicate through walls, has a theoretical range up to 2000 feet, etc.)
reports outside temperature, outside humidity, and the temperature of the water
keeps the current time, date, and minutes the system has been running for
has both an auditory and visible alarm
has configurable alarm settings and refresh intervals
Can provide alarms for water level, temperature (for example, if the water temperature is below freezing), and humidity
What I call “logic checking” – code that makes sure the information it is receiving makes sense

Considering all these features, that units with only one or two of my 20+ features cost hundreds, and that there is no manufacturer that makes anything commercial that even comes close to what my units do, you might be amazed to learn that all my parts together cost me… drumroll please…. Less than 80$!!!!! Seriously!!!

The end product is really two units. One unit is installed outside and should never have to be touched again, and the other unit is a box that sits inside, that has a user interface and displays all the information collected by the outside box. The outside unit is an Iduino Nano (an Arduino clone board), with an ATmega328, stuffed inside a weather-tight enclosure.

The External Unit

It has a 2.4Ghz radio attached to it, for communication to the inside unit, a temperature/humidity sensor inside the enclosure, various electronic parts such as resistors, and is powered by 5v supplied by the same outlet for the water pump, and has a long cable coming out of the enclosure that leads to a PVC pipe. The PVC has all the electronics on it that provide information about the water; depth and temperature.

The PVC Probe for the External Unit

In addition, the box that sits outside has a bright flashing LED that lights up when the PVC pipe is “electrified”, which is visible even from our second floor deck during daylight.

The internal unit is what took most of my time to build and code for, and what the user (my parents) will see at least a few times a day. It is a rather small box, but most of the front surface of it is occupied by an LCD screen, which serves to display all the information about the stream.

The Internal Unit

It has four buttons above the screen, and one on the side, as well as a power jack, a USB jack for reprogramming, a video jack (in case someone wants to use the LCD screen on the box to display video from some other source), and 9 (yes, nine) bright LEDS that flash in coordination with a buzzer to alert the user if there is something they should know about (such as the water level reaching too low of a level). Inside the unit there is a SainSmart UNO (another Arduino knockoff, albeit a great one), a transistor, a 9V battery, a “Real-Time Clock” (RTC) with a backup battery, and a buzzer. And of course, the other 2.4Ghz radio that communicates with the outside unit.


Diagnostics Screen

Clock Screen

List of “most” materials: (please note that this list is not so that someone can recreate what I built, just to give a starting point for ideas)

A “weatherproof” electrical junction box, such as this, this, or this. However, I actually am not very satisfied with mine (I believe I have the 6x4).
An aluminum flat bar for the common anode (I used this one and cut it to match the length of my PVC probe).
Lots of hookup wire (either 22 or 20 gauge, I recommend solid core, not stranded).
Arduino Uno (either name brand, or generic SainSmart)
Cheap auto backup/reverse LCD screen, such as this one that I used
(2) 2.4Ghz nRF24L01+ RF Transceiver Modules (Amazon, or you can find them much cheaper on eBay, but shipping might take 2-4 weeks)
Arduino Nano (or clone, such as this one that I used)
Tools: Soldering iron, solder, desoldering wick, hotglue gun, hotglue, epoxy (for weather sealing), rotary tool for cutting enclosures (I highly recommend this one), wirestripper and wirecutters, wrench, screwdriver
Components: transistors, multicolored LEDs, LED holders, momentary pushbuttons, sliding switches, perfboard (I used some random boards from RadioShack, but Amazon also sells them), buzzer, 9v battery, various resistors, terminal strips, RCA plug/cable, galvanized bolts and nuts (available at hardware stores), weatherproof PVC pipe (also at hardware stores)
Special parts: DHT22 (Amazon, or cheaper on eBay), DS18B20 waterproof temperature sensor (Amazon or eBay), Real Time Clock “RTC” module (Amazon, or cheaper on eBay), rubber stopper/plug for creating the seal around the cables going into the external unit (I got mine from Ace Hardware, but most hardware stores should carry them)
Weatherproof landscape lighting cable (I recommend 16 gauge or thicker – lower numbers are thicker) for the water depth probe and extending the USB power cable for the outside unit (Amazon, HomeDepot, or local hardware store)
AC-USB Adapter for external unit, AC-DC barrel adapter for inside Arduino UNO unit and to power screen (needs to be 5.5x2.1mm center positive, at least 2 amps to power both UNO and LCD screen, such as this one)

The “Technical” If you are viewing this page because I asked you to on FB, or one of my parents shared it with you, or for whatever other reason you might have, but you are not interested in electronics or software, or at least not in this application, or if the above sections confused you, this might be where you want to stop reading. I thank you for reading so far, and I realize that not everyone finds the same subjects interesting (I stop reading articles about sports generally after reading the title). Now, if you are interested in technology/microcontrollers/software etc., and you want to see more of how this was built and what exactly went into it, this is the section you want; here is where I will go into the nitty-gritty, and even be completely honest about the issues I encountered.

Source Code: As of right now, I have no plans to ever release my source code (either for the internal or external units). There are a few reasons why. First, this was my first programming/electronics project. As such, although my code works, it is messy and I would be embarrassed to have strangers look at it. Second, it would likely be of no help to anyone. I haven’t really provided enough information in this writeup so that someone could actually duplicate my project, and every stream is going to be different, so there is no way someone could duplicate the electronics in my project exactly, to where my code would actually run the same.

How it works: Before covering how I made it, maybe first I should address, in a more technical manner than I did above, how the units work. The inside unit has an LCD screen that displays information about the air and water outside. If the user presses a certain button, or a preset amount of time passes (right now set to 20 minutes), the inside unit will request an update from outside. It does this by sending an integer value over 2.4Ghz using the incredibly cheap nRF24L01+ Radio (can be purchased on eBay for as low as $1 per radio). The outside unit is constantly listening, and when it hears that integer over the radio, it starts to gather data to send back. It quickly supplies 5v to the common anode and then does analog reads of the 8 screws to get the water depth, measure the temperature and humidity of the air inside the enclosure by using a DHT22 sensor, and the temperature of the water by polling a DS18b20 waterproof temperature sensor. It gathers all that data in a matter of milliseconds, packs it into an array, and sends it back to the inside unit, which is now switched to a listening mode and is awaiting the updated information. Upon receiving the new information by way of the radio transmission, the internal unit inside the house rips apart the array and stores its values into variables in memory. It then reports back the values to the user by displaying them on the GUI. If one of the new values is outside of a preset range (that is adjustable), for example, if the water is below freezing, than the alarm LEDs will flash and an alarm buzzer will go off (that unfortunately sounds identical to our dishwashing machine, which means everyone will ignore it).

A sample error screen

Additionally, since there is too much information to display on the screen at once, the user can press two additional buttons to get extra information. Pressing one button with a clock icon shows the current date in DD/MM/YYYY format, as well as the current time in 12-hour format.

Clock Screen

It gets that information by talking a “Real-Time Clock”, which is a separate module that only functions to keep precise track of the current time and has a backup battery to ensure that it continues to keep the current time even in case of a power failure (real time clocks are in most computers and in many personal electronics). If the user presses another button, with the icon of a stethoscope, a diagnostics screen is displayed.


Diagnostics Screen

The diagnostics screen reads out the amount of free SRAM on the Arduino board, the data rate of the wireless 2.4Ghz radio, the CRC length for the radio, the PA level of the radio, the humidity of the outside enclosure (since the humidity sensor is inside the enclosure), as well as the precise analog values that the outside unit got when it read the analog pins. The main GUI, which is always displayed on the screen unless there is an error message to display or one of the buttons is being pressed or an update being fetched, displays the current level of the tank visually, through a drawing of a tank divided into 8 sections, displays the minutes since the inside unit last received an update from the outside unit, the level of the tank in percentage, the outside temperature of the air, and the outside temperature of the water.

My Background I’m a 21 year old male, living in Seattle, a full-time student, and a complete coffee addict. Keep in mind, particularly while reading this section, that I am completely new, not just to Arduino, but coding and electronics in general. Prior to this project my only experience with code was a summer camp as a child learning ActionScript (the code that is behind Adobe Flash), a robotics class in Highschool, that didn’t even use code (it used drag and drop blocks, which I actually was able to dig around and figure out it turned into plain C before compiling and sending to the robot), and basic HTML/CSS for some web sites. My only experience with electronics had been taking stuff apart (which I loved to do, sometimes to the horror of my parents as I dismantled electronic gifts and items I had scrimped and saved for) and maybe once or twice soldering super simple wire connections on something that had broken. I have had no classes on C, no classes on even basic circuits (except for maybe a few days in elementary school and a few days in AP Physics), and certainly no experience or education in microcontrollers.

For most people like me, who came to the table with little to no experience with electronics and software, the recommendation is usually to take things slow and use free online courses, video lectures, practice homework, and build-it-yourself kits (such as from RadioShack). You might take a few weeks to learn the basics of C programming, then another week or two to study basic circuits, then another week to combine the two to create some “hello world” programs and flash some LED bulbs. However, those that know me well also know that patience is not a virtue I really have. I was going to have NONE of that. So I didn’t. I knew what I wanted my end product to be, so I just made a mad dash to get to it.

How it Started: The Idea I already outline the rough idea at the beginning of this webpage – to be able to wirelessly check the water level of the outside tank – but how I chose to do that was a hard process. I started by breaking it down into parts; the first idea to tackle was the most difficult; how do I measure the water level? I brainstormed different ideas, as well as did online research, and eventually came up with a list of possibilities, only one of which met my needs. Possibles were:

A float. These are insanely common in measuring water level and are used just about everywhere. The Keurig pod coffee makers use magnetic floats in their reservoirs to know if there is enough water left to brew another cup, or if the warning light needs to be lit up. You can also buy commercial units with a float attached to an alarm.

The biggest problem with floats is their lack of accuracy. Most floats act basically as a giant switch – they are either on or off – there is no in-between. Considering how cheap they are, this is absolutely wonderful if you just want to be alerted when the water reaches one specific level that you set, or to have it turn something on, but it is bad if you want more points of accuracy. I believe continuous floats exist, that can output multiple values, but my guess would be that they are primarily for industrial use and probably cost hundreds, if not thousands.

A waterproof sonar sensor pointed downwards towards the bottom of the tank. The sound should ping back off the surface of the water and the sensor should be able to, very accurately, return the current level of the water. Here is an online example: link.

Problem: The accuracy of this sensor would be awesome to have. I could probably know the depth of the water down to a few millimeters. However, a truly waterproof one would likely be pricy, and most importantly, since there are rocks covering the top of the tank, and the tank is really a non-uniform hole in the ground, filled with a pump and various floating debris, it would not only be near impossible to mount the sensor in a place where it can echo off the water, but it is highly likely that debris in the water would mess with it and cause it to give back incorrect readings. A great idea for an actual “water tank”, in which water is relatively stagnant, but probably not good for a tank where water is continuously rushing in and out.

Pressure Differentials: I didn’t do much research here, but what I found made it sound difficult to work with in my scenario, as again, I am not really working with a “tank”, but rather a hole in the ground.
There are a few interesting ideas on this forum page, but not exactly what I ended up doing. Still useful to look at though.
Strings of Resistors. This is a very interesting concept that I did not come up with during brainstorming, but found online on a few different sites (link 1, link 2). It is actually a pretty neat way of measuring water – a bunch of resistors are in series, from a power source, to an analog input on the Arduino (also of course, back to ground). As the string dips below the water, the resistors start shorting out to the return path, and the resistance changes – this provides the Arduino with a variable resistance level that is directly correlational to the depth of the water.

Why didn’t I use it? Well, for one, after coming up with the next idea, this one seemed a bit complicated and convoluted for its purpose. Most importantly though – when electrical current is passed through a metal conductor under water, oxidation takes place, which results in the corrosion of the metal conductor. I could find ways to slow this down, but it would complicate my build further – the idea, without modifications, would likely result in failure in under a year, considering the thickness of the metal wires coming out of most resistors.

The chosen idea: A basic circuit. There is a flat aluminum stick that runs the entire length of my PVC probe and acts as a common anode, supplying a positive 5v (which is generated by the Arduino) to the tank. Also on the PVC pipe/probe are 8 screws, to which wires are attached. Each of these 8 wires goes to a separate analog input pin on the Arduino outside. If a screw is submerged in water, a circuit is created between the analog pin and the 5v of the common anode, so the analog pin reads a rather high voltage/value There is also a pulldown-resistor circuit connected to each of the analog input pins to reduce electrical noise and ensure that when a screw is not submerged in water, the pin only reads “0”. This was really my first or second idea, but I still did my online research just to confirm that I should go with it. After looking at the alternatives, it is the simplest, cheapest, and can provide me with as many accuracy levels as analog pins I have (I chose to just go with 8), although you could theoretical get even more levels by using digital logic circuits to return various levels to the Arduino depending on the level of the tank. My way works well though, and I didn’t over-engineer.

What is important to note, since I mentioned it as a negative for another idea, is that when metal is conducting electricity under water, oxidation occurs and the metal will rather quickly corrode. This method is no exception. However, I took several steps to ensure that it will occur at an extremely slow rate, to the point that it will likely fail because of some other component, such as an AC transformer failing, long before the PVC probe ever fails. The first step was to use a thick (1/8” thick) aluminum flat as the common anode to conduct the positive voltage. There were several different types of flats available, and at least according to my research, aluminum was less likely to corrode than the others, at least at my price point. I found the flat at Home Depot, but I know for a fact that Ace Hardware also carries them (as well as probably most hardware stores). For the conductive screws serving as “nodes”, I used galvanized screws as opposed to plain zinc screws – galvanized screws are more expensive and look duller, but they will resist corrosion longer while still conducting electricity. Again, easy to find at almost any hardware store. Finally, for the wires used, I used “low-voltage landscape lighting cable”, which is cheap for really long lengths (I needed about 80 feet), has a copper conductor mixed with other metal strands (great for electrical conductivity), is really thick (typically 16-gauge, great for slowing down corrosion failure), and is covered with a really thick PVC insulation, which means it can actually be buried underground and will survive exposure to harsh weather. I bought 50 feet of it at Home Depot, and then when I ran out of that, I bought another 100 feet at Ace Hardware (of which I used very little). It was much cheaper to buy at Ace, but your experience may differ. Of course, you can also always buy it online with Amazon.

Procedure: Once I knew what I wanted to do, and how I was going to do it, my first concern was whether it could stand up to constant use. In particular, since I was going to be passing current through a metal conductor underwater, I was worried about how quickly my metal anode and cathodes would corrode, and if the project was even worth attempting, if it was only going to fail in a month or two. I tested this by attaching the heavy gauge landscape cable I bought to the terminals of a 9V battery, stripping their PVC protection off, and then leaving the ends submerged at least an inch or two deep in a see through cup of tap water. I left this set up on a table, for about 3 days, continuously watching to see how quickly the wire corroded. After three days, of an electrical current continuously being passed through the water and the cables, I evaluated the results and took some pictures (below). All in all, it wasn’t terrible. There was visible oxidation after just a few hours of submersion (visible bubbles on the ends of the wires) and after a full three days, bits of both ends of wires had corroded away and the ends had become blackened. At the bottom of the cup was a layer of what looked like blue powder.

Blue Powder Substance

Basic inference and a quick Google search confirmed that it was indeed copper metal that had corroded away (now you know to keep an eye out for blue stains around copper pipes in your house!). This might sound bad, but I was expecting much worse. You see, instead of constantly running a current through my common anode, my plan was always to only run electricity through the water when getting an update for the internal unit, which I planned to make a default of 30 minutes (of course, this is user configurable). It is very unlikely that the water level would ever change drastically in less than 30 minutes, so it makes sense to only get updates that often. In addition, each time it gets an update, it only actually passes current through the tank for about 1-2 seconds, just long enough to get decent analog readings. If you want to see the actual duration, watch one of my videos and watch for the flashing LED on the external box.

Flashing LED

When it lights up, electricity is passing through the tank.

After being convinced that my idea would stand the test of time, I thought it best to design the hardware first. At first, I wanted to have the internal unit have individual LEDs, 8 of them in a vertical column, to light up to represent the water level of the outside tank (eg: if it was full to three levels, then light up the bottom 3 LEDs – this instructables is the closest to what I originally had in mind). I quickly discovered a few issues with this plan though, mainly that as I added more sensors and information to my idea (“I SHOULD HAVE IT RECORD HUMIDITY!!! YEAH FOR FEATURE BLOAT!”) I would have to have a small LCD panel to display the information, and then a user would have to press a button multiple times to display whatever page of information they want to see.

My original idea for display

As I was thinking about these issues, as well as others (see the issues section), I stumbled across some information about a unique and frankly amazing Arduino library that someone had developed. I am not sure if I found it through a Hackaday post or a “related video” section on Youtube when I was looking at videos about Arduinos, but however I found it, I immediately was astounded at the amount of functions the library provided and how it works. The library is called “arduino-tvout” or “TVOut”, and uses just 2 pins to generate a full NTSC or PAL video signal. JUST 2 PINS (in addition to a ground connection)!!! The second I saw the demo and the GoogleCode page, I knew I had to use it. It could display all the information I wanted to, on a single screen, and I could actually create my very own GUI (graphical user interface)!

Sometime around discovering TVOut I also started acquiring a lot more parts. I visited RadioShack and got transistors, resistors, and other various small components (make sure to ask for a student discount if you have a student ID – an extra 10% off all purchases!), shopped at Home Depot and picked up the copper wire, aluminum flat, and weatherproof enclosures, and finally, started purchasing the bulk of my supplies from either Amazon or eBay, eBay being my preferred retail website for electronic components, as they all are pretty much with free shipping and at a fraction of what I would pay at Amazon. The downside to ordering off eBay is that you are usually ordering directly from distributors or manufacturers in China or elsewhere in Asia, so you can expect anywhere between 1-4 weeks for delivery. It is very hard to go from Amazon Prime to having to wait 3 weeks for a package.

Prototyping Around this time, I was also looking for ways I could organize my ideas about the hardware for the project and experiment with configurations without having to prototype everything out on breadboards. Enter “Frtizing”, one of the coolest programs I have every used. Fritzing is a little hard to describe, but is close to analogous to website wireframing software, such as the Pencil Project – Fritzing allows you to easily create graphical representations of electronic circuits and prototype your ideas and save them.

Looking at Fritzing on my laptop

It is really great for a beginner like me, because it does not require that you understand or desire to use circuit schematics/electrical diagrams, only that you can drag and drop components and arrange wires. The downside is that if you really don’t know what you are doing, for example if you don’t know how to work a basic transistor, Fritzing might do you more harm than good. Fritzing doesn’t actually test your circuit in any way, it is just way of representing your circuit in a digital format, so it can’t tell you if your design will work or not. If you are unsure, it is best to go back and forth between Fritzing and an actual breadboard prototype, that way you can be sure your circuit works every step of the way. In my case, I did not do that – I designed it all in Fritzing without doing any testing and then soldered both units in a single night – a very stupid stupid thing to do, but I got incredibly lucky and all of my designs worked (except for a transistor mistake and a broken radio – see the issues section).

Building Shortly before I did all the soldering work, I cut out parts of the box to go inside the house with a rotary tool. I cut out a section of the front to allow room for the LCD panel (which is really just an unmodified car LCD screen I bought on Amazon, designed to be attached to reversing cameras), holes for LED holders and LEDs, holes for the 5 momentary pushbuttons, a hole for the extra RCA video plug, a hole for a sliding switch, and final two holes for the connectors for the Arduino Uno inside the box, one for the USB connector and the other for the barrel power connector.

Cutting the Enclosure

Messy wires!!!

I was able to make super precise cuts with the Black and Decker RTX-B 3-speed rotary tool I picked up from Target; so precise in fact, that many of my components, such as the screen and buttons, stayed in without glue (I did end up using glue, but just as a precaution). Not to sound like a sales pitch, but I really do love the rotary tool and although it does not carry the Dremel name, I think it actually has more power and torque than the Dremels I have seen, and it really is my new favorite tool. Amazon has it for exactly the same price as Target and with free two-day shipping (with Prime).

After cutting my enclosures and doing the majority of the solder work, I started work on the water probe. I used weather-resistant PVC (I am not sure if that is exactly what it is called, but it is grey and available at most hardware stores. It is stronger than the plain white PVC and has a higher rating.) and cut it to about the height of the water tank at the end of the stream. I then also cut the aluminum flat to the same length as the PVC (I used my Black and Decker rotary tool with a cut off dics to cut the aluminum flat). I cut a hole in the bottom of the pole and secured my DS18b20 waterproof temperature sensor in it, and then cut 8 more holes for the 8 analog sensors. I ran galvanized bolts through the 8 holes, using nuts to secure them, and threaded the ends of my 8 analog wires around the nuts, using mostly tension to hold them in place. I also used two bolts to secure the aluminum flat to the PVC pipe. To understand my design easier, I recommend looking at the pictures.

The PVC Probe for the External Unit

The build process for the external unit was much more difficult than for the internal unit. I had to figure out a way to get 14 cables (2 for power and ground to board, 8 for analog signals, 1 for common anode, and 3 for DS18b20 waterproof temperature probe), all of which are a thick 16-guage, organized and fed into a tiny enclosure box. I ended up cutting a large hole in the waterproof sealed enclosure box, then buying a plug/stopper at Ace Hardware made out of some plastic material and using it to plug the hole. I then carefully drilled out the center of the plastic stopper, until it was just a hair larger than all the cables bundled together. Then, I threaded all the 14 cables through the rubber stopper and attached them to multiple 5 position terminal strips inside the enclosure.


Starting to attach all the wires to the terminal strips

More wires attached!

Since the rubber stopper opening was made just barely large enough for the cables, the friction alone held them in place and kept them from getting yanked out of the enclosure, but I also added a very thick layer of Gorilla Glue Epoxy and regular hot glue around the stopper and the cables, to stop any water from leaking into the enclosure. I then connected the terminal strips to the Arduino pins, by way of thinner wire; I believe 22-guage (solid core, not threaded).

Cables attached, seal expoxied, almost done with external unit!

Finally, I wanted to add an indicator light to the external unit that could be visible from far away and would let the user know when there is electrical current running through the water, so I cut a small hole in the enclosure and placed an LED holder and a multicolored LED in it and soldered it to wires that attached to the Arduino. To protect the LED the tiny gap around it from water, I cut the end off of an empty RadioShack solder container, which was see through, and placed it over the LED. I then used plenty of epoxy around it to make sure it would stay attached and not let any moisture seep in. Instead of cutting the solder container, I could have also used the clear bubble lid that comes on those capsules you can get out of quarter vending machines that dispense little toys and stickers, but I couldn’t find any of those vending machines when I needed the part, so I improvised even further with the solder container.

Once everything was all wired up, it was time to start work on…

The Programming Like I said before, I have no real programming experience, so take the information I share here with that consideration. That being said, I ended up writing over a thousand lines of codes, which compile without errors, and I currently have no real bugs to report.

I really only used three programs to help my write my code. The first is the official Arduino IDE, which really provides very little in terms of an interface. There is no autocomplete or suggestions for functions, no collapsing of sections, and no debugging. However, it super easy to use, since it just requires one click to compile and upload to your Arduino board, and it is lightweight, so you can code without distractions. I would highly recommend starting out with the official Arduino IDE if you are new to programming, as its lack of features actually make it better for a beginner; there is let to get confused about and its limitations force you to learn and understand the code you are producing. If you are an experienced programmer, you can use pretty much any IDE you care to use (the most popular usually being Eclipse). I haven’t really gotten into alternative IDEs yet, but I have a feeling I might at some point, as Atmel Studio looks especially promising, considering there are ways to debug and use breakpoints with it.

Another piece of software I used was Paint.net. If you know what Photoshop is and what Microsoft Paint is, think of Paint.net as a happy medium between them. It is way more advanced than Microsoft Paint, and offers things like cloning, layers, transparency, and more, but stops short of the more advanced features of Photoshop. By not being quite as advanced as Photoshop, it is able to maintain a very small installation (currently uses a 3.5 MB installer), and it runs blazing fast even on the slowest of computers. Normally you would never need something like this for an Arduino or electronics project, but to prototype what my GUI would look like on the LCD screen using the TVout library, I needed a program I could use very easily and quickly to manipulate images on a pixel-by-pixel level. Paint.net allowed me to do just that and was integral in allowing me to create different GUI designs and even an animation.

Finally, I used one of my all time favorite programs, and it has been one of my favorites since long before I started work on this project, Notepad++. Sorry to keep using comparisons, but if you are familiar with Microsoft Notepad and Microsoft Visual Studio or Eclipse (or even Microsoft Word), Notepad++ is a somewhere in-between. It has the minimalist design of notepad and the small footprint and fast runtime, but it also has many of the features that make a full-fledged IDE so great, such as syntax highlighting and collapsing of sections, spellcheck (by way of a free plugin), auto-completion (available, not forced), multi-tabbed to switch between documents, a document map, zooming, and many more features. What I liked to do was to work on my Arduino sketches (.ino files) in the Arduino IDE, but also have them open in Notepad++. Then, if I wanted to navigate my code once it was really long, it was easier to use Notepad++ to examine it in great detail and look for bugs.

Using these three software programs, I was able to create two Arduino programs (sketches), one that runs on the unit outside, and the other that runs on the inside unit and displays the user interface. It took over a thousand lines of code, many hours of debugging, and many cups of coffee, but in the end the units work amazing and perform much better than I ever expected them to. They have been running for days now without any software bugs popping up, and my hope is that they will continue to run for quite some time. However, the path to this point was not as easy it might sound right now. That is because I have not yet discussed…

The Issues!!!

The first issue I ran into was that my LCD screen flickered and was making weird patterns. Not only that, but the entire program was acting strange and values were being reported and sent at values outside of what they should have been. Sometimes the board would just crash and nothing would work.

The fix: This was a memory issue, one that I would have been more aware of to start with if I had experience with microcontrollers. Turns out, the Arduino Uno, the board I was using, only has 2 kilobytes of SRAM, which is insanely small to today’s standards, or even to yesterday’s standards, considering that the original Gameboy released in 1989 had 4 times that amount of both general SRAM and dedicated to video. When I started writing my code and using the Arduino TVOut library I followed their suggestion and used a default resolution of 128x96. Can anyone see a problem with this? I should have. If you think about it for a second, since the TVout library outputs only black or white, which can be really thought of as a binary value, it would have to create a frame buffer of 128x96 pixels, with each pixel occupying a single bit of memory. If you use simple math and multiply 128 by 96, you get 12,288, which is how many bits of SRAM the code needs (1.5 kilobytes), just to display anything on the screen. Considering that 2kB is only equal to 16384 bits, using a resolution of 128x96 was leaving me with only 4,096 bits of SRAM, or half a kilobyte, to work with. Throw in some more code and the fact that I was using the TVout library to load an entire bitmap image into memory as a bootscreen, and it was no surprise that I was maxing out my memory and inducing a buffer overflow. By reducing my resolution to 120x54, I started using only 6,480 bits for the framebuffer, cutting my SRAM usage roughly in half. This was a huge amount of saved memory and I have had plenty of room for my bitmap bootscreen, all my large variables and arrays, and all the graphic elements.
The flickering / buffer overflow returned once more, when I completely removed delays that limited the speed at which my program ran. My guess is that if it runs too quickly, something happens with the TVOut library and how it generates frames and uses SRAM. I found that a TV.delay(10), which is 10 milliseconds, is enough to stop the SRAM from getting chewed up and short enough to not impact any functionality of the devices.

I couldn’t seem to get the TVout library to completely fill the LCD screen.

When I had to reduce the active resolution of my project to save SRAM I inadvertently solved my problem. I just sat there and played around with different low resolutions until I found one that occupied very little SRAM but also filled most of my screen. That ended up being 120x54. Keep in mind, if you are playing around with values, that the horizontal resolution (in this case 120 pixels wide) has to be evenly divisible by 8.

Everything was sideways!!! The TVOut library is designed to output text and graphics to a screen that is in a landscape orientation, like all TVs are and computer screens – this means that the screen is wider than it is tall. However, because I’m a dolt and didn’t check any of this out before soldering all my connections, cutting my enclosure, and gluing everything permanently in place, my screen ended up being in portrait orientation, which mean that using the TVout library without any modifications resulted in text being drawn sideways (if you were to tilt your head 90 degrees clockwise it would become readable, see the below pictures).

Really sideways, not upside down

Now the letters are correct, but it is writing them sideways

Success!!! YES!!!
The easiest solution would have also been the hardest. If I had more experience before starting this project, I’m sure I could have dug around in the library, found where it creates the frame buffer, and then modified the library to generate a buffer that was rotated 90 degrees counter-clockwise. Then I could have used all the TVOut functions as normal and it would have drawn the screen just as how I wanted it to. Instead, I did things the difficult way. First, I actually made my own font. Well, not quite true. I took the 6x8 font that they provided and rotated every character 90 degrees counter clockwise. My dad helped me out by writing a tiny Python script that rotated the binary chunks, or else I would have had to do it by hand (which I actually did for a different font). This still doesn’t fix the problem, as it would now write each character in the correct orientation, but would write the words in the wrong orientation. See the above pictures for a better look. The solution than become more convoluted – to write words on the screen and have them show up readable, I had to individually print out each character at different pixel locations, sort of like if every time you wanted to type on your computer you had to use your mouse to set the cursor in a different place (for every freakin’ character!).
Printing "Sending", by printing each character individually
This also made printing numbers very difficult, because I had to write snippets of code to parse variables, find out of a variable was holding a single, double, or triple digit number, and then parse it into different place values depending on its length, and then draw each place value in a different pixel location

The LCD screen could not be powered off the 5v pin from the Arduino.

This isn’t really a bug, and it makes sense. The LCD screen is bright and obviously needs a decent amount of power. Although 5v is enough voltage, current is what really matters, and an Arduino board only receiving a few hundred milliamps cannot source enough amps to power the LCD screen. To fix this, you can either solder connections from the VCC on the USB connector before it hits the regulator, or you can do as I did, and solder wires to the barrel connector on the UNO and solder the other ends to the power supply for the LCD screen. I then started using a 2 amp AC-DC adapter with the barrel connector, to ensure that both the board and the screen received enough current. It worked after that.

When I got to coding for the nRF24L01+ radios, I couldn’t get anything to work. I even tried a program that I had successfully used before with slightly different hardware, and it still didn’t work.

This was an annoying one. I spent hours checking my code, then a few more hours tracing every circuit on my PCBs with a multimeter and then even more time just trying and retrying to get them to work. Eventually I resorted to purchasing another Arduino to test with, so I could at least see which of the two units was not using the radio correctly. Once it arrived, I tested and discovered that the external unit was not responding to any messages. I traced its connections again and resoldered some connections, but it still didn’t work. Finally, I realized that this might actually not be my fault, and I swapped out the nRF24L01+ radio on that unit with an extra I had lying around (I said they were cheap), and all of a sudden, the exact same code that wasn’t working days ago, was working just fine. Turns out, when you pay a dollar for something that competitors charge roughly 20 for, you are bound to get a dud once in a while. It really was just a bad radio unit. I never really did any further testing to see why that particular radio stopped working, nor do I really care to find out since I purchased two new ones to replenish my supply for less than a cup of coffee. Luckily, instead of soldering the radio directly to the board, I had soldered jumper cables with female ends onto the board and then just placed the pins of the radio into the female ends. This meant that to replace the radio I didn’t have to desolder anything, I just had to pull the pins out of jumper cables and then slide the jumper cables over the pins of the new radio. What I really want to emphasize though, through telling this experience, is that sometimes things are really just broken and it is not your fault, so don’t just look at your own work, look at every part of the puzzle.
Sometimes the Arduino IDE will not recognize that a new library has been added and linked to your code, unless you do "Import library...", under the "Sketch" option on the top menu bar

I’m not sure if this has been addressed somewhere online, but it is just a strange, once in every 200 times, bug I have encountered.

Another nRF24L01+ radio issue: radio.read gets nothing. At one point, I was trying to figure out why using the radio.read function out of the nRF24L01+ radio library was getting stuck and returning nothing, when it should have been returning a payload that the outside unit was sending.

The documentation for the library says to call “startListening” before attempting a radio read. I did that, however what I found out is that if it is called multiple times, in rapid succession (such as during a while or for loop that is true), it will actually interfere with the radio.read and you will get nothing read. My guess is that “startListening” actually changes something on a hardware level on the radio and there needs to be some sort of delay between the hardware change and the radio attempting a read. Making sure that “startListening” was only called once and not repetitively/rapidly, fixed my problem.

After changing the datarate of the nRF24L01+ radios, nothing would work!

The documentation for the nRF24L01+ says that if you need a longer distance range between two radios, you can decrease the data rate of the transmissions. I did this by using “radio.setDataRate(RF24_250KBPS);”, but after doing so, nothing would work. I figured out, and I am not sure if this is addressed in the documentation, that you cannot set the data rate of the radio before using “radio.begin”. This should be kind of intuitive as to why, but I missed it the first time around, so worth warning about.

My biggest and most challenging issue to debug and fix: random lockup every 60 minutes. This was such a strange bug; every 60 minutes, pretty much on the dot, the GUI screen would freeze and say it had been 29 minutes since last updated (at that time, my code was set with 30 minute intervals between reads) and it would stay that way indefinitely. Even stranger, none of the buttons on the box responded to presses, even though there was no while or for loops that could cause their keypresses not to be read. The board couldn’t have actually crashed, or else nothing would be on the screen; for there to be an image on the screen, the TVOut library has to be actively functioning.

I initially thought this was a timing issue, since it was occurring every 60 minutes like clockwork. I tried changing the interval between reads from 30 minutes to 20 minutes, but then it just started failing every 3 read instead of every 2nd. Obviously, this was not the issue.
I next jumped to low RAM as a possibility. As I mentioned before, the SRAM of the Arduino UNO is extremely limited and it has caused trouble before. However, I was able to determine that this was not the case, as it had plenty of free SRAM.
Finally, I remembered that the TVOut library is insanely dependent on interrupts. This is why you cannot use serial at the same time as using TVOut. I started looking through the different libraries I was using and discovered that the library used by the Real Time Clock and the Wire library (which is necessary to talk to the RTC) also used interrupts. I didn’t want to completely stop using the library, so what I did was move the reading of the RTC out of the very fast looping main loop and into a subroutine, that is only ran once a user presses a button to display the current calendar date and time. Since I could no longer check unixtime using the RTC in the main loop, I replaced my uses of unixtime with TV.millis, which returns the number of milliseconds since the board has started running. Unfortunately, TVOut’s dependency on interrupts also screws with the boards timing, and so the TV.millis returns a rather inaccurate number. I was able to compensate by finding a factor I could multiple the number by to get a more accurate representation of the number of milliseconds the board has been running. Anyways, moving the reads of the RTC and the Wire.begin outside of the main loop fixed the problem. It has now been running for multiple days on end and has not frozen up once.

There is one issue that I am sad to say has not yet quite been resolved: humidity. As you should be able to see from the pictures, I build the external enclosure in a sealed electrical box from HomeDepot. It has a gasket that runs between the lid and the box, and I screwed the lid on tight enough to seal it. However, every day, since I first plugged it in outside, the humidity has been steadily going up on the readings on the inside unit. It started somewhere around 40 I think, and after about a week and half, had reading 90%. I freaked out, as the alarm kept going off and I had to keep raising the alert level so the high humidity wouldn’t trigger it, but most importantly, I was worried that the high humidity might damage the Arduino board (I’ll remind you that the humidity is measured from inside the enclosure, not the air outside). I live in Seattle Washington, so one fact that I cannot change is that it is very damp here and we get frequent rain (especially right now, when I am having the humidity issue). My current temporary fix was to go outside, put the entire enclosure in a ziplock bag with a bunch of silica gel packets (that absorb moisture in the air), then seal the bag with rubber bands, then cover the bag and the enclosure with saran wrap, and then secure the saran wrap with more rubber bands. After doing that, the humidity readings stabilized, and then dropped a few percentages, but they are still way too high, hovering around 90%. Once it stops raining (which might take a while here), I’ll unwrap it, maybe let it dry in the sun or take a blow drier out on low heat, and add some new silica gel packs and a better seal.

Things I could have done differently:

Instead of using the TVOut library, I could have used an LCD that was designed to be used with Arduino (such as one mounted on a shield), however, those use more pins and I am actually glad I went with TVOut since I learned more using it
I could have used a different Arduino with more RAM, such as the MEGA, or I could have used a completely different kind of microcontroller platform, such as the Teensy board, which actually has a much more powerful 32 bit ARM processor for about the same price, or I could have gone with a Raspberry Pi. Using a more powerful board would have meant I could have used a higher resolution with the TVOut library, as well as store more information in RAM. As it is, I actually really look how the display looks with the low resolution setting, as it has a nice retro feel and is easily visible from far away. For future projects I might try different boards, but I am glad I went with the UNO for the internal unit.

Thanks If you read to here, I feel sorry for you. That was a lot to read and I really went overboard in writing this. I thank you though, for taking the time to read it. If you didn’t read all the way to here, and just skipped to the bottom of the page, I can’t say I blame you.

Finally, I want to say thanks to my parents for raising me and encouraging me to do stuff like this, and especially my dad for sparking the original idea, helping me debug a few things, and letting me use his tools and workroom.

Virtual Tour Site: TheOldLW.com

Joshua Tzucker — Mon, 12 Sep 2011 07:00:00 GMT

For my senior project in high school, I took close to 10,000 photos of the inside of the old Lake Washington High School, before it was demolished, stitched them together into panoramic images, and joined the panoramic files into a giant interactive tour. I al...

Joshua Tzucker's Site

What Makes Svelte So Great - Features and Comparisons

Intro

Performance

Simplicity

Bi-Directional Variable Bindings

State, State, State (Did I Mention State Yet?) and Effectful Code

SFCs are SFC (so-frequently-cool) and Template Syntax vs JSX

Shorthand Syntax Everywhere

Batteries Included

Incredible Documentation and Tooling

It’s Fun to Use, Simple, and You’ll Write Less Code with Less Bugs

Nothing’s Perfect

Reactivity in Svelte Requires a Mental Model Shift

The talent pool is smaller

The Svelte ecosystem isn’t as “rich” as React’s

Conclusion / Wrap-Up

Further Reading

Thanks for Reading 👋

Shell Heredocs - for Fun and Profit, in Bash and Beyond

Intro

The Basics of heredocs

Fun and Productive Use Cases for Heredocs

Using heredocs to escape and store special text

Using heredocs to store and execute non-shell code

Taking Advantage of More Libraries From Your Terminal

Using heredocs as real files and virtual files

Here Strings

Wrap Up

Writing Slack Messages with Markdown (Multiple Options)

TLDR

Background

Wait, I thought Slack Supported Pasting Markdown?

The Solution / Shortcut

Converting Markdown to HTML Clipboard Content

Credit / Acknowledgments / Wrap Up

We Should Care More About Higher Level Documentation

Intro

Thinking of Documentation Levels as a Funnel

Examples of Higher-Level Documentation

Why More Effort Should be Put On The Early Part of the Funnel

Jobs Have Changed

Your Documentation is a Form of Marketing

Source Code Is More Accessible Than Ever

Why Are High-Level Docs Often Neglected?

Writing at All Levels is Tough

Exploiting The Rise of Tech Blogs and Community Posts

Don’t Overdo It

Throwing Stones

Pytest Productivity - Tips I Wish I Had Known Sooner

Intro

Pytest Real-Time Usage

You Can Stream Live Logs

You Can See and Interact With Exceptions As They Happen

You Can Debug at Any Level

Support for Flaky Tests

You Can Get Granular With Running Tests

Magical Fixtures

Conclusion

How to Bulk Delete Album or Track Scrobbles on Last.fm

Bulk Deleting Last.fm Scrobbles

How to Bulk Delete All Scrobbles of a Given Album in Last.fm

How to Delete All Scrobbles of a Specific Track in Last.fm

The Easier Solution for Tidal Advertisement Scrobbles

The Harder Solution

YouTube History Bulk Deletion Script

Nook Simple Touch - Factory Reset & Bypass Network Error

Solution(s)

Special Boot Mode

The 8 Failed Boots Method

Support Me and My Content

Monetary Support

Thank You

Writing for DigitalOcean - Gatsby Tutorial Series

Canvas Hit Detection Methods - With or Without Tolerances

Pure HTML Approach

Canvas

Canvas - Getting Coordinates from a Mouse Event

Demo - Without Padding

Canvas Hit Detection - With a Tolerance Allowance

`isPointInPath` vs `isPointInStroke`