I designed something that shows you what your friends have been hacking on, in real-time! Here's a blog of the process of ideation, graphic/interface design in Sketch, and the engineering of the Electron app. This was developed in Feb 2018, and everything is open-source (client, server).
Spotify has a great feature where you can see what your friends have been listening to. My simple idea was, what if I could see what my mates have been hacking on? Programming is a very interesting art form to consider, because while being the basis of our globally connected real-time communications, hacker culture is still embedded in text and IRC. For example, while live video and collaborative docs on Google have been the norm for years now (where's my tech-to-dog-year calculator), VS Code is only just on the cusp of such features. I wanted to experiment with the concept, while also developing my skills in building a polished app from idea-to-product. The process:
- Ideating + Mockups (Sketch)
- Researching technical feasibility
- Building the Product
Ideation and Mockups
Inspiration
The first part of the process was searching out inspiration for what I wanted to build. I compiled what could be called a moodboard, but was really more for interface design inspiration than anything else.
Inspiration from the original Spotify Who's Listening design-
The colour scheme of VS Code-
The minimalist floating window of the Helium app-
The design of other popular social networks with micro/realtime content-
Ideation
From the get-go, I wanted this project to incorporate an actually aesthetically-pleasing design. I generated a colour scheme.
What did I actually want to show as part of the feed? Did I want hackers to share their status in text (like microblogging for projects)? It would be an inadequate replacement for communicating. I contemplated what if there was a dropdown suite of statuses you could select (installing dependencies, compiling, etc), but that too didn't seem like valuable information. Finally, I decided on applying Occalm's razor to my instinct - all I want to see is the latest of who's hacking on what - a simple name would suffice and a link if the project is open-source.
The next question was - how much user involvement should we need in sharing a status?
What is the best way of indicating status? What is the most relevant information as to when it was happening? And how do we show the user their external-facing status?
And finally, if this were to be a product, we would need a logo. The most important aspect of this is communicating the metaphor of what the product is about. I decided to make a variation on the typical "messages" app icon, incorporated with some hacker symbolism. I didn't immediately arrive at this, below is some experimentation on different designs:
Researching Technical Feasibility
At this point, the problem was - "how do you detect the status of a hacker working on different code projects, in realtime?". Realtime in this domain was on the scale of 1-10 minutes.
I considered multiple approaches:
- a simple
hacking-nowcommand that you run in the shell to inform the client. problem: how do you detect when hacking stops? requires manual input too. - integrating into the IDE/editor of choice. problem: scale is too big.
- integrating with a GitHub API for tracking pushes. problem: only shows realtime hacking in a retrospective manner. Also does not work for non-public or not-yet-public projects.
Ideation here led me to a novel solution that covered 80% of use cases (something Apple espouses in their design principles: see this interesting prez) - I could track the git repositories across the file system for changes. There were a couple iterations on how this could be achieved:
I wanted this to be a cross-platform app, so file system watching had to work on macOS, Linux and Windows. All have different interfaces with different constraints. macOS and Linux have limits on the number of open file descriptors. Both of which can be changed by something like ulimit, although macOS has additional complexity stemming from different versions treating ulimits with varying permanency.
So, with a conservative estimate of 8K potential file watchers for my app - would I be able to watch the entire system for .git repository changes? I considered using various frameworks that get around the complexity of manually working around these performance concerns, including Facebook's Watchman. Even considering, there was a necessity to give them admin privileges, their cross-platform compatibility was not guaranteed, and bundling and deploying them very heavy. Although this was a potential solution, I intuited there was a more suitable approach.
And so I experimented with writing my own file system watcher for git repositories, git-monitor. I wrote it in Go for the maximum performance possible and possibility of easy parallelism with Goroutines. Go also cross-compiles quite easily as a single binary, which was perfectly minimalist. The first design simply ran a filesystem scan of all .git repositories every 1 minute and then diffed their timestamps of the file tree - this did not seem encouraging on battery life, especially considering my own Macbook was losing battery to heavy compilation tasks.
But then the brainwave hit - I could achieve the realtime property without the performance hit, we just had to be more specific with what we watch. We could scan the filesystem on the upper limit of our definition of realtime (every 10 minutes) for Git repositories, and detect the realtime editing by polling the ones we have detected on a smaller timescale (once every minute). This was made even better by the fact that Git already has a highly-performant algorithm for detecting if the file tree has changed, which we could make use of through running git ls-files --modified.
Building the Product
App
I used Electron to build the app, with React for the interface and MobX for state management.
I learnt about incorporating motion design with CSS3. For example, look at the smooth transitions when we receive an update about a user's status:
I got hackers onboarded simply through signing up with GitHub.
Finally, some useful tidbits of code. Here's one which shows a Spotify-style timestamp:
function spotifyStyleTime(time) {
let ago = moment.duration(moment().diff(time));
let times = [ago.years(), ago.weeks(), ago.days(), ago.hours(), ago.minutes()]
let fmts = ['y', 'w', 'd', 'h', 'm']
let biggestTime;
let i = _.findIndex(times, x => x != 0)
return `${times[i]}${fmts[i]}`;
}
This selects the correct Go binary for the platform:
function getGOARCH() {
switch(require('arch')()) {
case 'x86':
return '386';
case 'x64':
return 'amd64';
}
throw new Error("couldnt get arch")
}
function getGOOS() {
switch(require('os').type()) {
case 'Windows_NT':
return 'windows';
case 'Darwin':
return 'darwin';
case 'Linux':
return 'linux';
}
throw new Error("couldnt get os")
}
// ...
let arch = getGOARCH();
let os = getGOOS();
let cmdPath = path.join(getAppPath(), `/vendor/gitmon-${os}-${arch}`);
getAppPath here is important, because Electron apps are bundled in a bespoke archive format called ASAR. It has random access support and indexing, which makes it a lot easier for cross-platform file bundling and distribution. However, to run executables, you need to access the unpacked directory:
const electron = require('electron').remote;
export function getAppPath() {
let appPath = electron.app.getAppPath()
if(process.env.ELECTRON_ENV == 'development') {
return appPath;
} else {
return appPath.replace('app.asar', 'app.asar.unpacked')
}
}
It is also important to highlight here that you need to access the Electron main process for some API's. Electron's architecture incorporates:
- a main process, which provides a JS interface to native system elements such as menubars and notifications.
- a renderer process, which is an instance of Chrome's rendering engine, Chromium
Packaging and distribution
Packaging was the most satisfying aspect of this process. Seeing this really makes my eyes sparkle:
I advise using electron-builder for this purpose. There is unfortunately very scant documentation. My build structure had a src/ directory which was processed using Webpack into bundle.js, and otherwise there was the Electron main.js and renderer.js that must be included respectively. This build config in package.json worked for me:
{
"build": {
"appId": "co.liamz.whoshacking",
"productName": "Who's Hacking",
"copyright": "Copyright © 2018 Liam Zebedee Edwards-Playne",
"files": [
"static/",
"vendor/",
"build/",
"static/",
"index.html",
"main.js",
"renderer.js"
],
"asarUnpack": [
"vendor/"
]
},
// ...
}
Backend
The backend was interesting, as I'd never built something so "realtime" before. My stack was Node, Express, Passport (for OAuth), RethinkDB (for the database), and Socket.io for the realtime comms. I also used Chai for unit testing, which proved invaluable in the development process.
Middleware. I learnt about the middleware pattern while developing the Express and Socket.IO servers. Middleware is a good abstraction for microservices: usually you want to share a context of data, such as cookies/request/response/the user, while also keeping the request handling logic loosely coupled. My first intuition was that setting heaps of keys on a JSON object could get wildly out of hand, but realistically then you're putting too much into one design - do one thing, and do it well.
Authentication. One big headache was initially authenticating the user. Originally, I wanted the cookie to be set from the OAuth flow, and later used when we are doing the WebSocket handshake. However, the Express server and the Websocket server were started with different middleware for handling sessions (nomenclature for cookies, here). I spent a considerable duration trying to mediate the two, however in the end, the principle of least surprise took me to implement myself something simpler. We generate a token for the user upon successful authentication, and the client performs a GET (which sends the cookie and thus sets the req.user correctly from middleware) to an endpoint which returns it. Then, when connecting via the WebSocket, my own middleware is used to authenticate the user.
RethinkDB, changefeeds and WebSockets. This was the easiest aspect of development. Communications between the client and server were as simple as socket.on('currentStatus', () => { ... }). For the server-side, RethinkDB has a brilliant features called changefeeds, which are watches on database rows that are brilliantly simple to code for. Here's an example:
createConnPromise()
.then(conn => {
r.table('users')
.without(PRIVATE_FIELDS)
.map(user => {
return user.merge({
statuses: user('statuses').orderBy(r.desc('time')).limit(1)
})
})
.changes()
.run(conn, (err, cursor) => {
cursor.each((err, user) => {
if(user.new_val) {
socket.emit('status updated', { statusEvent: user.new_val })
}
})
})
})
This code firstly creates the database connection, creates a query on the users table that returns the most recent status for each user, and then creates a watch on it. We callback and send the new status down the socket for every change we receive. The abstractions are impeccably well-designed.
Conclusion
This was a project I developed in the span of a week or two. As you can see, there were a lot of little problems I had to solve, from the executable, to authentication, that arose from touching many different areas of the stack (HTTP, Sockets, Electron main/renderer process, etc.).