Who is talking?


Frameworks and Why (Clojure) Programmers Need Them

about 2 years ago | Daniel Higginbotham: Flying Machine Studios

It seems like there's a strong aversion to using frameworks in the Clojure community. Other languages might need frameworks, but not ours! Libraries all the way, baby! This attitude did not develop without reason. Many of us came to Clojure after getting burned on magical frameworks like Rails, where we ended up spending an inordinate amount of time coming up with hacks for the framework's shortcomings. Another "problem" is that Clojure tools like Luminus and the top-rate web dev libraries it bundles provide such a productive experience that frameworks seem superfluous. Be that as it may, I'm going to make the case for why the community's dominant view of frameworks needs revision. Frameworks are useful. To convince you, I'll start by explaining what a framework is. I have yet to read a definition of framework that satisfies me, and I think some of the hate directed at them stems from a lack of clarity about what exactly they are. Are they just glorified libraries? Do they have to be magical? Is there some law decreeing that they have to be more trouble than they're worth? All this and more shall be revealed. I think the utility of frameworks will become evident by describing the purpose they serve and how they achieve that purpose. The description will also clarify what makes a good framework and explain why some frameworks end up hurting us. My hope is that you'll find this discussion interesting and satisfying, and that it will give you a new, useful perspective not just on frameworks but on programming in general. Even if you still don't want to use a framework after you finish reading, I hope you'll have a better understanding of the problems frameworks are meant to solve and that this will help you design applications better. Frameworks have second-order benefits, and I'll cover those too. They make it possible for an ecosystem of reusable components to exist. They make programming fun. They make it easier for beginners to make stuff. Last, I'll cover some ways that I think Clojure is uniquely suited to creating kick-ass frameworks. (By the way: I've written this post because I'm building a Clojure framework! So yeah this is totally my Jedi mind trick to prime you to use my framework. The framework's not released yet, but I've used it to build Grateful Place, a community for people who are into cultivating gratitude, compassion, generosity, and other positive practices. Just as learning Clojure makes you a better programmer, learning to approach each day with compassion, curiosity, kindness, and gratitude will make you a more joyful person. If you want to brighten your day and mine, please join!) What is a Framework? A framework is a set of libraries that: Manages the complexity of coordinating the resources needed to write an application... by providing abstractions for those resources... and systems for communicating between those resources... within an environment... so that programmers can focus on writing the business logic that's specific to their product I'll elaborate on each of these points using examples from Rails and from the ultimate framework: the operating system. You might wonder, how is an OS a framework? When you look at the list of framework responsibilities, you'll notice that the OS handles all of them, and it handles them exceedingly well. Briefly: an OS provides virtual abstractions for hardware resources so that programmers don't have to focus on the details of, say, pushing bytes onto some particular disk or managing CPU scheduling. It also provides the conventions of a hierarchical filesystem with an addressing system consisting of names separated by forward slashes, and these conventions provide one way for resources to communicate with each other (Process A can write to /foo/bar while Process B reads from it) - if every programmer came up with her own bespoke addressing system, it would be a disaster. The OS handles this for us so we can focus on application-specific tasks. Because operating systems are such successful frameworks we'll look at a few of their features in some detail so that we can get a better understanding of what good framework design looks like. Coordinating Resources Resources are the "materials" used by programs to do their work, and can be divided into four categories: storage, computation, communication, and interfaces. Examples of storage include files, databases, and caches. Computation examples include processes, threads, actors, background jobs, and core.async processes. For communication there are HTTP requests, message queues, and event buses. Interfaces typically include keyboard and mouse, plus screens and the systems used to display stuff on them: gui toolkits, browsers and the DOM, etc. Specialized resources are built on top of more general-purpose resources. (Some refer to these specialized resources as services or components.) We start with hardware and build virtual resources on top. With storage, the OS starts with disks and memory and creates the filesystem as a virtual storage resource on top. Databases like Postgres use the filesystem to create another virtual storage resource to handle use cases not met by the filesystem. Datomic uses other databases like Cassandra or DynamoDB as its storage layer. Browsers create their own virtual environments and introduce new resources like local storage and cookies. For computation, the OS introduces processes and threads as virtual resources representing and organizing program execution. Erlang creates an environment with a process model that's dramatically different from the underlying OS's. Same deal with Clojure's core.async, which introduces the communicating sequential processes computation model. It's a virtual model defined by Clojure macros, "compiled" to core clojure, then compiled to JVM bytecode (or JavaScript!), which then has to be executed by operating system processes. Interfaces follow the same pattern: on the visual display side, the OS paints to monitors, applications paint to their own virtual canvas, browsers are applications which introduce their own resources (the DOM and <canvas>), and React introduces a virtual DOM. Emacs is an operating system on top of the operating system, and it provides windows and frames. Resources manage their own entities: in a database, entities could include tables, rows, triggers, and sequences. Filesystem entities include directories and files. A GUI manages windows, menu bars, and other components. (I realize that this description of resource is not the kind of airtight, axiomatic, comprehensive description that programmers like. One shortcoming is that the boundary between resource and application is pretty thin: Postgres is an application in its own right, but from the perspective of a Rails app it's a resource. Still, hopefully my use of resource is clear enough that you nevertheless understand what the f I'm talking about when I talk about resources.) Coordinating these resources is inherently complex. Hell, coordinating anything is complex. I still remember the first time I got smacked in the face with a baseball in little league thanks to a lack of coordination. There was also a time period where I, as a child, took tae kwon do classes and frequently ended up sitting with my back against the wall with my eyes closed in pain because a) my mom for some reason refused to buy me an athletic cup and b) I did not possess the coordination to otherwise protect myself during sparring. When building a product, you have to decide how to create, validate, secure, and dispose of resource entities; how to convey entities from one resource to another; and how to deal with issues like timing (race conditions) and failure handling that arise whenever resources interact, all without getting hit in the face. Rails, for instance, was designed to coordinate browsers, HTTP servers, and databases. It had to convey user input to a database, and also retrieve and render database records for display by the user interface, via HTTP requests and responses. There is no obvious or objectively correct way to coordinate these resources. In Rails, HTTP requests would get dispatched to a Controller, which was responsible for interacting with a database and making data available to a View, which would render HTML that could be sent back to the browser. You don't have to coordinate web app resources using the Model/View/Controller (MVC) approach Rails uses, but you do have to coordinate these resources somehow. These decisions involve making tradeoffs and imposing constraints to achieve a balance of extensibility (creating a system generic enough for new resources to participate) and power (allowing the system to fully exploit the unique features of a specific resource). This is a very difficult task even for experienced developers, and the choices you make could have negative repercussions that aren't apparent until you're heavily invested in them. With Rails, for instance, ActiveRecord (AR) provided a good generic abstraction for databases, but early on it was very easy to produce extremely inefficient SQL, and sometimes very difficult to produce efficient SQL. You'd often have to hand-write SQL, eliminating some of the benefits of using AR in the first place. For complete beginners, the task of making these tradeoffs is impossible because doing so requires experience. Beginners won't even know that it's necessary to make these decisions. At the same time, more experienced developers would prefer to spend their time and energy solving more important problems. Frameworks make these decisions for us, allowing us to focus on business logic, and they do so by introducing communication systems and abstractions. Resource Abstractions Our software interacts with resources via their abstractions. I think of abstractions as: the data structures used to represent a resource the set of messages that a resource responds to the mechanisms the resource uses to call your application's code (Abstraction might be a terrible word to use here. Every developer over three years old has their own definition, and if mine doesn't correspond to yours just cut me a little slack and run with it :) Rails exposes a database resource that your application code interacts with via the ActiveRecord abstraction. Tables correspond to classes, and rows to objects of that class. This a choice with tradeoffs - rows could have been represented as Ruby hashes (a primitive akin to a JSON object), which might have made them more portable while making it more difficult to concisely express database operations like save and destroy. The abstraction also responds to find, create, update, and destroy. It calls your application's code via lifecycle callback methods like before_validation. Frameworks add value by identifying these lifecycles and providing interfaces for them when they're absent from the underlying resource. You already know this, but it bears saying: abstractions let us code at a higher level. Framework abstractions handle the concerns that are specific to resource management, letting us focus on building products. Designed well, they enable loose coupling. Nothing exemplifies this better than the massively successful file abstraction that the UNIX framework introduced. We're going to look at in detail because it embodies design wisdom that can help us understand what makes a good framework. The core file functions are open, read, write, and close. Files are represented as sequential streams of bytes, which is just as much a choice as ActiveRecord's choice to use Ruby objects. Within processes, open files are represented as file descriptors, which are usually a small integer. The open function takes a path and returns a file descriptor, and read, write, and close take a file descriptor as an argument to do their work. Now here's the amazing magical kicker: file doesn't have to mean file on disk. Just as Rails implements the ActiveRecord abstraction for MySQL and Postgres, the OS implements the file abstraction for pipes, terminals, and other resources, meaning that your programs can write to them using the same system calls as you'd use to write files to disk - indeed, from your program's standpoint, all it knows is that it's writing to a file; it doesn't know that the "file" that a file descriptor refers to might actually be a pipe. Exercise for the reader: write a couple paragraphs explaining precisely the design choices that enable this degree of loose coupling. How can these choices help us in evaluating and designing frameworks? This design is a huge part of UNIX's famed simplicity. It's what lets us run this in a shell: # list files in the current directory and perform a word count on the output ls | wc The shell interprets this by launching an ls process. Normally, when a process is launched it creates three file descriptors (which, remember, represent open files): 0 for STDIN, 1 for STDOUT, and 2 for STDERR, and the shell sets each file descriptor to refer to your terminal (terminals can be files!! what!?!?). Your shell sees the pipe, |, and sets ls's STDOUT to the pipe's STDIN, and the pipe's STDOUT to wc's STDIN. The pipe links processes' file descriptors, while the processes get to read and write "files" without having to know what's actually on the other end. No joke, every time I think of this I get a little excited tingle at the base of my spine because I am a: This is why file I/O is referred to as the universal I/O model. I'll have more to say about this in the next section, but I share it here to illustrate how much more powerful your programming environment can be if you find the right abstractions. The file I/O model still dominates decades after its introduction, making our lives easier without our even having to understand how it actually works. The canonical first exercise any beginner programmer performs is to write a program that prints out, Wassup, homies?. This program makes use of the file model, but the beginner doesn't have to even know that such a thing exists. This is what a good framework does. A well-designed framework lets you easily get started building simple applications, without preventing you building more complicated and useful ones as you learn more. One final point about abstractions: they provide mechanisms for calling your application's code. We saw this a bit earlier with ActiveRecord's lifecycle methods. Frameworks will usually provide the overall structure for how an application should interact with its environment, defining sets of events that you write custom handlers for. With ActiveRecord lifecycles, the structure of before_create, create, after_create is predetermined, but you can define what happens at each step. This pattern is called inversion of control, and many developers consider it a key feature of frameworks. With *nix operating systems, you could say that in C programs the main function is a kind of onStart callback. The OS calls main, and main tells the OS what instructions should be run. However, the OS controls when instructions are actually executed because the OS is in charge of scheduling. It's a kind of inversion of control, right? 🤔 Communication Frameworks coordinate resources, and (it's almost a tautology to say this) coordination requires communication. Communication is hard. Frameworks make it easier by translating the disparate "languages" spoken by resources into one or more common languages that are easy to understand and efficient, while also ensuring extensibility and composability. Frameworks also do some of the work of ensuring resilience. This usually entails: Establishing naming and addressing conventions Establishing conventions for how to structure content Introducing communication brokers Handling communication failures (the database is down! that file doesn't exist!) One example many people are familiar with is the HTTP stack, a "language" used to communicate between browser and server resources: HTTP structures content (request headers and request body as text) TCP handles communication failures IP handles addressing Conventions The file model is a "common language", and the OS uses device drivers to translate between between the file model and whatever local language is spoken by hardware devices. It has naming and addressing conventions, letting you specify files on the filesystem using character strings separated by slashes that it translates to an internal inode (a data structure that stores file and directory details, like ownership and permissions). We're so used to this that it's easy to forget it's a convention; *nix systems could have been designed so that you had to refer to files using a number or a UUID. The file descriptors I described in the last section are also a convention. Another convention the file model introduces is to structure content as byte streams, as opposed to bit streams, character streams, or xml documents. However, bytes are usually too low-level, so the OS includes a suite of command line tools that introduce the further convention of structuring bytes by interpreting them as characters (sed, awk, grep, and friends). More recently, more tools have been introduced that interpret text as YAML or JSON. The Clojure world has further tools to interpret JSON as transit. My YAML tools can't do jack with your JSON files, but because these formats are all expressed in terms of lower-level formats, the lower-level tools can still work with them. Structure affects composability. The file model's simplicity is what allows it to be the "universal I/O model." I mean, just imagine if all Linux processes had to communicate with XML instead of byte streams! Hoo boy, what a crazy world that would be. Having a simple, universal communication system makes it extremely easy for new resources to participate without having to be directly aware of each other. It allows us to easily compose command line tools. It allows one program to write to a log while another reads from it. In other words, it enables loose coupling and all the attendant benefits. Communication Brokers Globally addressable communication brokers (like the filesystem, or Kafka queues, or databases) are essential to enabling composable systems. Global means that every resource can access it. Addressable means that the broker maintains identifiers for entities independently of its clients, and it's possible for clients to specify entities using those identifiers. Communication broker means that the system's purpose is to convey data from one resource to another, and it has well-defined semantics: a queue has FIFO semantics, the file system has update-in-place semantics, etc. If Linux had no filesystem and processes were only allowed to communicate via pipes, it would be a nightmare. Indirect communication is more flexible than direct communication. It supports decoupling over time, in that reads and writes don't have to happen synchronously. It also allows participants to drop in and out of the communication system independently of each other. (By the way, I can't think of the name for this concept or some better way to express it, and would love feedback here.) I think this is the trickiest part of framework design. At the beginning of the article I mentioned that developers might end up hacking around a framework's constraints, and I think the main constraint is often the absence of a communication broker. The framework's designers introduce new resources and abstractions, but the only way to compose them is through direct communication, and sometimes that direct communication is handled magically. (I seem to recall that Rails worked with this way, with tight coupling between Controller and Views and a lack of options for conveying Controller data to other parts of the system). If someone wants to introduce new abstractions, they have to untangle all the magic and hook deep into the framework's internals, using -- or even patching! -- code that's meant to be private. I remember running into this with Rails back when MongoDB was released; the document database resource was sufficiently different from the relational database resource that it was pretty much impossible for MongoDB to take part in the ActiveRecord abstraction, and it was also very difficult to introduce a new data store abstraction that would play well with the rest of the Rails ecosystem. For a more current example, a frontend framework might identify the form as a resource, and create a nice abstraction for it that handles things like validation and the submission lifecycle. If the form abstraction is written in a framework that has no communication broker (like a global state container), then it will be very difficult to meet the common use case of using a form to filter rows in a table because there's no way for the code that renders table data to access the form inputs' values. You might come up with some hack like defining handlers for exporting the form's state, but doing this on an ad-hoc basis results in confusing and brittle code. By contrast, the presence of a communication broker can make life much easier. In the Clojure world, the React frameworks re-frame and om.next have embraced global state atoms, a kind of communication broker similar to the filesystem (atoms are an in-memory storage mechanism). They also both have well defined communication protocols. I'm not very familiar with Redux but I've heard tell that it also has embraced a global, central state container. If you create a form abstraction using re-frame, it's possible to track its state in a global state atom. It's further possible to establish a naming convention for forms, making it easier for other participants to look up the form's data and react to it. (Spoiler alert: the framework I've been working on does this!) Communication systems are fundamental. Without them, it's difficult to build anything but the simplest applications. By providing communication systems, frameworks relieve much of the cognitive burden of building a program. By establishing communication standards, frameworks make it possible for developers to create composable tools, tools that benefit everybody who uses that framework. Standards make infrastructure possible, and infrastructure enables productivity. In this section I focused primarily on the file model because it's been so successful and I think we can learn a lot from it. Other models include event buses and message queues. I'm not going to write about these because I'm not made of words, ok?!? Environments Frameworks are built to coordinate resources within a particular environment. When we talk about desktop apps, web apps, single page apps, and mobile apps, we're talking about different environments. From the developer's perspective, environments are distinguished by the resources that are available, while from the user's perspective different environments entail different usage patterns and expectations about distribution, availability, licensing, and payment. As technology advances, new resources become available (the Internet! databases! smart phones! powerful browsers! AWS!), new environments evolve to combine those resources, and frameworks are created to target those environments. This is why we talk about mobile frameworks and desktop frameworks and the like. One of the reasons I stopped using Rails was because it was a web application framework, but I wanted to build single page applications. At the time (around 2012?), I was learning to use Angular and wanted to deploy applications that used it, but it didn't really fit with Rails's design. And that's OK. Some people write programs for Linux, some people write for macOS, some people still write for Windows for some reason (just kidding! don't kill me!). A framework is a tool, and tools are built for a specific purpose. If you're trying to achieve a purpose the tool isn't built for, use a different tool. More Benefits of Using Frameworks So far I've mostly discussed how frameworks bring benefits to the individual developer. In this section I'll explain how frameworks benefit communities, how they make programming fun, and (perhaps most importantly) how they are a great boon for beginners. First, to recap, a framework is a set of libraries that: Manages the complexity of coordinating the resources needed to write an application By providing abstractions for those resources And systems for communicating between those resources Within an environment So that programmers can focus on writing the business logic that's specific to their product This alone lifts a huge burden off of developers. In case I haven't said it enough, this kind of work is hard, and if you had to do it every time you wanted to make an application it would be frustrating an exhausting. Actually, let me rephrase that: I have had to do this work, and it is frustrating and exhausting. It's why Rails was such a godsend when I first encountered it in 2005. Frameworks Bring Community Benefits Clear abstractions and communication systems allow people to share modules, plugins, or whatever you want to call framework extensions, creating a vibrant ecosystem of reusable components. If you accept my assertion that an operating system is a framework, then you can consider any program which communicates via one of the OS's communication systems (sockets, the file model, etc) to be an extension of the framework. Postgres is a framework extension that adds an RDBMS resource. statsd is an extension that adds a monitoring resource. Similarly, Rails makes it possible for developers to identify specialized resources and extend the framework to easily support them. One of the most popular and powerful is Devise, which coordinates Rails resources to introduce a new user authentication resource. Just as using Postgres is usually preferable to rolling your own database, using Devise is usually preferable to rolling your own authentication system. Would it be possible to create a Devise for Clojure? I don't think so. Devise is designed to be database agnostic, but because Clojure doesn't really have a go-to framework that anoints or introduces a go-to database abstraction, no one can write the equivalent of Devise in such a way that it could easily target any RDBMS. Without a framework, it's unlikely that someone will be able to write a full-featured authentication solution that you can reuse, and if you write one it's unlikely others would see much benefit if you shared it. I think it's too bad that Clojure is missing out on these kinds of ecosystem benefits. Another subtler benefit frameworks bring is that they present a coherent story for how developers can build applications in your language, and that makes your language more attractive. Building an application means coordinating resources for the environment you're targeting (desktop, mobile, SPA, whatever). If your language has no frameworks for a target environment, then learning or using the language is much riskier. There's a much higher barrier to building products: not only does a dev have to learn the language's syntax and paradigms, she has to figure out how to perform the complex task of abstracting and coordinating resources using the language's paradigms. If your goal is to create a mass-market product, choosing a language that doesn't have frameworks for your target environments is a risky choice. Finally, frameworks become a base layer that you can create tooling for. The introduction of the filesystem made it possible for people to write tools that easily create and manipulate files. Rails's abstractions made it easy to generate code for creating a new database table, along with an entire stack - model, view, controller - for interacting with it. Frameworks Make Development Fun If you still think frameworks are overkill or more trouble than they're worth, believe me I get it. When I switched from Rails to Clojure and its "libraries not frameworks" approach, I loved it. A framework felt unnecessary because all the pieces were so simple that it was trivial for me to glue them together myself. Also, it was just plain fun to solve a problem I was familiar with because it helped me learn the language. Well, call me a jaded millenial fart, but I don't think that this work is fun anymore. I want to build products, not build the infrastructure for building products. I want a plugin that will handle the reset password process for me. I want an admin panel that I can get working in five minutes. Frameworks handle the kind of work that ideally only has to be done once. I don't want to have to do this work over and over every time I want to make something. For me, programming is a creative endeavor. I love making dumb things and putting them in front of people to see what will happen. Rails let me build (now defunct) sites like phobiatopia.com, where users could share what they're afraid of. The site would use their IP address to come up with some geo coordinates and use Google Maps to display a global fear map. A lot of people were afraid of bears. Frameworks let you focus on the fun parts of building an app. They let you release an idea, however dumb, more quickly. Frameworks Help Beginners Frameworks help beginners by empowering them to build real, honest-to-god running applications that they can show to their friends and even make money with, without having to fully understand or even be aware of all the technology they're using. Being able to conjure up a complete creation, no matter how small or ill-made, is the very breath of wonder and delight. (I don't know exactly what this means, but I like how it sounds!) There's a kind of thinking that says frameworks are bad because they allow beginners to make stuff without having to know how it all works. ActiveRecord is corrupting the youth, allowing them to build apps without even knowing how to pronounce SQL. There's another line of thinking that says it's bad to try to make things easier for beginners. It's somehow virtuous for people to struggle or suffer for the sake of learning. Hogwash. Fiddlefaddle. Poppycock. Joy beats suffering every time, and making learning more joyful allows more people to reap the benefits of whatever tool or product you've created. I am a photographer. I have a professional camera, and I know how to use it. Some of my photos require a fair amount of technical knowledge and specialized equipment: tea This isn't something you can create with a camera phone, yet somehow I'm able to enjoy myself and my art without complaining that point-and-shoot cameras exist and that people like them. Novices benefit greatly from expert guidance. I don't think you can become a master photographer using your phone's camera, but with the phone's "guidance" you can take some damn good photos and be proud of them. And if you do want to become a master, that kind of positive feedback and sense of accomplishment will give you the motivation to stick with it and learn the hard stuff. Frameworks provide this guidance by creating a safe path around all the quicksand and pit traps that you can stumble into when creating an app. Frameworks help beginners. This is a feature, not a bug. A Clojure Framework Frameworks are all about managing the complexity of coordinating resources. Well, guess what: Managing Complexity is Clojure's middle name. Clojure "Managing Complexity" McCarthy-Lisp. Personally, I want a single-page app (SPA) framework, and there are many aspects of Clojure's design and philosophy that I think will make it possible to create one that seriously kicks ass. I'll give just a few examples. First, consider how Linux tools like sed and awk are text-oriented. Developers can add additional structure to text by formatting it as JSON or YAML, and those text-processing tools can still work the structured text. In the same way, Clojure's emphasis on simple data structures means that we can create specialized structures to represent forms and ajax request, and tools to process those structures. If we define those structures in terms of maps and vectors, though, we'll still be able to use a vast ecosystem of functions for working with those simpler structures. In other words, creating specialized structures does not preclude us from using the tools built for simpler structures, and this isn't the case for many other languages. Second, Clojure's abstraction mechanisms (protocols and multimethods) are extremely flexible, making it easy for us to implement abstractions for new resources as they become available. Third, you can use the same language for the frontend and backend!!! Not only that, Transit allows the two to effortlessly communicate. This eliminates an entire class of coordination problems that frameworks in other languages have to contend with. In my opinion, the Clojurian stance that frameworks are more trouble than they're worth is completely backwards: Clojure gives us the foundation to build a completely kick-ass framework! One that's simple and easy. One can dream, right? My ambition in building a SPA framework is to empower current and future Clojure devs to get our ideas into production fast. I want us to be able to spend more time on the hard stuff, the fun stuff, the interesting stuff. And I want us to be able to easily ship with confidence. The framework I'm building is built on top of some truly amazing libraries, primarily Integrant, re-frame, and Liberator. Integrant introduces a component abstraction and handles the start/stop lifecycle of an application. re-frame provides a filesystem and communication broker for the frontend. Liberator introduces a standard model for handling HTTP requests. If my framework is useful at all it's because the creators of those tools have done all the heavy lifting. My framework introduces more resources and abstractions specific to creating single-page apps. For example, it creates an abstraction for wrapping AJAX requests so that you can easily display activity indicators when a request is active. It creates a form abstraction that handles all the plumbing of handling input changes and dispatching form submission, as well the entire form lifecycle of fresh, dirty, submitted, invalid, succeeded, etc. It imposes some conventions for organizing data. As I mentioned, the framework is not quite ready for public consumption yet becaause there's still a lot of churn while I work out ideas, and because there's basically no documentation, but I hope to release it in the near future. If you'd like to see a production app that uses the framework, however, I invite you to check out Grateful Place, a community site for people who want to support each other in growing resilience, peace, and joy by practicing compassion, gratitude, generosity, and other positive values. By joining, you're not just helping yourself, you're helping others by letting them know that you support them and share their values. Please click around and look at the snazzy loading animations. And if you feel so moved, please do join! I love getting to interact with people in that context of mutual support for shared values. One of the only things I care about more than Clojure is helping people develop the tools to navigate this crazy-ass world :D In the mean time, I'll keep working on getting this framework ready for public consumption. Expect another blawg article sharing some details on how Grateful Place is implemented. Then, eventually, hopefully, an actual announcement for the framework itself :) If you don't want to wait for my slow butt, then check out some ofthe amazing Clojure tools that already exist: Luminus Fulcro which probably does everything I want my framework to, only better re-frame remains my favorite frontend framework duct is great but its docs aren't that great yet Coast on Clojure, a full stack web framework (Sorry if I neglected your amazing Clojure tool!) Thanks to the following people who read drafts of this article and helped me develop it: Mark Bastian Dmitri Sotnikov aka @yogthos Joe Jackson Sergey Shvets Kenneth Kalmer Sean whose last name I don't know Tom Brooke Patrick whose last name I don't know (update: It's Patrick French!) Fed Reggiardo Vincent Raerek Ernesto de Feria Bobby Towers Chris Oakman The TriClojure meetup

Buddy Systems at Work: a Framework for Rewarding Relationships on Remote Teams

over 2 years ago | Daniel Higginbotham: Flying Machine Studios

One of my work mottos is "support the hell out of each other": helping your friends grow and succeed is far more rewarding than chasing after the next gold star for yourself. While I was at Reify Health I was lucky enough to work with other devs who had a similar ethos. There were no rock stars, and everyone looked out for each other. I'm happy to have gotten to work somewhere where I made real, honest-to-god friends :) During my final weeks there I collaborated with coworkers on an experimental framework for supporting each other in a formal context. This framework would let us explore how to be there for each other in more meaningful ways while maintaining boundaries. This seemed particularly valuable in Reify's mostly-remote environment. With their permission, I'm sharing the initial framework proposal below: The Buddy System This doc outlines a proposal for a buddy system. The main idea behind the buddy system is for us to support each other by regularly spending time chatting with each other about things beyond whatever story is at hand. Exactly what that looks like will be up to you; this doc includes suggestions for how to have rewarding conversations. First, here are some ground rules: Ground Rules It’s confidential. Whatever you talk about remains between you and your buddy unless you say otherwise. This is meant to provide each other with time and space to speak about what’s going on :D It’s completely voluntary. You don’t have to participate, and if you do participate you can shape it however you want to: the frequency and content of chats is up to you. You can opt out or change buddies whenever you want. If you decide to drop out of the buddy system, it’s totally up to you if you want to explain why, or not. The buddy system is meant to consciously cultivate a space where we can be vulnerable with each other; part of making that safe is setting the expectation that people can opt out without having to explain why. By the same token, in joining the buddy system you accept that your buddy may at some point wish to opt or change buddies, and you commit to try to not take that personally if it happens 👍 Every three months we’ll change buddies - this gives us a chance to grow together, and also a chance to keep getting new perspectives by buddying with other folks on the team. There’s no impact on your yearly review. Though “Buddy System” would be a wonderfully Orwellian name for a mandatory chatting initiative. It’s reciprocal. Buddies commit to listening to and sharing with each other. If you feel like you have room to grow with listening or sharing, that’s totally cool - and a good first step is to share that with your buddy :) It’s not hierarchical. Being a Buddy means whole-hearted listening, asking questions, and just being supportive, which is independent of any junior or senior designations. The point is to support each other as human beings, not to meet project objectives :-) Suggestions Tell your buddy what you want to get out of the buddy system. Do you want to talk about your career? Your goals at Reify? How to improve our architecture? By letting your buddy know what you’re looking for, you can get all Jerry Maguire and help them help you. Listen your asses off. Avoid the urge to give advice unless asked for it, but instead try to truly understand what your buddy is saying, and try to help them gain clarity. Potential topics include goal-setting and “life at Reify.” Goal setting: It can be very useful to talk to someone about your broader goals and to have them hold you accountable. Sharing a goal often makes it more meaningful and gives you more motivation to reach it. Life at Reify: We have retros, which are useful, but regular one-on-one conversations also have a place in supporting each other These topic suggestions are not meant to be exhaustive. When asking for feedback, communicate what kind of feedback you want: A pat on the back High-level feedback Nitty-gritty feedback Notes Daniel is acting as facilitator in that he’s organizing discussion about the Buddy System so that we all come to a shared understanding of what it is and what we want to get out of it. However, Daniel isn’t, like, the Buddy System’s owner, and can’t tell you what to do. Some reading this might be uncomfortable with the ideas outlined in the buddy system - and that's totally ok! A key part of the framework is you don't have to buy into it. But I think it speaks volumes about the level of trust and mutual good feeling among the team that almost everyone was excited about it while respecting that we're all different and not everyone prefers the same way of interacting. On the other hand, new hires were especially excited because they felt like it would help them acclimate to their new environment. To me, a team that is enthusiastic about these ideas is a team that I'd want to work with; it's a team that actually likes the idea of relating to each other on a real, human level. One topic that came up was, how is this different from just being friends? Is it really necessary to have a structure that everyone agrees to? Does that bring an element of artifice into friendship? My thinking is that structure is a precondition for vulnerability. Structure tells us to be extra mindful in how we interact with each other, and it provides boundaries (like a clear time limit) to protect us from getting overwhelmed. By designating a time and place to consciously interact with each other, it also resolves the problem of never knowing when is "the right time" to bring something up. This would be helpful for new hires, sure, but even in my personal life I've found that structure is much better at fostering deeper interactions than just winging it. Knowing that everyone is "in it" together and that "it" won't last forever provides a solid foundation for trust. Final note: vulnerability here does not mean you have to spill your guts, emotionally leaking on each other all the time. It does mean feeling safe to express that you're struggling or that you feel dissatisfied or hurt in some way. What are your thoughts? Have you tried something like this where you worked, and how did it go? P.S. Reify is hiring!

Techniques for Efficiently Learning Programming Languages

over 4 years ago | Daniel Higginbotham: Flying Machine Studios

Learning programming languages is a skill: do it well and you'll experience one dopamine hit after another as you master something new; do it poorly and you'll feel constantly frustrated and even give up. What follows are the best techniques for learning programming languages that I've picked up over years of teaching programming by writing books and articles, doing talks, and running a training course. Many of these techniques are pulled from books explaining the latest research in efficient learning, and you can find those books (along with other great programming books) at Community Picks: Learn Programming. Test Yourself Constantly to Defeat The Illusion of Competence One of the worst ways to learn is to re-read or re-watch material. This kind of review gives you the feeling that you understand the topic covered because it seems like you're understanding the topic effortlessly. Researchers call this the illusion of competence. A significantly better approach (and one of the best techniques you can employ) is to test yourself constantly. Instead of re-reading what a function or class or object is, ask yourself to define these concepts or use them in a short program; force yourself to somehow demonstrate your understanding. This process often feels uncomfortable, but it's much more efficient at forming long term memories. You can take this one step further and test yourself before you've covered the material by, for example, attempting exercises before reading a chapter. Remarkably, this has also been shown aid memory formation. The impressive impact that testing has on learning is called the testing effect, and here are some specific ways you can take advantage of it: Before reading a chapter or watching a video, try guessing at what you're about to learn and write it down. Try doing a chapter's exercises before reading the chapter. Always do exercises, even the hard ones. It's OK to give up on an exercise and come back to it later (or never, even), but at least try it. (More on this in the next section.) Read a short program and try to recreate it without looking at the original code. Or, go smaller and do this with a function. Immediately after learning a new concept like objects, classes, methods, or higher-order functions, write code that demonstrates that concept. Create diagrams that illustrate concepts, both in isolation and how they relate to each other. Blog about a concept you just learned. Try explaining the concept to a non-technical friend. (I did this a lot when writing Clojure for the Brave and True; being able to explain an idea in layman's terms forces you to understand the idea deeply.) Many of these techniques boil down to write some code! With programming it's easy to believe we're learning a lot just by reading because programming is text-heavy and conceptual. But it's also a skill, and like any other skill you have to perform it to get better. Writing code is the best way to reveal your incorrect assumptions about programming. The faster you do that, the faster you can make corrections and improve. If you'd like to learn more about the testing effect, check out make it stick: The Science of Successful Learning. Take Time to Unfocus If you're stuck on a problem or don't understand something you just read, try taking a walk or even a shower -- anything to enter a relaxed, unfocused state of mind. It probably seems counterintuitive that one of the best ways to get unstuck is to stop trying for a little while, but it's true. The problem is that it's easy for us to put on mental blinders when we're focusing hard on a problem. I mean, that's pretty much what "focus" means. But by focusing hard, we're exploring only a small portion of the solution space. By unfocusing, our unconscious mind is able to explore and make connections across vast swaths of our experience. To me it's like when you're trying to find a destination on a paper map (remember those?). You can unconsciously become convinced that the city you're trying to reach should be right here! in the upper-left qudrant of the map, so you look at it over and over without success. Then you put the map down and take a deep breath and stare at nothing for a minute, and when you look at the map again the actual location jumps out at you immediately. We've all had the experience of having a sudden insight in the shower; now you have a slightly better understanding of why that happens, and you can employ the technique on purpose. Personally, I will actually take a shower if I'm stuck on something, and it's remarkable how well the technique works. And how clean I am. If you'd like to learn more about the focused and diffuse modes of thinking, check out A Mind for Numbers: How to Excel at Math and Science (Even If You FLunked Algebra). Don't Waste Time Being Frustrated Related to the last section: don't waste time being frustrated with code. Frustration leads us into doing stupid things like re-compiling a program or refreshing the browser with the hope that this time it will be magically different. Use frustration as a signal that there's a gap in your knowledge. Once you realize you're frustrated, it can help to take a step back and clearly identify the problem. If you've written some code that's not working, explicitly explain to yourself or someone else the result that you expected. Use the scientific method and develop a hypothesis for what's causing the unexpected behavior. Then test your hypothesis. Try again, and if a solution still eludes you, put the problem aside and come back to it later. I can't tell you how many times I've thrown my laptop in disgust over a seemingly unsolvable problem, only to look at it the next day and have an obvious solution pop into my head immediately. This happened last week, even. Identify Which Programming Language Aspect You're Dealing With Personally, I find it useful to keep in mind that when you're learning a programming language, you're actually learning four things: How to write code: syntax, semantics, and resource management The language's paradigm: object-oriented, functional, logic, etc. The artifact ecosystem: how to build and run executables and how to use libraries Tooling: editors, compilers, debuggers, linters It's easy to get these four facets mixed up, with the unfortunate result that when you run into a problem you end up looking in completely the wrong place. Someone who's completely new to programming, for example, might start out by trying to build iOS apps. They might try to get their app running on a friend's phone, only to see some message about needing a developer certificate or whatever. This is part of the artifact ecosystem, but an inexperienced person might see this as a problem with how to write code. They might look at every line they wrote to figure out the problem, when the problem isn't with their code at all. I find it easier to learn a language if I tackle each of these aspects systematically, and in another blog post I'll present a general list of questions that need answering that should help you in learning any language. Identify the Purpose, External Model, and Internal Model Whenever you’re learning to use a new tool, its useful to identify its purpose, external model and internal model. When you understand a tool's purpose, your brain gets loaded with helpful contextual details that make it easier for you to assimilate new knowledge. It's like working on a puzzle: when you're able to look at a picture of the completed puzzle, it's a lot easier to fit the pieces together. This applies to languages themselves, and language libraries. A tool's external model is the interface it presents and the way it wants you to think about problem solving. Clojure’s external model is a Lisp that wants you to think about programming as mostly data-centric, immutable transformations. Ansible wants you to think of server provisioning in terms of defining the end state, rather than defining the steps you should take to get to that state. A tool's internal model is how it transforms the inputs to its interface into some lower-level abstraction. Clojure transforms Lisp into JVM bytecode. Ansible transforms task definitions into shell commands. In an ideal world, you wouldn’t have to understand the internal model, but in reality it’s almost always helpful to understand a tool's internal model because it gives you a unified perspective on what might seem like confusing or contradictory parts. When the double-helix model of DNA was discovered, for example, it helped scientists make sense of higher-level phenonema. My point, of course, is that this blog post is one of the greatest scientific achievements of all time. Tutorials often mix up a tool's external model and internal model in a way that’s confusing for learners; it's helpful to be aware of this so that you can easily identify when it's causing you frustration. Spaced Repetition Helps You Remember Spaced Repetition been proven to be one of the best ways to encode new information in long-term memory. The idea is to quiz yourself at ever-increasing time intervals to minimize memory decay using the fewest number of repetitions. The Guardian wrote a great introductory article. Sleep and Exercise Take care of your body! It's more than just a vehicle for your brain. If you want to be able to stay focused and learn efficiently, getting adequate sleep and exercise beats the pants off caffeine and energy drinks. More tips? If you have any useful tips, please leave them in the comments! If you'd like more excellent resources on learning to program, check out the Community Picks: Learn Programming, a community-curated collection of the best books for learning programming. It includes a wide array of subjects, including introductory programming books, books on craftsmanship, and books on soft skills and interviews.

Zen and the Art of Blowing Things Up

over 4 years ago | Daniel Higginbotham: Flying Machine Studios

The Dark Knight came out at the same time that I was becoming intensely interested in Buddhism, and the movie struck me as a kind of extended Buddhist parable, with Bruce Wayne as an icon of the suffering that results from clinging and the Joker as a very fucked up enlightened being. Here are some of my main reasons: (warning: spoilers!) Suffering and Impermanence Buddhism teaches that suffering comes from constantly scrambling for stability in an unstable world. Nothing is permanent, but we strive mightily to hold on to the illusion of permanence, only to suffer when reality once again asserts itself. In the scene where Batman is in the interrogation room beating the stuffing out of the Joker, the Joker just laughs and says, "You have nothing! Nothing to threaten me with. Nothing to do with all your strength." And that cuts to the heart of who the Batman is: a person whose reaction to the first shock of impermanence (his parents dying) was to dedicate his life to developing his strength so that he would never experience that pain again. The Batman's entire being is an exercise in doing just one... more... thing... to establish control. I mean, who would want to live that life? And here's the Joker, ready with the Zen slap: he tells Batman where to find Harvey and his girlfriend Rachel, giving him the illusion of control over who to save. And we know how that turned out: the Joker lied to the caped crusader, telling him Rachel was at one address, but actually Harvey was there. Batman races to save Rachel, only to find find Harvey instead. Nothing to do with all that strength. This really hit home for me because, at the time, I was in a relationship with someone that had a chronic illness. I could try to do everything to make her life less painful and more enjoyable, but there was only so much I or anyone could do. Natural Metaphors and Endless Desire At one point, the Joker says, "I'm just a dog chasing cars, I wouldn't know what to do if I caught one." Or something like that. This is a double whammy: "a dog chasing cars" is totally something a zen master would say, but besides that it expresses our human tendency to constantly strive for the next thing. We're all just dog chasing cars, not knowing what to do when catch one. You get a job: now you want a better job. You get a house: now you want a bigger house. Better to just say fuck it, put on some makeup, and start blowing up hospitals. No Self The Buddhist idea of "no self" is actually somewhat nuanced and I'm purposefully misrepresenting it here, but it would be difficult to argue that the Joker clings to any notion of self in the way that most of us non-enlightened psychopaths do. He gives multiple stories for how he got his scars, revealing the Buddhist truth that our sense of self is just a story we tell ourselves and that no one story is better than any other. He gives up his name and identity to become a fucking clown. And people really hate clowns! Interconnectedness One Buddhist teaching is the idea of interconnectedness: that we are all one, but not in the wavy-gravy way your high uncle used to ramble about. My take on the idea is that it's related to no self: you cannot point to a "you" that is separate from the world you exist in. It's a fallacy to believe your "you" is somehow different from the collection of experiences of the world, so that you are inextricably intertwined with the world. You can't point to one thing outside yourself and say "I am not this," because you are pointing at it you moron, and so in that moment who you are is a guy pointing at that thing, and life is the accumulation of such experiences of things outside of "you." ANYWAY I am totally not your high uncle, OK? Point is, the idea of interconnectedness is meant to foster a sense of compassion. You can see that at the end of the movie when the Joker plays his hilarious "blow up the other boat" gag. Two boats full of strangers become enlightened, refusing to do harm to people they haven't even met.

My Writing Process

about 5 years ago | Daniel Higginbotham: Flying Machine Studios

The letter below is a response to an email from a Twitter friend. For context: I've written some instructional material that's enjoyed success online, including Clojure for the Brave and True and Clean Up Your Mess: A Guide to Visual Design for Everyone. Robert asked me a few questions about my process. Hi Robert! You're in the unfortunate position of being my permission to write at length about my creative process without seeming completely self indulgent: someone's actually interested in listening to me describe the chaos between my ears! Be prepared to read way more than you asked for. How do you decide who you are writing for? This one’s key for me. When I’ve written in the past, I’ve struggled with knowing how much or how little to say. I’m curious to learn how you deal with that. First: I've found it useful to follow the standard advice and pick an individual or two and write for them. The argument goes that you'll find it much easier to choose words and have a consistent tone, and that's proven true for me. When I wrote Clean Up Your Mess, I was one of three programmers working in an office full of charming people who did business analysis for management consultants, and I wrote the guide for a couple of them that I had become friends with. These analysts produced powerpoint decks by the barrel, and every single one was ugly. Similarly, Clojure for the Brave and True started as a guide for a couple coworkers that I was mentoring. However, picking your audience is only half the process. The other half is picking who the author is. I'm going to go on a bit of a tangent to explain this. When I started writing Clean Up Your Mess, my personal life was a mess. In hopes of making it less messy I had been spending most of my free time reading Buddhist literature and listening to lectures by Thich Nhat Hanh and Pema Chodron (in fact, my site opens with a Thich Nhat Hanh quote). I was spending a great deal of time meditating and reflecting. (You're interested in Buddhism too, right?) One of the ideas that really grabbed me was that our self-conceptions are just stories we tell ourselves with no reality to them. I won't belabor the point here; suffice to say that I found it useful and powerful to be able to perceive my identity as a construct as ephemeral as smoke. Around the same time, I started taking improv comedy classes, where I would spend two hours a week constantly coming up with new, unique characters and living in them for a few minutes at a time. When you're sold on the idea that your day-to-day you is itself just a character, improv becomes way easier and more fun. It's also fun to do this in real life - like, recently my wife and I were at the grocery store and I decided my character was "Super Excited About Carrots Guy", and I started shouting "OH MY GAWD JESS CARROTS! THEY HAVE CARROTS JESS! CARROTS!" Everyone was laughing as they escorted me out! Anyway, during this time of buddhism and improv I was writing Clean Up Your Mess and when I sat down to write I got to play with new identities, new characters, and see which was the most fun. It became much easier to find words and keep a consistent tone when I decided on the author's character. So, you mentioned not knowing how much or how little to write. Knowing your character helps. In Clojure for the Brave and True, one of my editors would sometimes whittle down sentences to the bone, leaving only pure information; I would undo those edits with gusto, adding back the shape and texture of the author's voice. There's another aspect of writing instructional material, which is that 95% of the time you can take more time to explain something. Even if you're writing for experienced audiences, at least take the time to explain acronyms and jargon. How often and how much do you write? Not enough and not enough! Writing was easiest, however, when I had a consistent schedule of writing at least one hour every single day. Let me tell you about my garage. My garage is a living f*cking nightmare. It's like Cthulhu and the dread of the dark you had as a child made disgusting love and produced a horde of horrors that died in giant filthy clumps all over. At the beginning of the year I suppressed my gag reflex long enough to clean it out with the goal of setting up a photo studio, which I did, only to let it sit for three months. So three months later I go in there to actually take photos and guess what: it's a complete horror show, again! If I had only spent time every day or even every week maintaining it, I'm sure it would have been much easier to keep clean. If I had cleaned it out, all those tiny insect monsters wouldn't have been able to breed, there wouldn't have been two dozen spider egg sacs, etc. When I don't write consistently it's like that. I have to crawl into the nest of no one cares what you write and this time everyone's going to hate it called my mind and clear it all out before I can actually start writing. When I write consistently, on the other hand, it's actually quite pleasant! One more note: much of the writing process is research. Every time I find myself struggling to explain something, it's because I don't understand it well enough. When that happens I put down the keyboard and pull out the pen and notebook. I try to articulate the questions that will lead me to understand what I want to explain. I read and I draw diagrams and play with words until I feel like I understand the concept. Becoming aware that, for me, writer's block is a lack of understanding, has helped me be much more productive. Collaboration Who does your drawings? My wife! She's pretty great. Do you have someone who edits for you? I did when I worked with No Starch. They were awesome! Clojure for the Brave and True is much, much better because they kept pushing me to write more clearly and explain things more fully. For other projects, I'll ask friends to look at drafts. How much does it cost? I paid my wife in carrots. OH MY GAWD JESS THEY HAVE CARROTS! No Starch gets 85% of hard copy sales and 77.5% of ebook sales (or something like that). I hope this was helpful!

Understanding Recursion

about 5 years ago | Daniel Higginbotham: Flying Machine Studios

During my Clojure training I've found that recursion routinely trips people up. It definitely tripped me up for a long time. I've tried to develop an explanation that's clear and intuitive, so if you're still scratching your head about recursion, read on! A classic recursion example is calculating n factorial, which is n multiplied by every natural number before n; 3 factorial is 6 (3 times 2 times 1), 4 factorial is 24, 5 factorial is 120. The code snippet that follows is a typical implementation of factorial; if you're reading this, then presumably it's confusing - which is great! It means that I haven't written this article for nothing. function factorial(n) { if (n == 1) { return n; } else { return n * factorial(n - 1); } } What makes this function recursive is that factorial calls itself. That's also what makes the function tricky; the function calls itself!? We're used to functions calling other functions to get work done. For example, this function uppercases a string and prepends "Yo, " to it: function yoShout(str){ return "Yo, " + str.toUpperCase(); } yoShout("gimme a donut"); // "Yo, GIMME A DONUT" In this tiny example, yoShout does its work by using the toUpperCase function. It's easier to understand than a recursive function because yoShout treats toUpperCase as a black-box abstraction. You don't have to tax your brain by loading toUpperCase's implementation details into your short-term memory. Let's re-write factorial to use function calls this way, with function's body calling another function in order to get its work done. To calculate 3 factorial, you could write a series of functions, factorial_1, factorial_2, and factorial_3, like this: function factorial_1() { return 1; } function factorial_2() { return 2 * factorial_1(); } function factorial_3() { return 3 * factorial_2(); } These functions feel safe and comfy. factorial_3 calls factorial_2, something we're completely familiar with, and likewise factorial_2 calls factorial_1. factorial_3 also does not care how factorial_2, just like in the string example. Unfortunately, these functions are also completely impractical; can you imagine writing factorial_1000? The recursive implementation doesn't have this problem. My suggestion is to try seeing the recursive implementation from the same perspective as the nonrecursive imiplementation. Here's the code again: function factorial(n) { if (n == 1) { return n; } else { return n * factorial(n - 1); } } You can look at this and say, "Oh, if n isn't 1, then this function does its work by calling some black-box function named factorial with the argument n - 1." You can look at the call to factorial(n - 1) as a call to a completely different function - one that just happens to have the same name and algorithm. That's it! I hope it helps. If you've been confused by recursion, I'd love to hear your feedback!

Timeless Programming Tools

over 5 years ago | Daniel Higginbotham: Flying Machine Studios

I've been programming professionally for a dozen years. Some of what I've learned is best forgotten (oh god, Cold Fusion), but there have been many tools, techniques, and concepts that continue to be useful every day. Here are mine; I'd love to hear what yours are so I can experience both the joy of discovery and regret for not learning about the tool earlier. Relational Algebra / SQL I feel lucky that, during my fourteenth summer, I apparently had no friends and so had nothing better to do than try and slog through a book on MySQL and the now-defunct mSQL. You can see from the reviews that the book "is sketchy, incomplete, and almost totally useless." But, it did introduce me to SQL and databases. Soon after, I learned relational algebra (the theory underlying RDBMSs) and that investment has been one of the best of my life. I can't count the number of times a LEFT OUTER JOIN has saved my bacon. Friends be damned! Learning relational algebra provided the foundation I needed to move easily from MySQL to Oracle and MS SQL Server when I joined EnterpriseCo, and in general knowing how to interact with databases without a framework or ORM helped me quickly advance career-wise. It's why, at 20, I was able to land a contract building a custom site for the city of Santa Fe, New Mexico, instead of just cobbling together Wordpress and Drupal plugins. If you come from Rails or some other framework that handles all the database interaction for you, one of the best things you can do for your career is to learn relational theory and SQL. Read a book by C. J. Date. The Unix Process Model Understanding Unix processes helped me understand what's actually happening when I run a program. It's also helped me understand what exactly a web server is and what I'm doing when I write a web application. The book Advanced Linux Programming has a chapter on processes for free online. Actually, the whole book is free. When you don't know about processes, programming is much harder and more mysterious. It's harder to understand performance, and it's harder to understand how programs interact with each other. If you ever feel a vague sense that you don't really get what's going when you run the apps you write, learning the process model will go a long way toward clearing things up. Regular Expressions Yeah, yeah, we've all heard the joke: "something something regular expressions, then you have two problems." Personally, I don't get it, because regular expressions are seriously badass. I remember going through O'Reilly's big fat regex book while I worked from 11pm till 7am as a night auditor at a hotel when I was 18, and being blown away at how powerful they are. To say that we programmers deal with text all the time is so obvious, it's not even worth saying. Regular expressions are an essential tool, and here's where you can learn to use them. Finite State Machines Regular expressions are built as finite state machines. Here's a great tutorial on FSMs showing how to actually build a regular expression. It's extremely cool! I think FSMs are covered in computer science 101, but since I only went to college for a year and even then I studied works written a couple millennia before before the computer revolution, I didn't actually learn about them until about six years ago. My colleagues and I were having trouble with a mobile app - we needed the initialization process to happen in a particular way, and the logic for ensuring that was getting pretty tangled. Once we took the time to learn about FSMs, though, it was easy to express the process as a series of states and transitions. I've since found that most tricky UI code can be improved this way. Just a couple months ago I was having trouble building a typeahead element from scratch with hoplon. Once I identified that the difficulty was in keeping track of all the possible states, it only took a few minutes drawing a state machine diagram and I was back on track. Emotion Management In my personal life I'm constantly learning about and practicing ways to manage emotions. This stems from both my personal aspiration to improve the lives of others and from the completely selfish reason that it helps me do good work. Emotion management is probably the most important meta-skill you can develop. I mean, emotions are at the core of who you are as a human being. The book Non-Violent Communication is an excellent resource for dealing with emotions. Also, my friend Alex Harms recently wrote a book specifically for technical people. Those are my programming power tools - I hope you find them useful!

Brave Clojure: Become a Better Programmer

over 5 years ago | Daniel Higginbotham: Flying Machine Studios

Next week week I'm re-launching www.braveclojure.com as Brave Clojure. The site will continue featuring Clojure for the Brave and True, but I'm expanding its scope a bit. Instead of just housing the book, the purpose of the site will be to help you and the people you cherish become better programmers. Like many other Clojurists, I fell in love with the language because learning it made me a better programmer. I started learning it because I was a bit bored and burnt out on the languages and tools I had been using. Ruby, Javascript, Objective-C weren't radically different from each other, and after using them for many years I felt like I was stagnating. But Clojure, with its radically different approach to computation (and those exotic parentheses) drew me out of my programming funk and made it fun to code again. It gave me new tools for thinking about software, and a concomitant feeling that I had an unfair advantage over my colleagues. So of course the subtitle of Clojure for the Brave and True is learn the ultimate language and become a better programmer. And, four years since I first encountered Rich Hickey's fractal hair, I still find Clojure to be an exceptional tool for becoming a better programmer. This is because Clojure is a fantastic tool for exploring programming concepts, and the talented community has created exceptional libraries for such diverse approaches as forward-chaining rules engines and constraint programming and logic programming, just to name a few. By learning a variety of programming concepts, you become a better programmer because you're better able to recognize what kind of problem you're working on and apply the right tools. To give just one example: when I'm working on complex UI widgets my work is always made easier if I model the widget using a finite state machine (FSM) - but I had to actually spend time learning what FSMs are and how to use them (those of you with comp sci degrees are probably chuckling at this). You may have started learning Clojure because you heard that functional programming, Lisp, and immutability can make your life easier, but the fun doesn't stop there! At Brave Clojure, I hope you'll continue using the ultimate language to explore the vast and infinitely fascinating world of computer programming with me. The first topic we'll explore is parallel programming. Here's the beginning of an ebook on the topic that I'm nearly finished with: Yes, it would be fair for you to describe my startup as the Google of palm reading. I don't know if you know this, but the fortune telling industry has no quality control standards. There is no Six Sigma of fortune telling. Sometimes it seems like these people are just making things up! We can do better. Did I say we? Yes! I want to you to embark on this startup journey with me. Lucky you! My - our - plan is to disrupt the field with an app that lets a user position their phone so that the camera faces their palm and takes a picture, then predicts the future. I call it: iFacePalm. And, uh, there's parallel programming in it. While I plan on spending most of my time helping you and all of your closest friends, enemies, and even frenemies become better programmers by writing about these big ideas, I'm also going to spend time covering practical Clojure topics like deployment and debugging. To make this happen, I'm working a (great) job only two days a week and spending the rest of the time on Brave Clojure, so I'll need to find a way to derive income from the site. I'm not sure yet what will be free and what won't. My dream is to make all the content available online for free. That's what gives me joy, and I think it's the best way to have a positive impact. Ideally, I would be supported in this endeavor through monthly donations, but I'll likely try different approaches over time. Finally, I've been putting together two sites to help you up your game: Open Source Clojure Projects, a directory of active projects, and Clojure Work, the first Clojure-only job board. The latter is still in beta, and I'm going to officially release it and move it to https://jobs.braveclojure.com next week. The best way to get better at Clojure is to actually write Clojure, and working on real projects (and even getting paid to do so) is a great way to do that. Companies who buy job board ads will also be supporting me in making more content - hooray! If you get excited about learning more programming ideas, or learning to develop Clojure programs better, or getting paid cold hard cash to write your favorite language, then I cordially invite you to consider signing up for my newsletter, where I'll announce every release: Writing Clojure for the Brave and True was one of the most rewarding experiences of my life in large part because I got to interact with so many kind, thoughtful, intelligent people. I look forward to your company as I embark on this new chapter of my journey!

A Directory of Open Source Clojure Projects

over 5 years ago | Daniel Higginbotham: Flying Machine Studios

Every now and again I'll come across a post where someone asks about getting involved in open source clojure projects. Because every question deserves a single-page app dedicated to its answer, I made Open Source Clojure Projects, a directory of active projects welcoming new contributors. Each project has (or should have) clear instructions on: Developing the project locally Running tests if there are tests Contributing code (pull request? tests required?) Contacting other devs - slack, mailing list, IRC, etc Also, there's a "beginner-friendly" flag so that new Clojurists can easily find projects with tasks that are appropriate for their skill level. So far, 22 projects have been added, which is awesome. A random sampling: milestones, which generates the best possible schedule from a set of tasks reverie, a CMS for power users Scheje, a little Scheme on Top of Clojure (!!!) system, a friendly layer on top of Stuart Sierra's component library If you have an open source project and want people to contribute, please add it! The Stack A few folks have asked me about what tools I used to create the site. I'll explain more below but briefly: The backend is a weird, unholy mess of fantastic classic libraries and sideshow code that I keep transplanting from one project to another so that I can answer the question what kind of abomination will this morph into this time?. The backend uses Github as a database, though, and that's pretty neat. The frontend uses re-frame and includes some form handling code that I'm proud of and that's actually worth stealing. Finally, there's also some code for deploying with Ansible that's worth stealing. In the time-honored fashion of technical blog posts everywhere, I'll now "dive in" and elaborate on each of those bullet points. Backend For the backend, the site uses ring, compojure, liberator, http-kit, and the weird set of tools that have accreted in my projects over the years. Even though the code footprint for the backend is pretty small, it's pretty idiosyncratic, containing handfuls of half-formed ideas I keep noodling around with. Hopefully someday soon I'll be able to really nail down my approach to backend dev and share it, because it does allow me to write apps quickly. One cool part of the site is that it uses a Github repo, Open Source Projects, as its database. Browse the projects directory and you can see every project that's listed on the site stored as an EDN file. When the backend starts up it reads from Github, and whenever someone posts or edits a listing it writes the EDN file using Github's API. The nice thing about this approach is that I didn't have to worry about standing up a server or keeping backups. And it was just fun to code. Here's the source for using Github as a db - I definitely see potential in reusing this approach for similar lightweight directory sites. Frontend re-frame has a delightful README that gracefully introduces you to reactive programming. If you haven't read it then stop reading this article and read that instead; I am 100% confident that it's a better use of your time. re-frame is a joy to use, and it's become my go-to tool for frontend projects. I've written a fair amount of code for working with forms and submitting values. This code forms a kind of nano-framework on top of re-frame, and it's allowed me to speed up the development process from project to project. Here's an example form which uses the form helpers. If you'd like for me to write a detailed explanation of this code, please leave a comment letting me know :) Boot deserves a special mention because it makes it so easy to develop ClojureScript apps. With Boot you get live reload (every time you change your frontend files, the changes are automatically injected into the browser), giving you a near-instantaneous feedback cycle that makes development much more enjoyable. It's also easy to incorporate Sass compilation - and your Sass files also get the live reload treatment. If you're not using Boot to build your ClojureScript app then chances are you're causing yourself undue suffering. And hey, maybe you're into that? No judgment. But if you're not, then take the time to learn it - it's a great investment in yourself as a Clojure developer. Deployment The infrastructure directory contains scripts for provisioning and deploying to a server with Ansible. It also has a Vagrantfile so that you can stand up a test server locally; just run vagrant up to both create the virtual machine and provision it. To deploy, cd into the infrastructure directory and run ./build.sh && ./deploy-dev.sh. Those two shell scripts are extemely simple, so take a look at them to see what's involved in deployment. The Ansible configuration is almost completely reusable. You can check it out in infrastructure/ansible. The roles directory in particular has Ansible tasks for creating an nginx server that can run Clojure app. It also includes an Upstart script for controlling startup/shutdown of your Clojure app (assuming you use Ubuntu). The Future Eventually, I'd like to add more tools for project discovery and for showing project stats. @yogthos, for example, suggested incorporating a project's stats from github so that users can see how active and popular each project is, information that can help them decide which project to contribute to. Also, I'm in the process of revamping braveclojure.com to include more resources for Clojure programmers. The site hovers around 3rd or 4th Google result for "Clojure", just below clojure.org and the Wikipedia entry. So, it's kind of popular I guess. My hope is that by featuring Open Source Clojure Projects, I'll help more new developers get involved and also help project maintainers squash some tickets.

Not a Superpredator

over 5 years ago | Daniel Higginbotham: Flying Machine Studios

I'm the son of a Vietnamese emigrant. I grew up in the South, where I experienced and witnessed racism. I have black step-siblings and my wife is black (naturally, so are my in-laws). As such, I try to educate myself about race issues in America. I don't share my thoughts much publicly but do want to write more on the subject. In the mean time, here's a (slight edited) exchange I had with a friend on Facebook about a recent event, Ashley Williams's protest during a Hillary Clinton speech. Me: A room full of white people at a $500/person Hillary Clinton event tells young black woman she's being rude and inappropriate for demanding Hillary Clinton explain why she supported the system that destroyed countless black and latino lives. I think that accurately sums up the prevailing attitude toward race in America: black people, please don't cause me mild discomfort by bringing my attention to our society's racial caste system. It's impolite. Friend: Wow. This thing seems like... a really, really huge distortion. She [Hillary] actually said: "You’re the first person to do that, **and I’m happy to address it."** She offered to talk about it. But the demonstrator wasn't there to talk, and wasn't in listening mode. Second, what she said, 20 years ago, is that she thought 'SuperPredators' exist and should be dealt with. It was a popular idea at the time. She did not say that they were one color or another: just that they exist and should be contained. WTF ??? It's indisputable that America needs to deal with the legacy of excessive force and excessive incarceration against people of color. But if we make it sound like any talk about incarcerating anybody, ever, must be inherently racist... we've already lost the battle to have serious conversations about what non-racially-biased policing would look like. Me: Hey man! Thanks for sharing your thoughts. Here are more of mine, and I hope they don't come across as combative - I really appreciate getting to talk about this stuff with someone as thoughtful as you. I wasn't actually commenting on Hillary's reaction, but the reactions of those in the (almost entirely white) room with her: people hissing at her and telling her she's being rude and inappropriate. That's what I'm referring to when I say the video reflects the prevailing attitude toward race. To me, it's as if the message toward the BLM activist [note: I've since learned she actually isn't with Black Lives Matter] is, "Excuse me, but you're interrupting the event we paid $500 to see. Please take your complaints about the racial caste system, which doesn't negatively affect me, elsewhere." And it is always "rude" or "inappropriate" or "inconvenient" for those topics to be discussed. When Beyonce dances at the Super Bowl, people lose their minds: they're just trying to enjoy football, they don't want to have to think about the lives of people who aren't like them! When another black man or woman gets murdered by police and people take to the streets demanding change, they're somehow offensive and ungrateful for criticizing law enforcement. Most often, the reaction is to denounce the method and ignore the message. Last year, Bernie Sanders visited Liberty University, and David Nasser, a university VP, got some of the biggest applause of the event for saying that the problem with the criminal justice system is a "sin issue, not a skin issue" - meaning that it was not a political issue, but one better addressed by encouraging people to be better or something. Same message: stop speaking up, black people. So that's what I was meaning to communicate originally. Here are my thoughts on Hillary. I'm not sure if you're familiar with "dog-whistle" politics, using coded language to court the racist vote. At the time, the Democratic party was trying to win back the vote of poor, Southern whites who'd switched to the Republican party thanks to the republicans' Southern Strategy, an effort to win those votes by appealing to racist fears and biases. Anyway, Democrats had the idea, "hey maybe if we also play on those racial fears, we'll get those votes back!" And that's what they did. When Hillary and other political/media figures talked about "superpredators", it was generally understood to mean "urban" people, and that was generally understood to mean black people. So when the demonstrator brings this up, she's not saying that "any talk about incarcerating anybody, ever" is inherently racist. She had reasons for singling out that phrase and using it to call attention to Hillary's record. Some might disagree, but I think her reasons were good ones. And even if "superpredator" wasn't meant that way, the result was still the crushingly unjust system we have today - and Hillary's support helped build that system. Using terms like "superpredator" and "bring them to heel" to accidentally perpetuate a racial caste system might even be worse. Michelle Alexander, author of The New Jim Crow, writes about it persuasively. The New Jim Crow goes into even greater detail about how mass incarceration and the criminal justice system are tools for racial control, even if many of the actors in the system (police, lawyers) aren't racist. I'm a Bernie fan, and I don't have rabid Hillary hatred. If it comes down to Hillary versus whichever clown the Republican party decides on, I will vote for Hillary. I respect her vast experience and think she would do a decent job at pushing for progress. When it comes to racial justice, though, I think Bernie has a much better record. Most of all, I'm an Ashley Williams fan. It takes courage to do what she did. Maybe she wasn't in listening mode (it's hard for me to really articulate my thoughts on this. She did give a good pause so that HC could speak. And anyway the power was so asymetrical that I don't know how to think about it. How do you really try to talk when you're being carted off by security?). But I'm glad she did it - it brought attention to an under-reported problem, and Hillary did in fact end up apologizing. And no damage was done, except perhaps to the sense of propriety of a few stuffy rich people.

Announcing Job Search Sanity

almost 6 years ago | Daniel Higginbotham: Flying Machine Studios

One of the least fun experiences in my life was getting fired twice in three months back in 2008 when the economy decided to go all YOLO and drive off a cliff with its middle finger waving out the window. The first time, I was a wreck. I was in a relationship with someone who had a chronic medical condition and the attendant medical bills. As the sole money earner, I was pretty bummed! Every day I felt dread and panic as I pored over job listings. I made many stress-induced mistakes like emailing the same contact twice or forgetting phone interviews. But somehow I found a job at a startup pretty quickly, only to be regrettably let go a couple months later because they were running out of money. This time, I did things differently. I took a Getting Things Done inspired approach to the job search. Approaching the search in an organized, methodical way made me feel in control. It also gave me confidence that I was making real progress on finding a job because I could actually see all the steps I was taking. At the end of the day I knew I had done what I could to move forward. Today I'm releasing a free web application, Job Search Sanity, to help others do the same. It helps you keep track of each job opportunity and the next actions you need to do to move forward. It also keeps cover letters and interview remindres in place. And one of its best features is that it lets you save your job searches on the site so you can keep them all in one place instead of visiting dozens of sites every day. If you're looking for a job and feel overwhelmed, then please give it a try! If you know someone who's looking for a job, please send them a link! Personal Notes The main thing that motivated me to create this site was seeing my brother struggle with finding employment after graduating with a Pharm.D. (doctor of pharmacy) degree last year. After spending a decade putting himself through school, he had the modest expectation of finding decent-paying, stable work. Instead, he found that the pharmacy market had radically changed while he was getting his degree, and jobs were scarce. I can only imagine how crushing it would feel to commit so much of your life (and accumulating large student loan debts) to learning profession, only to find that the rules had changed and there was actually little demand for your skills. To find that the future you've been working toward for so long has vanished, replaced with utter uncertainty. It was a rough time for him, but thankfully he did eventually find a job as a pharmacist. While he was looking, he came to live with me, and I was so glad that I was able to help in some way (and that I got to spend so much time with him), but part of me wished I could have done more. Cut to the present day, where my wife has been looking for a better job for a while, and I can see the stress affecting her in similar ways (though to a much lesser degree, thankfully). All of which is to say - I know that looking for a job can be incredibly stressful, and I hope that by providing this free app I can help people in similar situations. I kind of need to make something that I actually accept payments for soon, but for this site I didn't want their to be any barriers to signing up and using it. Perhaps I'll figure out way to monetize it, but for now I'm proud of the site and I'll count it a success if it helps even a dozen people as they find a job. Development Notes Job Search Sanity was built using: The Boot build tool. Boot made it easy to integrate a Clojure backend, ClojureScript frontend, and Sass compilation Liberator, to structure the handling of back-end API calls Datomic for the database re-frame as the lightweight front-end structure imposed on reagent I absolutely loved working on the site because the tools made it so much fun. One of my personal goals was to stretch myself design- and UX-wise, and the above programming tools made that possible by making it so simple and easy to implement the basic functionality. ClojureScript has come incredibly far in the past couple years. A year or two ago I looked at it but stuck with Coffeescript because cljs looked like a pain in the ass to use, but that isn't the case any longer. The only downside here is that, once again, Clojure has spoiled me, and I never want to go back to my previous tools.

Why Clojure's a Great Next Language for Rubyists

almost 6 years ago | Daniel Higginbotham: Flying Machine Studios

HAVE frothing at the mouth YOU flailing arms CONSIDERED eyes are unfocused and mad LISP flipping furniture over — @ra If you're a Rubyist itching to learn a new language, then I want to convince you (using only a minimum of mouth frothing and chair flipping) that Clojure's a great pick for you. Please excuse me if this article reeks of the glassy-eyed fanaticism of the true believer, but the fact is that I think Clojure is crazy stupid fun to use, and intellectually rewarding to boot. I fell in love with Ruby back in 2005, and since then only Clojure has been able to elicit the same level of affection. I think this is because Clojure has the same fundamental attributes that make Ruby so appealing: it's simple, it's powerful, and above all, it's fun. The rest of this love letter cough I mean, blog article, is about how Ruby and Clojure exhibit these attributes in very different ways, and how Clojure's different approach to computation makes it fascinating and rewarding to learn. My first exposure to Ruby was DHH's whoops-laden Ruby on Rails demo. That video saved me from the Lovecraftian horror that was PHP circa 2005. You may have had a similar experience, with Ruby rescuing you from C++ or Java or COBOL or whatever. It's a vast understatement to say that, compared to PHP, Ruby is elegant. Whereas PHP seemed almost to strive for inconsistency, Ruby employs only a few, simple ideas: everything is an object, you advance your program by sending messagse to objects, and message dispatch is governed by a couple easily-understood rules. Clojure is also a very simple language. It wouldn't be a stretch to say that simplicity is one of the core tenets of the Clojure philosophy; its creator, Rich Hickey, has given a number of talks elaborating on what it means for something to be simple, including Simple Made Easy and Clojure, Made Simple. Clojure's simplicity, however, takes a radically different form. In Clojure, you primarily care about two things: data and functions. You pass data to functions and get new data back. Yes, there's other interesting stuff like state management constructs and macros, just as there are other interesting aspects of Ruby like modules and blocks. But the heart of Clojure is so simple any eighth grade algebra student can understand it: hand some value to a function, and it hands you a new value back. "But wait a minute," you might be thinking, "You can't just pass data around like that. Haven't you read POODR? What about encapsulation? What if those functions mutate your data?" In Clojure, data is immutable. You can't change existing data, you can only derive new data; when you add 3 and 4, you don't change 3 or 4 to make 7, you derive 7. If this sounds crazypants and impossible to work with, all I can do is assure you that it does work (and point you to this book I wrote available free online, Clojure for the Brave and True which explains how) and hope that your intellectual curiosity will motivate you to explore this crazypants paradigm that lots of people seem to love. Here's a teaser: many of the problems addressed by object-oriented design simply aren't problems in Clojure. You don't have to care about inheritance vs. composition because behavior isn't chained to data. You don't have to care about data hiding because there's no risk that some object somewhere will silently change it. As a Rubyist, you know that simplicity buys you power, where power is defined as the ability to express computational ideas concisely. For example, Ruby's blocks allow you to write anonymous algorithms, something that's impossible in C. Another example: Ruby's simple method dispatch scheme makes it easy for you to reason about and design robust programs. It lets you easily extend code and buys you reuse through multiple inheritance. Most enticingly, it lets you do metaprogramming. Clojure's simplicity also buys you power. As a Lisp, Clojure employs a syntax which just so happens to mirror the internal abstract syntax trees that represent Clojure code. I'm going to wave my hands a bit here and say that this lets you use macros, the most powerful metaprogramming tool available to any language. In the Ruby community, we like to talk about code that writes code; macros take this idea to a whole different level. It's not an exaggeration to say that Ruby's metaprogramming system provides only a subset of the capability provided by Lisp macros. Super duper hand wavy, I know, but the point is: if you like how powerful Ruby is, you will lose your flipping mind over how powerful Clojure is. Macros let you easily implement ideas from all corners of computer science. It's not an accident that Clojure has so many libraries that let you use different paradigms. There's core.logic for logic programming, instaparse for building parser generators, core.typed for haskell-ish typed programming, and much more. And macros aren't even the coolest part. I'll let you in on a secret: macros are the flashy, shiny bait that wild-eyed Clojurists lay out to ensnare curious programmers. Nay, even more powerful is Clojure's design of programming to abstractions. This design makes it incredibly easy to reuse your code and extend existing code. Of course I can't tell you how in the short space afforded by this digital outburst, but you'll grow as a programmer if you learn about it. These last two paragraphs also hint at what makes Clojure so dadgum fun. First, by learning Clojure you get introduced to one of the most loved and respected computational traditions, Lisp and the lambda calculus. It also has a clear philosophy, articulated by Rich Hickey, that will make you a better programmer. Not only that, other programming vistas become more readily accessible. Second, Clojure is a joy to write. Here's a quote from Matz (Ruby's creator) on his design philosophy: By using Ruby, I want to concentrate the things I do, not the magical rules of the language, like starting with public void something something something to say, "print hello world." I just want to say, "print this!" I don't want all the surrounding magic keywords. I just want to concentrate on the task. That's the basic idea. So I have tried to make Ruby code concise and succinct. Clojure, likewise, is concise. It lets you focus on solving the problem at hand rather than figuring out the magic whatever. People are falling in love with Clojure, spending their nights and weekends learning it hacking with it, because the experience is its own reward. If you're looking to stretch yourself and explore more of what the world of programming has to offer, then I recommend Clojure to you with all of my crazed heart. Here are some resources for getting started: Clojure Distilled, a very short overview of the language Clojure for the Brave and True, my own full-length crazypants introductory book Living Clojure, an excellent introductory book by Carin Meier, someone who BUILDS ROBOTS LispCasts for those of you who prefer video The Rich Hickey FanClub lists all his talks Now go out there and start learning some Clojure!

Looking for Beta Testers

almost 6 years ago | Daniel Higginbotham: Flying Machine Studios

Over the last couple months I've been working on a web site that helps job seekers reduce stress by following a structured, methodical process, and the site's ready for some beta testing. If you'd like to help out, please sign up for the beta test! This isn't a unicorn-disrupting once-a-generation app, just a simple site to help people during a stressful process. For the beta test, I'm just looking for feedback on what's broken, what's confusing, and what could be useful but isn't implemented. Thanks! P.S. in case you're wondering, the site's all Clojure, with compojure/liberator/datomic on the backend and reagent and re-frame on the frontend.

Why I wrote Clojure for the Brave and True

almost 6 years ago | Daniel Higginbotham: Flying Machine Studios

It seems weird to say that I'm almost painfully filled with gratitude for all the good shit in my life, and that that was what motivated me to start writing a free programming book, but that's the truth. Before I started writing Clojure for the Brave and True back in 2013, I had been deliberately trying to figure out what I could do to improve other people's lives even just a smidge. By that point I had been doing a gratitude practice (just spending a few minutes to think of three things I was grateful for) regularly for a year and a half, and felt that my life was so full with good shit that I felt almost obligated to do something for others so that they'd have more good shit in their lives. "Obligation" might seem strange here, and it's not 100% what I mean. It's hard to accurately put the feeling into words. Someday I'd like to take the time to do a good job writing about it, but for now - My life experiences have instilled in me a belief that kindness and generosity are obligations. To take just one experience: for years, I witnessed firsthand the kind of suffering that chronic illness can inflict on someone. That was just nature doing it's thang, cruel, impersonal, implacable. But it was compounded by the constant minor (and sometimes major) abuses delivered by people. Life is painful enough without humans heaping additional indignities on each other. Beyond this baseline of "don't be an asshole" (incidentally, that's the central tenet of my personal ethics), I think it's imperative to learn to live with kindness, love, and generosity. The bleak perspective on this is that the ship is going down for all of us, and the only thing we have to hold on to is each other. The not-bleak perspective is that we are social creatures, wired to find fulfillment in contributing to each others' happiness; that our greatest joys are found in human connection. Which is why I wrote a programming book featuring hobbit violence, man-eating trolls, and mopey teenage vampires! Really, though, programming is a huge part of why my life is so full of good shit. I grew up living in poverty, and now I make a very comfortable living doing an activity that I genuinely enjoy. I have great coworkers, good health insurance, I live in a good house and have a decent car, and I'm confident I can provide for my family. I even have enough free time for hobbies and friends. So, I wanted to provide something free to the world to help others improve their lives. Learning Clojure has personally made me a much better programmer, and I think others can benefit from it just as much. In learning Clojure, you learn a new way of thinking about programming. It's intellectually rewarding and it has immense practical value; it makes your life as a programmer a lot easier. Plus, there's something romantic about learning a language that carries on a 60-year legacy. Like you're engaging in a tradition that extends far beyond you. And it's fun! Clojure feels like something you'd want to use in your spare time, for fun, but what's better is that you can actually get paid to use it. The other main reason I wrote the book was that I needed some way to tell the world about the were-Simmons. The danger is real! Wake up, sheeple! were-Simmons

Brave and New

almost 6 years ago | Daniel Higginbotham: Flying Machine Studios

Clojure for the Brave and True is now available in print! You can use the coupon code ZOMBIEHUGS to get 30% off at No Starch (plus you'll get a free sticker), or buy it from Amazon. The web site has been updated, too! (Don't forget to force refresh.) One of the reasons I went with No Starch as a publisher was that they supported the idea of keeping the entire book available for free online. It makes me super happy to release the professionally-edited, even better book for free. I hope it makes you laugh, cry, and give up on object-oriented programming forever. Writing this book was one of the most ambitious projects of my life, and I appreciate all the support I've gotten from friends, family, and readers like you. Thank you from the bottom of my crusty heart! Is that it, then? Is the journey over? No! It's not! In fact, back in April I quit my job in part so that I'd have more time for writing. I have some new Clojure articles on reducers and transducers in the works, plus I plan to write about web development again and dive into fun libraries like core.logic, clara, and much more. There's just so much cool stuff to learn about! If you'd like me to let you know when they're published then follow me on twitter @nonrecursive or join the 1600+ member mailing list: Lastly, to celebrate the book release, my wife (the book's illustrator) put together a t-shirt and mug store! The gear features the dwarf and warpig combo on the book cover :) This run only lasts one week - future runs will feature other illustrations from the book or from new projects. The money from that will help me continue avoiding full-time employment so that I can keep creating fun, high-quality Clojure content! Thank you, and have fun Clojuring! Brave and True [Update] I got asked for a list of the major differences. Here they are: Illustrations! Almost every chapter now has exercises The first macro chapter, Read and Eval, is massively improved. I'm hoping this will gives readers an excellent conceptual foundation for working with macros There's now a joke about melting faces There used to be two Emacs chapters (basic emacs and using Emacs for Clojure dev), now there's just one The concurrency chapter got split into two chapters Appendices on Leiningen and Boot were added The "Do Things" chapter is much friendlier I spend a lot more time explaining some of the more obscure topics, like lazy sequences. Many of the chapters got massive overhauls. The functional programming chapter, for example, was turned completely inside out, and the result is that it's much, much clearer Overall, everything should be clearer

Review of Clojure Applied

almost 6 years ago | Daniel Higginbotham: Flying Machine Studios

A while back, Clojure community czar and all around good guy Alex Miller asked me to review a beta version of his nearly complete book Clojure Applied, co-authored with Ben Vandgrift. This was great news for me because I was planning on buying it, but this way I would get it for free. Delighted, I downloaded my free copy, added a "review book" item to my todo list, then merrily ignored it until a couple weeks ago when I saw Alex in person. "Sorry I haven't gotten around to reviewing your book. I'm super excited about it!" I gushed at him guiltily. "What's the final release date?" Very politely, he responded, "It's already released." It turns out, the resulting dose of healthy embarrassment was all I needed to actually get off my ass and read and review the book! And I am glad I did, because it's a good one. Clojure Applied is aimed at people who know Clojure basics and want to learn how to write idiomatic code and create production applications, and if this describes you, then you should get it. In case you need more convincing, the rest of this post goes on about how good the book is. What I like most about Clojure Applied is that it's concise and fast-paced. It doesn't hold your hand, blathering on about basic details that you learned months ago. At the same time, it's very approachable, using straightforward and brisk language. It even has a few fun moments (two words: spaghetti tacos). To use a cliche that offends my vegetarian sensibilities, it's all meat and no filler, which is perfect for programmers who are... uh... hungry for knowledge. The most valuable parts of the book are the sections on building a system and testing. Decomposing a your program into a system of interacting components that manage their own state isn't a topic covered by beginner books, and it's very helpful. It's easy to write programs whose architectures go against Clojure's grain of functional programming, immutability, and state management, but Clojure Applied shows you how to do it right. On the testing front, Clojure Applied is nice because it covers some of the latest additions to the testing ecosystem, like test.check. Another nice aspect of the book is that it introduces you to a cornucopia of useful standard library and Java functions and constructs, like the juxt function, the reduced function, persistent queues, thread pools with the executor service, etc etc etc. All in all, it's a good book. If you're looking for a good second book on Clojure, then Clojure Applied is an excellent choice!

Clojure for the Brave and True Is Off to the Printers!

about 6 years ago | Daniel Higginbotham: Flying Machine Studios

Today, my fantastic production editor at No Starch reached out and pressed the basketball-sized "RELEASE IT!" button at No Starch headquarters, encasing Clojure for the Brave and True cucumber-infused amber and sending it on its way to the bookeries of Melrose Park, Illinois. At least, I'm pretty sure that's what happend. Either way, the book is off to the printers! It's done! After two years of working on it, it's actually done! Aaaaaah! It'll be available at the beginning of September, and you can preorder it from No Starch (free ebook included with print book) or Amazon. (Leanpubbers, you'll be getting the fully-updated ebook by the time the book is published.) The main work (writing a book) is done, but there's actually still more stuff to do. I need to finish rebuilding the website to include all the updated content and to make it easier to add more content. What?!, you ask, more content?! Yes indeedy! The book may be finished, but there's still so much more to write about! Expect a Sidequests series by the end of the year. Thanks for all the pull requests, emails, comments, and other feedback. And I don't want to spoil the Acknowledgements section, but I want to give my Special Thanks to Jess Higginbotham, Alan Dipert, Bridget Hillyer, Joe Jackson, Pat Shaughnessy, Alex Rothenberg, Thomas Newton, Jalil Fanaian, Chris Parker, Christian Lilley, and Mark Daggett. I hope you're having a great day. I am! Now if you'll excuse me, I'm going to go rediscover what it feels like to have free time. Sock Gnome

Brave and True Early Access

over 6 years ago | Daniel Higginbotham: Flying Machine Studios

Guess what! You can now start reading the fully-revised Clojure for the Brave and True through No Starch Press’s early access program! Use the coupon code ZOMBIEHUGS for 30% off! You’ll get the first four chapters, complete with fun new illustrations and professional editing. The fifth chapter will be up soon, and the final release is scheduled for June 25th. It’s also a way to give feedback (which I will appreciate very much!) before it’s cast in adamantium for printing, or whatever it is you do to books to print them these days. If you’re enjoying the book, I’d love it if you shared this blog post or tweeted about the early access :) I’m really happy with how the book has turned out. The No Starch editors have been awesome at helping to improve the clarity of the book, making it even more beginner-friendly. Most of the chapters have seen major rewrites, with significant improvements to the structure and content. I also appreciate all the comments, emails, and pull requests I’ve gotten. It warms the cockles of my heart to receive such generosity. I think this book will be useful for Clojure beginners, and I’m especially happy that it will remain available for free online. (I’ll be updating the site with the new book once it’s published, i.e. when I actually have time again for the first time in 2 years :) I’m grateful for all the work the Clojure community has put into creating such a delightful language and useful ecosystem, and I’m glad to make my own little contribution. I hope you enjoy it! Clojure for the Brave and True

Boot, the Fancy New Clojure Build Framework

over 6 years ago | Daniel Higginbotham: Flying Machine Studios

Build tools are known to inspire the entire gamut of emotions from bored impatience to Homeric rage (I'm looking at you, Grunt). Personally, I've never given them much thought; they've always seemed like tedious overhead, an unfortunate necessity for getting real work done. Recently, though, I've started learning about Boot, and I've found that build programming can actually be interesting. This article will explain Boot's underlying concepts and guide you through writing your first Boot tasks. If you're interested in using Boot to build projects right this second, then check out its github README and its wiki. Boot's Abstractions Created by Micha Niskin and Alan Dipert, Boot is a completely controversy-free addition to the Clojure tooling landscape. On the surface, it's "merely" a convenient way to build Clojure applications and run Clojure tasks from the command line. But dig a little deeper and you'll see that Boot is like the lisped-up lovechild of Git and Unix in that it provides abstractions that make it much more pleasant to write code that exists at the intersection of your operating system and your application. Unix provides abstractions that we're all familiar with to the point of taking them for granted. (I mean, would it kill you to take your computer out to a nice restaurant once in awhile?) The process abstraction allows you to reason about programs as isolated units of logic that can be easily composed into a stream-processing pipeline through the STDIN and STDOUT file descriptors. These abstractions make certain kinds of operations, like text processing, very easy. Similarly, Boot provides abstractions that make it actually pleasant to compose independent operations into the kinds of complex, coordinated operations that build tools end up doing, like converting Clojurescript into Javascript. Boot's task abstraction lets you easily define units of logic that communicate through filesets. The fileset abstraction keeps track of the evolving build context and it provides a well-defined, reliable method of task coordination, as opposed to the ill-defined, ad-hoc task coordination which programmers have to impose on other build tools. That's a lot of high-level description, which hopefully is great for when you want to hook someone's attention, which hopefully I have now done. But I would be ashamed to leave you with a plateful of metaphors. Oh no, dear reader; that was only the appetizer. For the rest of this article you will learn what that word salad means by building your own Boot tasks. Along the way, you'll discover that build tools can actually have a conceptual foundation. Tasks Like make, rake, grunt, and other build tools of yore, Boot lets you define tasks. Tasks are named operations that take command line options dispatched by some intermediary program (make, rake, Boot) Boot provides the dispatching program, boot, and a Clojure library that makes it easy for you to define named operations and their command line options with the deftask macro. So that you can see what all the fuss is about, it's time to create your first task. Normally, programming tutorials encourage have you write something that prints "Hello World", but I like my examples to have real-world utility, so your task is going to print "My pants are on fire!", information which is objectively more useful. First, install Boot, then create a new directory named boot-walkthrough, navigate to that directory, and finally create a file named build.boot and put in this in it: (deftask fire "Prints 'My pants are on fire!'" [] (println "My pants are on fire!")) Now run this task with boot fire; you should see the message you wrote printed to your terminal. This demonstrates two out of the three task components - the task is named (fire) and it's dispatched by boot. This is already super cool – you've essentially created a Clojure script, standalone Clojure code that you can easily run from the command line. No project.clj or directory structure or namespaces needed! Let's extend the example to demonstrate how you'd write command line options: (deftask fire "Announces that something is on fire" [t thing THING str "The thing that's on fire" p pluralize bool "Whether to pluralize"] (let [verb (if pluralize "are" "is")] (println "My" thing verb "on fire!"))) Try running the task like so: boot fire -t heart # => My heart is on fire! boot fire -t logs -p # => My logs are on fire! In the first instance, either you're newly in love or you need to be rushed to the emergency room. In the second, you are a boy scout awkwardly exclaiming your excitement over meeting the requirements for a merit badge. In both instances, you were able to easily specify options for the task. This refinement of the fire task introduced two command line options, thing and pluralize. These options are defined using the options DSL. In the case of thing, t specifies the option's short name and thing specifies the long name. THING is a little complicated, and I'll get to it in a second. str specifies the option's type, and Boot uses that both to validate the argument and convert it. "The thing that's on fire" is the documentation for the option. You can view a task's documentation with boot task-name -h: boot fire -h # Announces that something is on fire # # Options: # -h, --help Print this help info. # -t, --thing THING Set the thing that's on fire to THING. # -p, --pluralize Whether to pluralize Pretty groovy! Boot makes it very, very easy to write code that's meant to be invoked from the command line. Now, about THING. THING is an optarg and it indicates that this option expects an argument. You don't have to include an optarg when you're defining an option (notice that the pluralize option has no optarg). The optarg doesn't have to correspond to the full name of the option; you could replace THING with BILLY_JOEL or whatever you want and the task would work the same. Finally, you can also designate complex options using the optarg. (That link will take you to Boot's documentation on the subject.) Basically, complex options allow you to specify that option arguments should be treated as as maps, sets, vectors, or even nested collections. It's pretty powerful. Boot provides you with all the tools you could ask for in building command-line interfaces with Clojure. And you've only just started learning about it! The REPL Boot comes with a good number of useful built-in tasks, including a REPL task; run boot repl to fire up that puppy. The Boot REPL is similar to Leiningen's in that it handles loading your project code so that you can play around with it. You might not think this applies to the project you've been writing because you've only written tasks, but you can actually run tasks in the REPL (I've left out the boot.user=> prompt): ;; pass arguments as flags (fire "-t" "NBA Jam guy") ; My NBA Jam guy is on fire! ;=> nil ;; or as keywords (fire :thing "NBA Jam guy") ; My NBA Jam guy is on fire! ;=> nil (fire "-p" "-t" "NBA Jam guys") ; My NBA Jam guys are on fire! ;=> nil (fire :pluralize true :thing "NBA Jam guys") ; My NBA Jam guys are on fire! ;=> nil And of course, you can also use deftask in the REPL – it's just Clojure, after all. The takeaway is that Boot lets you interact with your tasks as Clojure functions, because that's what they are. Composition and Coordination If what you've seen so far was all that Boot had to offer, it'd be a pretty swell tool, though not very different from other build tools. One thing that sets Boot apart, though, is how it lets you compose tasks. For comparison's sake, here's an example Rake invocation (Rake is the premier Ruby build tool): rake db:create db:migrate db:seed In case you were wondering, this will create a database, run migrations on it, and populate it with seed data when run in a Rails project. What's worth noting, however, is that Rake doesn't provide any way for these tasks to communicate with each other. Specifying multiple tasks is just a convenience, saving you from having to run rake db:create; rake db:migrate; rake db:seed. If you want to access the result of Task A within Task B, the build tool doesn't help you; you have to manage that coordination yourself. Usually, you'll do this by shoving the result of Task A into a special place on the filesystem, and then making sure Task B reads that special place. This looks like programming with mutable, global variables, and it's just as brittle. Handlers and Middleware Boot addresses this problem by treating tasks as middleware factories. If you're familiar with Ring, Boot's tasks work very similarly; feel free to skip to the next section. If you're not familiar with the concept of middleware, then allow me to explain! First, the term middleware refers to a set of conventions that programmers adhere to so that they can flexibly create domain-specific function pipelines. That's pretty dense, so let's un-dense it. I'll go over the flexible part in this section, and cover domain-specific in the next. To understand how the middleware approach differs from run-of-the-mill function composition, here's an example of composing everyday functions: (def strinc (comp str inc)) (strinc 3) ; => "4" There's nothing interesting about this function composition. This function composition is so unremarkable that it strains my abilities as a writer to try and actually say anything about it. There are two functions, each doing its own thing, and now they've been been composed into one. Whoop-dee-do! Middleware introduce an extra step to function composition, and this gives you more flexibility in defining your function pipeline. Suppose, in the example above, that you wanted to return "I don't like the number X" for arbitrary numbers, but still return the stringified number for everything else. Here's how you could do that: (defn whiney-str [rejects] {:pre [(set? rejects)]} (fn [x] (if (rejects x) (str "I don't like " x) (str x)))) (def whiney-strinc (comp (whiney-str #{2}) inc)) (whiney-strinc 1) ; => "I don't like 2 :'(" Now let's take it one step further. What if you want to decide whether or not to call inc in the first place? Here's how you could do that: (defn whiney-middleware [next-handler rejects] {:pre [(set? rejects)]} (fn [x] (if (= x 1) ; ~1~ "I'm not going to bother doing anything to that" (let [y (next-handler x)] (if (rejects y) (str "I don't like " y " :'(") (str y)))))) (def whiney-strinc (whiney-middleware inc #{3})) Here, instead of using comp to create your function pipeline, you pass the next function in the pipeline as the first argument to the middleware function. In this case, you're passing inc as the first argument to whiney-middleware. whiney-middleware then returns an anonymous functions which closes over inc and has the ability to choose whether to call it or not. You can see this at ~1~. We say that middleware take a handler as their first argument, and return a handler. In the example above, whiney-middleware takes a handler as its first argument, inc here, and it returns another handler, the anonymous function with x as its only argument. Middleware can also take extra arguments, like rejects, that act as configuration. The result is that the handler returned by the middleware can behave more flexibly (thanks to configuration) and it has more control over the function pipeline (because it can choose whether or not to call the next handler). Tasks are Middleware Factories Boot takes this pattern one step further by separating middleware configuration from handler creation. First, you create a function which takes n many configuration arguments. This is the middleware factory and it returns a middleware function. The middleware function expects one argument, the next handler, and it returns a handler, just like in the example above. Here's a whiney middleware factory: (defn whiney-middleware-factory [rejects] {:pre [(set? rejects)]} (fn [handler] (fn [x] (if (= x 1) "I'm not going to bother doing anything to that" (let [y (handler x)] (if (rejects y) (str "I don't like " y " :'(") (str y))))))) (def whiney-strinc ((whiney-middleware-factory #{3}) inc)) As you can see, it's nearly identical to the previous example. The change is that the topmost function, whiney-middleware-factory, now only accepts one argument, rejects. It returns an anonymous function, the middleware, which expects one argument, a handler. The rest of the code is the same. In Boot, tasks can act as middleware factories. In fact, they usually do, I just didn't present them that way above in order to keep things simple. To show this, let's split the fire task into two tasks: what and fire. what will let you specify an object and whether it's plural, and fire will announce that it's on fire. This is great, modular software engineering because it allows you to add other tasks like gnomes, to announce that a thing is being overrun with gnomes, which is just as objectively useful. (Exercise for the reader: create the gnome task.) (deftask what "Specify a thing" [t thing THING str "An object" p pluralize bool "Whether to pluralize"] (fn middleware [next-handler] (fn handler [_] (next-handler {:thing thing :pluralize pluralize})))) (deftask fire "Announce a thing is on fire" [] (fn middleware [next-handler] (fn handler [thing-map] (let [updated-thing-map (next-handler thing-map) verb (if (:pluralize thing-map) "are" "is")] (println "My" (:thing thing-map) verb "on fire!"))))) Here's how you'd run this on the command line: boot what -t "pants" -p -- fire And here's how you'd run it in the REPL: (boot (what :thing "pants" :pluralize true) (fire)) Wait a minute, what's that boot call doing there? In Micha's words, "The boot macro takes care of setup and cleanup (creating the initial fileset, stopping servers started by tasks, things like that). Tasks are functions so you can call them directly, but if they use the fileset they will fail unless you call them via the boot macro." Wait a minute, what's a fileset? Filesets I mentioned earlier that middleware are for creating domain-specific function pipelines. All that means is that each handler expects to receive domain-specific data, and returns domain-specific data. With Ring, for example, each handler expects to receive a request map representing the HTTP request. This might look something like: {:server-port 80 :request-method :get :scheme :http} Each handler can choose to modify this request map in some way before passing it on to the next handler, say by adding a :params key with a nice Clojure map of all query string and POST parameters. Ring handlers return a response map, which consists of the keys :status, :headers, and :body, and once again each handler can transform this data in some way before returning it to its parent handler. In Boot, each handlers receives and returns a fileset. The fileset abstraction gives you a way to treat files on your filesystem as immutable data, and this is a great innovation for build tools because building projects is so file-centric. For example, your project might need to place temporary, intermediary files on the filesystem. Usually, with build tools that aren't Boot, these files get placed in some specially-named place, say, project/target/tmp. The problem with this is that project/target/tmp is effectively a global variable, and other tasks can accidentally muck it up. The fileset abstraction works by adding a layer of indirection on top of the filesystem. Let's say Task A creates File X and tells the fileset to store it. Behind the scenes, the fileset stores the file in an anonymous, temporary directory. The fileset then gets passed to Task B, and Task B modifies File X and asks the fileset to store the result. Behind the scenes, a new file, File Y, is created and stored, but File X remains untouched. In Task B, an updated fileset is returned. This is the equivalent of doing assoc-in with a map; Task A can still access the original fileset and the files it references. The mechanics of working with filesets are all explained in the fileset wiki, but I hope this gives a good conceptual overview! Everything else The point of this article was to explain the concepts behind Boot. However, it also has a bunch of features, like set-env! and task-options! that make life easier when you're actually using it. It does amazing magical things like providing classpath isolation so that you can run multiple projects using one JVM and letting you add new dependencies to your project without having to restart your REPL. If Boot tickles your fancy, check out its README for more info on real-world usage. Also, its wiki provides top-notch documentation. If you're new to Clojure, then check out Clojure for the Brave and True, an introduction to the language written by yours truly. Have fun!

The core.async Chapter is Up

almost 7 years ago | Daniel Higginbotham: Flying Machine Studios

The first draft of Mastering Concurrent Processes with core.async is now up on Clojure for the Brave and True. Here's a preview: One day, while you are walking down the street, you will be surprised, intrigued, and a little disgusted to discover a hot dog vending machine. Your scalp tingling with guilty curiosity, you won't be able to help yourself from pulling out three Sacagawea dollars and seeing if this contraption actually works. After accepting your money, it will emit a click and a whirr, and out will pop a fresh hot dog, bun and all. Enjoy!

Vern, a Library For Database Fixtures

almost 7 years ago | Daniel Higginbotham: Flying Machine Studios

At work, I've been using yesql and wanted a way to compactly describe some seed data. My first attempt was an utter abomination, but I think I've worked out something pleasant that could be useful to other people. vern is a 54-line project that I'd love to get feedback on before publishing. It's named after one of my cats, this guy: vern Code Here's an example of vern in use: (def fixtures [:endpoints [:wallace {:name "wallace" :id 1} :gromit {:name "gromit" :id 2}] :permissions [;; these permissions belong to wallace [{:endpoint [:endpoints :wallace]} :wallace-read {:name "read" :id 10} :wallace-write {:name "write" :id 11}] ;; these permissions belong to gromit [{:endpoint [:endpoints :gromit]} :gromit-read {:name "read" :id 12} :gromit-write {:name "write" :id 13}]]]) (def entities (atom [])) (do-named (fn [processed group-name entity] (swap! entities #(conj % (:data entity))) (get-in entity [:data :id])) fixtures) @entities ; => [{:id 1 :name "wallace"} {:id 2 :name "gromit"} {:id 10 :endpoint 1 :name "read"} {:id 11 :endpoint 1 :name "write"} {:id 12 :endpoint 2 :name "read"} {:id 13 :endpoint 2 :name "write"}] First, we assign the fixture the colorful name data, then create a "database", which in this case is just an atom holding an empty vector. do-named is where the real action happens. processed is a map of all the named entities that have been processed so far. For example, after processing the first two entities, the value of processed will be {:endpoints {:wallace 1 :gromit 2}} Most likely you won't need it but it's there in case you do. group-name will be :endpoints and then :permissions in this example. In real life, I use this to determine which table I need to insert the entity's data into. entity will be something like {:name :wallace :data {:name "wallace" :id 1}} In the example, you're just using conjing (:data entity) onto the entities atom's vector. Here's what I do in real life, though: (defn load-fixtures [db fixtures] (d/do-named (fn [processed group-name entity] (:generated_key (q/insert! group-name db (:data entity)))) fixtures)) This uses yesql's functionality to insert a row in a database and return the value of the key that was generated for every entity in the fixture. The reason you want to return the generated key is so that you can refer to this record in later entities. If a subsequent entity is defined as {:name "belongs-to-wallace" :endpoint-id [:endpoints :wallace]} then do-named uses the corresponding value in the processed map. When do-named sends the first "permission" entity to its function, the entity looks like this: {:name :wallace-read :data {:name "read" :id 10 :endpoint 1}} The last feature is that you can group common attributes together in a vector. You can see this in the permissions: [[{:endpoint [:endpoints :wallace]} :wallace-read {:name "read" :id 10} :wallace-write {:name "write" :id 11}]] You could have written these entities as follows: [:wallace-read {:name "read" :id 10 :endpoint [:endpoints :wallace]} :wallace-write {:name "write" :id 11 :endpoint [:endpoints :wallace]}] But that kind of repetition can pretty old pretty quickly. Here's the general form of the fixtures: [:entity-group-key [:entity-1-key ;; entity keys are optional {:name "zip"} ;; using a sequential structure allows you to reference ;; another entity {:parent-id [:entity-group-key :entity-1-key] :name "zip's child 1"} ;; you can define common associations by grouping entities in a ;; sequential; the first element contains the common associations [{:parent-id [:entity-group-key :entity-1-key]} {:name "zip's child 2"} {:name "zip's child 3"}]] :entity-group-2 [{:entity-group-1-id [:entity-group-key :entity-1-key]}]] Request for Feedback This is a tiny library, and I hope it might yield a couple fun rounds of code golf. In particular, I have the sneaking suspicion that I'm neglecting some useful functions from Clojure's standard library. I considered using zip but that seemed too heavyweight. Also, I think I could just pass the entity's data to the do-named function. I'm also curious if there's a name for this kind of pattern, where you aren't solely processing one sequence item at a time but care about what you last processed as well (this is how names get assigned to entities).

Clojure Conj 2014 - Thank Yous, Noir Recommendations, Personal Highlights

almost 7 years ago | Daniel Higginbotham: Flying Machine Studios

This year's Clojure Conj was a complete blast! I got to meet and reconnect with great, friendly, brilliant people, and had enormous fun the whole time. And by the way - if you were hoping to get a sticker, please email me at daniel@flyingmachinestudios.com and let me know how I can get one to you! The main thing I want to say is THANK YOU SO MUCH to everyone who went out of their way to share their appreciation for Clojure for the Brave and True; I know how easy it is to feel nervous about approaching a stranger! The book is a labor of love, an attempt at both expressing myself creatively and bringing joy to others through entertainment and education, and it is immensely gratifying to hear that my efforts are paying off. Putting myself out there over and over with each chapter and revision is nervewracking as hell, but y'all really make me feel like it's worth it. I have to admit, though, that I'm finding it hard to adjust back to normal life. Why aren't the other people waiting in this airport with me coming up and thanking me? Don't they know who I am!?!? Seriously, though - thanks everybody, and I hope you're enjoying your Clojure journey :) Second, I'm grateful to Alex Miller (@puredanger), Cognitect, and all of the excellent speakers for creating such a great experience for everyone. Send them your thanks, and go watch the talks at the ClojureTV YouTube channel. Did anyone else freaking love the crime museum? What an awesome choice for a party venue! If you're into noir/hardboiled entertainment, here are some recommendations: True Detective - I am completely nuts over the first season. The acting is superb, and the storytelling is some of the best you'll see in any TV show or movie. It actually ruined all TV for me - it was out around the same time as The Walking Dead, and it made The Walking Dead feel clunky and boring. This is the gold standard for noir entertainment. Sleeper, a gritty graphic novel by Ed Brubaker. Also check out his "Fatale" and "Criminal" series. His works are suspenseful and heartbreaking, with great art and great storytelling. Hard Magic / The Grimnoir Chronicles Frankly, I expected this book to be complete trash, but man did it grab me! I came away incredibly impressed at the author's ability to craft an action-packed novel that actually had depth with prose that exceeds most literary works. Now, normally I try to write stuff that's useful for readers, but the rest of this post is essentially fanboi raving because OMG I got to meet Bozhidar Batsov! Bozhidar is the Emacs Knight we need, and holy crap I got a private CIDER lesson from him. Expect an article on all the cool CIDER features you're not using in the near future. Other fanboi highlights: Getting to real-life meet Mike Marsh, one of the most prolific posters on Grateful Place and a thoughtful, smart, and genuine good guy. Meeting Brian Jennings, the man responsible for Emacs 24.4's lose-focus hooks. He gets about a million geek points for persuading the Emacs team, including and especially RMS, that the feature was a good idea. How cool is that? Totally bombing one of my programming jokes with a Clojure Olympian. That gets me geek points, right? Right? Eh? Eh? Hearing about a cool swarm intelligence project from a college hacker, Anthony, who also happens to be an awesome dude Spending time with Isaac Praveen and the rest of the Sparx team. Thanks for the drink, and thanks for the company! These guys are doing cool stuff and they are hiring Clojure devs. Meeting Angela Harms and Jason Felice, my new Buddha buddies. I'm hoping to see a lot of cool stuff at Maitria :D Talking to Johann Bestowrous (@opinionsonline) - do yourself a big favor and follow him on twitter, he is a funny dude Getting a sneak peek at the next version of Hostel Rocket. These guys are also hiring Clojure Devs, and they have a really cool product. It was great hanging out with folks, and I hope everybody had a safe trip home!

tmux in 5 Minutes

almost 7 years ago | Daniel Higginbotham: Flying Machine Studios

If you work on projects that require you to open multiple terminal tabs, then tmux (and its super buddy, tmuxinator) will help you be more productive! tmux allows you to run multiple sessions in one terminal, and tmuxinator allows you to save tmux configurations. For example, if you're a Rails developer, you could easily open separate terminal sessions for running a Rails server, a Rails console, and tailing logs. When working on a Clojure-based forum I have four terminal sessions running: a shell, a grunt server building the frontend, datomic, and a shell for deployments: tmuxinator.png In order to start all this up, I only have to run one command: mux ath. This is much more convenient than trying to remember which services I need and manually starting each one up. Below are instructions for getting started with tmux and tmuxinator. First, install tmux using the instructions (for mac users) in this gist. Next, install tmuxinator using gem install tmuxinator Create your first tmuxinator config file under ~/.tmuxinator/sample.yml. Its contents should look like this, where the command under server is whatever's appropriate for your environment: name: sample root: ~/path/to/your/project pre: git pull windows: - shell: - server: bundle exec rails s The pre option runs that command in the root directory before trying to open any "windows". (I think of windows as tmux's "virtual tabs"). You can then start this tmux session with "mux sample". To navigate back and forth between the windows, use C-b n for "next window", and C-b p for "previous window". C-b means "hold down control and hit the 'b' key". To leave a tmux session, you use the key binding C-b d. If you leave the session, it's still actually running; any process you started in your windows is still executing. I rarerly ever use other commands, but if you need more, here's a tmux cheatsheet. To completely end a tmux session, you have to kill it. To do that, you run the command tmux kill-session -t sample, where "sample" is the name option in your tmuxinator config. I've created an alias for this, alias "tmk"="tmux kill-session -t". That way I only have to type tmk sample. I hope you find this useful! For more information, you can check out tmuxinator's github repo. You can do some pretty crazy stuff, like split your terminal into multiple panes. There's also a a handy book on tmuxinator available from the Pragmatic Programmers if you want to really go nuts with it. I hope you've found this tip useful!

Clojurers to Follow

almost 7 years ago | Daniel Higginbotham: Flying Machine Studios

I follow all these Clojurers (Clojurians? Clojurists? Clojeoisie?) because their tweets are interesting and useful. If you're into Clojure, then I'm sure you'll want to follow them, too: Bridget Hillyer (@BridgetHiller) Bridget is one of the main forces behind ClojureBridge, an organization which teaches free Clojure workshops to women. She's also one of the coolest people I know, and I'm proud to call her a friend! Alan Dipert (@alandipert) Alan is the smartest person I know, and I'm proud to call him a friend, too. He's also super funny and nice, which seems unfair to me. Everything he writes will learn you good. I haven't discussed this with my wife yet, but I plan on naming my firstborn Dipert in his honor. Eric Normand (@ericnormand) Eric runs the Clojure Gazette and LispCast. He's also a friendly guy who's passionate about teaching Clojure. Carin Meier (@gigasquid) Carin has written many excellent articles and given many excellent talks on Clojure, specifically on using it to dominate the world with robots. Bozhidar Batsov (@bbatsov) Bozhidar is my Emacs hero! He maintains CIDER and writes about Emacs. I don't understand how he manages to accomplish so much. My guess, though, is that he's written an AI clone of himself in elisp. Fogus (@fogus) Fogus goes by one name, which is completely badass. I really enjoy his tweets! David Nolen (@swannodette) I have no idea what a swannodette is, but good god David knows all the cool stuff. How does he do it? Alex Miller (@puredanger) Alex is the Clojure Community Czar. Follow him to hear about Clojure events and developments. Tweet him about all your stacktrace woes. Rich Hickey (@richhickey) Rich Hickey created a programming language called Clojure. It's pretty good! That's it for now! If there's anyone you'd like me to add, please let me know! I'm @nonrecursive.

New Clojure for the Brave and True Chapter: Interacting with Java

about 7 years ago | Daniel Higginbotham: Flying Machine Studios

After many months of editing existing chapters, I've written a brand-new chapter, "Interacting with Java". Here's a snippet: There comes a day in every Clojurist's life when she must venture forth from the sanctuary of pure functions and immutable data structures into the wild and barbaric Land of Java. This treacherous journey is necessary because Clojure is hosted on the Java Virtual Machine (JVM), granting it three fundamental characteristics. First, you run Clojure applications the same way you run Java applications. Second, you need to use Java objects for core functionality like reading files and working with dates. Third, Java has a vast and wondrous ecosystem of incredible libraries, and Clojure makes it painless for you to use them. In this way, Clojure is a bit like a utopian community plunked down in the middle of a non-utopian country. It's preferable to interact with other utopians, but every once in a while you need to talk to the locals in order to get things done. I appreciate all the tweets and emails I've received about the book; it's very gratifying and it helps keep me motivated. I'll be going to the Clojure Conj again this year. This will be my third Conj, and I'm so excited for it. I don't usually enjoy going to conferences, but I love the Conj because the talks are absolutely incredible. I hope I see you there!

New Clojure for the Brave and True Chapter: Concurrency, Parallelism, and State. And Zombies.

over 7 years ago | Daniel Higginbotham: Flying Machine Studios

There's a new chapter up! It's on "Concurrency, Parallelism, and State". And zombies. And Lady Gaga. And dwarven berserkers. And mind reading :D This has definitely been the most challenging chapter to write so far. I've tried to explain the concepts in a way that's fun and easily understandable for folks who haven't don't much, if any, parallel programming. If anything's confusingly written I'd love to hear about it :) By the way - if you're at ClojureWest, then lucky you! Sadly, I am not, but I'd like to recommend Bridget Hillyer and Jennifer Eliuk's talk on ClojureBridge. Bridget (and many others) have been doing a ton of work to put on the first free Clojure workshop for women. If you can't be there, send them some love on twitter, @bridgethillyer and @7maples! Another recommendation: Alan Dipert and Micha Niskin's talk on Hoplon. Alan is hilarious and brilliant, and Hoplon is really fun to use. It's a very different and very enjoyable way to do frontend programming. Up next: some chapter revisions! After that, I'll write about Java for Clojurists.

Clojure for the Brave and True to be Published, Remains Free Online

almost 8 years ago | Daniel Higginbotham: Flying Machine Studios

Good news everybody: I've signed a deal with No Starch Press to publish Clojure for the Brave and True! Not only that, the entire book will continue to be available for free online! Please sign up for notifications on new chapter releases: My hope is that this will provide as many people as possible with a thorough, entertaining introduction to Clojure. By being published through an established, respected company, it's possible for Clojure for the Brave and True to reach aspiring Clojurists through channels which I don't have the time or means to cultivate. By keeping the book online for free, I get to stay true to my hippie aspirations of free learning for everyone. What's more, the book will now be professionally edited — a big win for Clojure noobies and, I think, for the Clojure community. This is super cool! I can't think of any other publisher that allows an author to continue publishing a book for free online as he writes. I'm really excited by the book deal and am looking forward to collaborating with No Starch. They really "get" that writing the book is a creative, artistic endeavor for me. Also, writing Clojure is fun, I want my book to be fun, and No Starch really understands fun. Fun! Those who bought the book on Leanpub, have no fear: you will continue to receive updated PDF's and will receive the final, No Starch PDF at no additional cost. Thanks for buying the beta book! If you haven't already, please have a look at Clojure for the Brave and True. If you have already — thank you! I've really appreciated all the feedback and encouragement I've gotten so far, and I'm very happy to be able to contribute back to the community. By the way, there's a new chapter up: Writing Macros. Also, I'm going to be at the Conj in a couple weeks and would love to chat with folks about learning Clojure or anything else Clojure. Hope to see you there!

Stepping Off the Happiness Treadmill

about 8 years ago | Daniel Higginbotham: Flying Machine Studios

"Happiness is not something that you pursue; it is something you allow." — Chade-Meng Tan I did not expect to have a life-altering insight while tidying my kitchen some night during the winter of 2011. Yet there I was, sponge in hand and mind on idle, when out of nowhere: "No matter how much money you make, you will never feel like you have enough. Some part of you will always want more." Whatever I was doing at that moment - scrubbing a pot, wiping up crumbs, I don't remember exactly - I stopped. My stomach sank, and I felt a kind of melancholy blossom. What a sobering realization! Ultimately, I'm glad that some wayward neurons shoved those thoughts into my consciousness. The changes I've made since then have made me much more satisfied and without a doubt saved me from years of needless stress and frustration. At the time, I had been investing tremendous energy into building a web app in hopes that I'd start earning a little passive income. True, I also did it for the joy of conquering a technical challenge and to meet a need of my own. But I was largely motivated by anxiety, a ceaseless feeling that I had to find some way, any way, to make more money. You see, I grew up poor. Meal ticket poor, welfare housing poor, single Vietnamese refugee mother raising two children on a McDonald's wage poor. Wear the same outfit twice before washing it to save on soap and water poor. "Toys for tots" - I was one of those tots. In that kind of climate it's almost inevitable that you internalize a constant fear that you will not have enough money. Eventually my mother, through her own extraordinary effort and later with the help of my stepfather, was able to pull our family out of poverty and into a comfortable, lower-middle-class lifestyle. But the money worries never went away. My mother took a night job because it paid well, but even then she would often study when she wasn't working or sleeping so that she could advance herself further. The result was that she didn't get to spend much time with my brother and me, something she still regrets. One of my most heart-breaking experiences was hearing my mom reveal how lonely she was, that she never got to see me or my brother. She would come home at 3 in the morning and her only company was the hamster busily running nowhere on her hamster wheel. Thus it was that at 26 years old, firmly situated in a great job which more than met my financial needs, I too continued to busy myself with schemes to earn more money. And thus it was that some pre-conscious part of my brain, clearly wiser than the guy at the controls, shouted at me while I cleaned: "This is going to get you nowhere." After the initial shock, I understood how unwise my path was. That whatever part of me feels anxiety about money will never, never be quieted by more money. One idea that's stuck with me in my reading is that you get better at whatever you're doing. By allowing my actions to be driven by my compulsion for more money, I was not making things better. I was only reinforcing a broken behavior. I had to find another way. If I couldn't learn to be satisfied with what I had - which was enough to meet my basic needs and much more - then I was screwed. That night I resolved to put my effort into only those things I truly want to get better at: being a good friend, being a good brother, being a good partner, and being more creative. But I knew that the desire for more money would always be there, trying to siphon my time away and pour it into pointless schemes. And then I read about the difference that gratitude makes. First, in "The Pursuit of Perfect" by Tal Ben-Shahar. Then, in "The Happiness Advantage" by Shawn Achor. And yet again in "The Tools" by Phil Stutz and Barry Michels. I began my own daily gratitude practice, beginning each day by thinking of three things that I'm grateful for. I even created a little online community for sharing gratitude. This practice has been instrumental in helping me overcome my compulsion for more money. The craving has diminished as, daily, I take stock of my life and recognize the good things in it. I let myself fully experience the truth that my life is full of reasons for wonder and joy. In a way, I think of my craving as my mom internalized. Always looking out for me, always vigilant about my well being. Day by day I tell her, "It's OK. You don't have to work so hard. Things are OK. You can relax." And day by day, she does. In the mean time, I have flourished. I've done the creative work and learning that I find meaningful, taking up the violin, throwing myself into improv comedy, and learning more about programming. I've moved much closer to my family and have gotten to see them more in the past eight months than in the past eight years. And in a few weeks I will be marrying the love of my life. Life is good, and I am grateful for it.

Building a Forum with Clojure, Datomic, Angular, and Ansible

about 8 years ago | Daniel Higginbotham: Flying Machine Studios

After many long months I've finished re-writing Grateful Place. The site now uses Clojure as an API server, with Datomic for the database, Angular for the front end, and Vagrant and Ansible for provisioning and deployment. This setup is awesome, the best of every world, and I love it. Below we'll dive into the code base, covering the most important parts of each component and how everything works together. We'll cover: Clojure API Server Liberator for Easy API'ing Going on a Spirit Journey with Friend Testing a Clojure Web App is More Fun than Testing Rails Serving files generated by Grunt/Angular The Uberjar Datomic Why Oh Why Did I Do This The Poopy Code I Wrote to Make Basic Things "Easier" The Good Code I Ripped Off to do Migrations Mapification with Cartographer, My Very Own Clojure Library!!! Angular Peeking and Secondary Controllers Directives to the Rescue Infrastructure Creating a Local Sandbox with Vagrant Provisioning with Ansible Building and Deploying with a Janky Bash Script and Ansible Development Workflow Emacs Bookmarks and Keybindings tmuxinator Config Actually doing development All source is available on github. This article isn't meant to be a tutorial. However, if you have any questions about how things work or about how to work on the codebase, please leave a comment and I'll do my best to clarify. About the Site Grateful Place is my attempt at creating the kind of online community that I'd like to belong to. It's still in its infancy, but I'd like for it to become a site where people consciously help lift each other up. One way to do this is by expressing gratitude on a daily basis, which science says increases happiness. Some of the features include watching forum threads, liking posts, and creating a basic profile. I have a lot more planned, like tags and "summary" views, and I think the combination of Clojure and Angular will make it fun and easy for me to continue development :) If you want to have a look without diving head-first into hippified waters (what, do you have something against happiness?), you can use the demo site with username/password test101/test101. Be warned, though, that that server might have some bugs. Now, on to the code! Clojure API Server I'm very happy with using Clojure as an API server. The libraries involved are lightweight and transparent, with no magic, and that makes it so easy to fully understand what's going on. That was my experience with Liberator: Liberator for Easy API'ing Liberator provided me with a much better abstraction for handling logic which I was repeating in all of my controllers. For example, my create functions all basically looked like this before I moved to Liberator: (defn create! [params auth] (protect (:id auth) (if-valid params (:create validations/post) errors (let [post-map (create-post params)] {:body post-map}) (invalid errors)))) The above code implements a decision tree: First, the protect macro is used to ensure you're authorized to do whatever you're trying to do. The first argument is a boolean expression, in this case (:id auth), which just checks whether you're logged in. If the boolean expression is true, run everything that follows. Otherwise return an error status and error messages (see implementation). Check whether params is valid using the specified validation, in this case (:create validations/post). If it's valid, run the let statement, otherwise make the validation errors available in errors and run (invalid errors). There are a couple things that I didn't like about this approach. First, there was too much distance between the logical branches. For example, protect is basically an if statement, but the else is hidden. Also, the actual code I wrote in if-valid is a bit long, which makes it difficult to visually understand how (invalid errors) relates. Second, this approach required me to introduce more nesting in order to add more steps or checks in the workflow. This would make it even harder to understand as I'd mentally have to descend and ascend a few conditionals in order to understand what's going on. I'd end up with something like: - Decision one - Decision one first branch: Decision two - Decision two first branch: Decision three - Decision three first branch ... Lots of code here physically creating distance between branches - Decision three second branch ... More code causing more distance - Decision two second branch - Decision one second branch... what was the decision even? I can't remember and now it's hard for me to visually associate this branch with its parent decision So essentially, I'd have to keep an ever-growing decision tree in my head. The physical representation of the tree, the code, would help to obscure the logic flow as I added more code. Here's how the same function looks when rewritten using Liberator: (defresource create! [params auth] :allowed-methods [:post] :available-media-types ["application/json"] :authorized? (logged-in? auth) :malformed? (validator params (:create validations/post)) :handle-malformed errors-in-ctx :post! (create-content ts/create-post params auth record) :handle-created record-in-ctx) Holy shnikes! That's so much clearer! Liberator improved my code by providing a pre-defined, HTTP-compliant decision tree, providing sensible default logic for nodes, and by allowing me to easily associate my own logic with the nodes. This allows me to concentrate on one node at a time, instead of having to keep an increasingly complicated tree structure in my head. For example, I can physically place the logic for malformed? next to the code that I want to run if the request is malformed, specified by handle-malformed. Liberator has excellent documentation and using it is a big win. It lets me just plug my own bits of logic into a carefully-coded, useful HTTP framework. I definitely recommend it. Going on a Spirit Journey with Friend Friend still kinda makes my head hurt. It's a useful library that gets the job done, but I feel like using it requires poring over the source code until you attain that brief flash of enlightenment that allows you to pound out the code you need for as long as some dark, atavistic, pre-conscious part of your brain can hold everything together. After that you pray that you won't need to change anything because, Jesus, that trip can take a lot out of you. I don't know, maybe that's why peyote was invented. Anyway, that's a testament to my own need to learn (and perhaps a need for slightly clearer documentation) and not to the quality or value of the library itself. Everything hangs together, working with Ring in a stateless way, which I really appreciate. OK enough of my blathering. We want code! The actual tricky bits were: Getting Friend to return errors instead of redirecting Creating an authenticated session as soon as a user registers instead of requiring login It turned out that the first wasn't all that difficult. I think. Here's the code on github. It's also listed below, in the next code block. The key bit is :login-failure-handler, which simple returns a map for Ring. I also have :redirect-on-auth? listed twice. I'm not sure if this is necessary but every once in awhile I like to do some shaman programming, throwing chicken bones and listening to the wind in hopes that everything turns out OK. Things are working and I'm not going to mess with them. Creating the authenticated session is a different story. There are a lot of things going on. In order, they are: User submits registration form Request goes through a bunch of Ring middlewares that wrap the request, adding keyword params and handling json and whatnot Request hits the middleware created by Friend The request "hits" the users/attempt-registration Friend workflow If the registration is valid, return a friend authentication map. Friend "knows" that this is not meant to be a response sent to the browser, so the authentication map gets added to the Ring request map and the resulting map gets sent to the next Ring middleware The next ring middleware is routes The users/registration-success-response route matches users/registration-success-response returns a Ring map, providing a body. The response is a map like {:id 1234 :username "flyingmachine"}. This then gets used by Angular. Here's all the relevant code. Steps are indicated in brackets, like [1] or [2] or [3]. Step 1 is omitted as that's not code, you silly goose. ;; The ring app, https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/server.clj#L29 (defn wrap [to-wrap] (-> to-wrap (wrap-session {:cookie-name "gratefulplace-session" :store (db-session-store {})}) (wrap-restful-format :formats [:json-kw]) wrap-exception wrap-keyword-params wrap-nested-params wrap-params)) ; The ring app (def app (-> routes ;; [6] after a successful registration the routes ;; middleware is called auth ;; [3] after request is wrapped, send it to friend wrap ;; [2] )) ;; Friend middlware (defn auth [ring-app] (friend/authenticate ring-app {:credential-fn (partial creds/bcrypt-credential-fn credential-fn) :workflows [(workflows/interactive-form :redirect-on-auth? false :login-failure-handler (fn [req] {:body {:errors {:username ["invalid username or password"]}} :status 401})) users/attempt-registration ;; [4] session-store-authorize] :redirect-on-auth? false :login-uri "/login" :unauthorized-redirect-uri "/login"})) ;; [4] Friend runs this workflow function. If the workflow function ;; returns falsey, then friend tries the next workflow function. In ;; this case, when a user submits a registration form then the `when` ;; boolean expression is true and the function will not return falsey. ;; If the registration is successful it will return an authentication ;; map and continue to step 5. If the registration is unsuccessful it ;; will return a Ring response map, which is basically a map that has ;; the keys :body or :status. ;; https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/controllers/users.clj#L20 (defn attempt-registration [req] (let [{:keys [uri request-method params session]} req] (when (and (= uri "/users") (= request-method :post)) (if-valid params (:create validations/user) errors ;; [5] Here's where we return the authentication map, which ;; Friend appends to the request map, sending the result to the ;; next middleware (cemerick.friend.workflows/make-auth (mapify-tx-result (ts/create-user params) record) {:cemerick.friend/redirect-on-auth? false}) (invalid errors))))) ;; [7] The compojure route, https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/middleware/routes.clj#L67 (authroute POST "/users" users/registration-success-response) ;; [8] the final step in our journey (defn registration-success-response [params auth] "If the request gets this far, it means that user registration was successful." (if auth {:body auth})) I'm both proud and appalled that I wrote all that code. Testing a Clojure Web App is More Fun than Testing Rails For testing I decided to try out Midje. Midje is easy to get used to, and @marick has articulated a clear and compelling philosophy for it. But before we get into some code let me explain the heading, "testing a Clojure web app is more fun than testing Rails." This has to do with Clojure itself and not with any testing library. There's no real magic in any of the code I wrote. Everything is just a function. You give it an input and it returns an output. You give your application a Ring request and it goes through all the layers and returns a Ring response. You don't have to do any crazy setup hijinks or create special environments like you do in Rails - especially like you have to do when testing controllers. This makes testing so much easier and more fun. So, that said, I feel like there's not much remarkable with my tests. There's a lot of room for improvement. I ended up creating a lot of helper functions to DRY up my controller tests, and those might prove helpful to someone else. I also ended up writing a crazy-ass macro for creating functions with default positional arguments: (defmacro defnpd ;; defn with default positional arguments [name args & body] (let [unpack-defaults (fn [args] (let [[undefaulted defaulted] (split-with (comp not vector?) args) argcount (count args)] (loop [defaulted defaulted argset {:argnames (into [] undefaulted) :application (into [] (concat undefaulted (map second defaulted)))} unpacked-args [argset] position (count undefaulted)] (if (empty? defaulted) unpacked-args (let [argname (ffirst defaulted) new-argset {:argnames (conj (:argnames argset) argname) :application (assoc (:application argset) position argname)}] (recur (rest defaulted) new-argset (conj unpacked-args new-argset) (inc position))))))) unpacked-args (unpack-defaults args)] `(defn ~name (~(:argnames (last unpacked-args)) ~@body) ~@(map #(list (:argnames %) `(~name ~@(:application %))) (drop-last unpacked-args))))) ;; Examples (defnpd response-data [method path [params nil] [auth nil]] (data (res method path params auth))) (defnpd res [method path [params nil] [auth nil]] (let [params (json/write-str params)] (server/app (req method path params auth)))) The next big step for me with testing is to get off my butt and figure out how to run some kind of autotest process with Midje. If you're new to Clojure and are wondering what testing library, I think clojure.test works just fine. It's easier to understand than Midje, but Midje seems more powerful. Serving files generated by Grunt/Angular While developing, the frontend files are located completely outside of the Clojure application. The directory structure looks like: /server /src /gratefulplace - server.clj /resources ... /html-app /app - index.html /.tmp /scripts - app.js /controllers - topics.js ... So I needed some way to get the Clojure app to actually serve up these files. I also needed to be able to serve the files when they're packaged as resources in the final uberjar. This turned out to be really easy: ;; https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/config.clj ;; Example config (def conf (merge-with merge {:html-paths ["html-app" "../html-app/app" "../html-app/.tmp"]})) (defn config [& keys] (get-in conf keys)) ;; https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/middleware/routes.clj#L33 ;; Serve up angular app (apply compojure.core/routes (map #(compojure.core/routes (compojure.route/files "/" {:root %}) (compojure.route/resources "/" {:root %})) (reverse (config :html-paths)))) ;; Route "/" to "/index.html" (apply compojure.core/routes (map (fn [response-fn] (GET "/" [] (response-fn "index.html" {:root "html-app"}))) [resp/file-response resp/resource-response])) We're just iterating over each possible path for the front end files and creating both a file route and a resource route for them. This is a lazy way to do things, resulting in a few unnecessary routes. In the future, it would be nice to make the app "know" whether to use the single resource route, html-app, or whether it needs to use the file routes, ../html-app/app and ../html-app/.tmp. The Uberjar As I started to deploy the forum I found that I needed and easy way to run database-related tasks. Here's what I came up with: (ns gratefulplace.app (:gen-class) (:require [gratefulplace.server :as server] [gratefulplace.db.manage :as db])) (defn -main [cmd] (cond (= cmd "server") (server/-main) ;; I know there's repetition here please don't hate me :'( (= cmd "db/reload") (do (println (db/reload)) (System/exit 0)) (= cmd "db/migrate") (do (println (db/migrate)) (System/exit 0)))) So you can run java -jar gp2.jar server and get a server running, or reload the database or run migrations. I could also have used lein on the server, and I'll probably do that eventually. For now I'm just creating uberjars and copying them over. Holy cow, the Clojure section is over! Let's talk about Datomic now! Datomic Why Oh Why Did I Do This When I set about re-writing the site it felt risky to use Datomic because a) I didn't know how to use it and b) it didn't seem like it would add much value over postgres or mysql for my tiny side project. But those were also compelling reasons to go with it: a) it's exciting to learn a completely new way of working with databases, designed by some really freaking smart people who know which side of the bread is buttered and b) it's just a tiny side project and I can do whatever I want. Ultimately I'm happy with the decision. I've learned a lot by researching Datomic (see "Datomic for Five-Year-Olds") and using it has afforded the same simple, lightweight experience as using Clojure. You won't find any mind-blowing code here – I'm still trying to learn how to use Datomic well – but hopefully you'll find it useful or interesting. The Poopy Code I Wrote to Make Things "Easier" I wrote a number of wrapper functions in the misleadingly-name gratefulplace.db.query namespace: (ns gratefulplace.db.query (:require [datomic.api :as d]) (:use gratefulplace.config)) ;; This is dynamic so I can re-bind it for tests (def ^:dynamic *db-uri* (config :datomic :db-uri)) (defn conn [] (d/connect *db-uri*)) (defn db [] (d/db (conn))) ;; Don't make me pass in the value of the database that gets boring (def q #(d/q % (db))) ;; I'll give you an id, you give me a datomic entity or nil (defn ent [id] (if-let [exists (ffirst (d/q '[:find ?eid :in $ ?eid :where [?eid]] (db) id))] (d/entity (db) exists) nil)) ;; Is this an entity?! Tell me! (defmulti ent? class) (defmethod ent? datomic.query.EntityMap [x] x) (defmethod ent? :default [x] false) ;; I'll give you some conditions, you'll give me an entity id (defn eid [& conditions] (let [conditions (map #(concat ['?c] %) conditions)] (-> {:find ['?c] :where conditions} q ffirst))) ;; I want one thing please (defn one [& conditions] (if-let [id (apply eid conditions)] (ent id))) ;; I want all the things please (defn all [common-attribute & conditions] (let [conditions (concat [['?c common-attribute]] (map #(concat ['?c] %) conditions))] (map #(ent (first %)) (q {:find ['?c] :where conditions})))) ;; Passing the connection all the time is boring (def t #(d/transact (conn) %)) (defn resolve-tempid [tempids tempid] (d/resolve-tempid (db) tempids tempid)) ;; I make a lot of mistakes so please make it easy for me to retract them (defn retract-entity [eid] (t [[:db.fn/retractEntity eid]])) Some of these functions simply reduce the code I write by a tiny bit, for example by allowing me to not pass a connection or database value into every single database-related function, which would make no sense for me as I only have one database. Others, like one and all provide me with an "easier" way of performing common queries but at the expense of sometimes writing queries in roundabout ways or taking away some of my flexibility. For example, in the all function I'm limited to only one data source. The result is that I sometimes have to use the datomic.api functions in places where I'd prefer not to, and the codebase doesn't quite feel cohesive. One example of this is the query function in the watches controller: (defresource query [params auth] :available-media-types ["application/json"] :handle-ok (fn [ctx] (map (comp record first) (d/q '[:find ?watch :in $ ?userid :where [?watch :watch/user ?userid] [?watch :watch/topic ?topic] [?topic :content/deleted false]] (db/db) (:id auth))))) I have to call datomic.api/q directly because I want to pass in ?userid. I'm not sure whether I should drop these functions entirely and just use the datomic api or whether I should continue tweaking them to meet my needs. The Good Code I Ripped Off to do Migrations The gratefulplace.db.manage namespace has some code I stole and modified from Day of Datomic. It's a really cool, simple way of ensuring that migrations get run. The basic idea is that you keep track of schema names which have been installed, then install any schemas that haven't been installed. It's a simple, logical approach and the code that implements it is pretty neat, as you would expect from Stu Halloway. Mapification with Cartographer, My Very Own Clojure Library!!! Cartographer is the result of my attempt to easily do some processing and pull in relationships when converting a Datomic entity to a map. I think the README explains it all so you can learn more about it there. Here are some of the maprules used in GP2: ;; https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/db/maprules.clj (defmaprules ent->topic (attr :id :db/id) (attr :title :topic/title) (attr :post-count (ref-count :post/topic)) (attr :author-id (comp :db/id :content/author)) (attr :last-posted-to-at (comp format-date :topic/last-posted-to-at)) (has-one :first-post :rules gratefulplace.db.maprules/ent->post :retriever :topic/first-post) (has-one :author :rules gratefulplace.db.maprules/ent->user :retriever :content/author) (has-many :posts :rules gratefulplace.db.maprules/ent->post :retriever #(sort-by :post/created-at (:post/_topic %))) (has-many :watches :rules gratefulplace.db.maprules/ent->watch :retriever #(:watch/_topic %))) (defmaprules ent->post (attr :id :db/id) (attr :content (mask-deleted :post/content)) (attr :formatted-content (mask-deleted #(md-content (:post/content %)))) (attr :deleted :content/deleted) (attr :created-at (comp format-date :post/created-at)) (attr :topic-id (comp :db/id :post/topic)) (attr :author-id (comp :db/id :content/author)) (attr :likers #(map (comp :db/id :like/user) (:like/_post %))) (has-one :author :rules gratefulplace.db.maprules/ent->user :retriever :content/author) (has-one :topic :rules gratefulplace.db.maprules/ent->topic :retriever :post/topic)) There are definitely some edge cases where this approach gets strained but overall it's served me well. I ended up creating a macro which allows you to easily create a function that, when applied to a datomic entity, returns a map using maprules created with Cartographer: ;; https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/server/src/gratefulplace/db/mapification.clj (defmacro defmapifier [fn-name rules & mapify-opts] (let [fn-name fn-name] `(defn- ~fn-name ([id#] (~fn-name id# {})) ([id# addtl-mapify-args#] (if-let [ent# (or (db/ent? id#) (db/ent id#))] (let [mapify-opts# (merge-with (fn [_# x#] x#) ~@mapify-opts addtl-mapify-args#)] (fyingmachine.cartographer/mapify ent# ~rules mapify-opts#)) nil))))) Angular I've been learning Angular since last November and I love it. Using it, I feel like I finally have the right tools for creating web apps. Peeking and Secondary Controllers I wanted to implement the idea of "peeking" at things on the forum. For example, if you click on a user link you'll just view a summary of his info in the right column instead of completely leaving the page you're on. The idea is that, while reading a thread, you might find a response interesting. You want to know a little more about the author but don't want to lose your place. So you "peek" at him, which shows you some info and preserves your place in the thread. It was just something fun I wanted to try. However, as far as I know Angular doesn't make this very easy for you. The approach I took was to have a Foundation controller which places the Support service on the scope. Since all other controllers are nested under Foundation, they'll have access to $scope.support. The purpose of Support is define a way to show views in the right column and make data accessible to the view. For example, the author directive has the following: https://github.com/flyingmachine/gratefulplace2/blob/v1.0.0/html-app/app/scripts/directives/author.coffee#L8 $scope.peekAtAuthor = (author)-> User.get id: author.id, (data)-> _(data.posts).reverse() data.posts = _.take(data.posts, 3) Support.peek.show("user", data) The base view has the following: <div id="more"> <nav class="secondary"> <ng-include src="support.secondaryNav.include()"></ng-include> </nav> <ng-include src="support.peek.include()"></ng-include> </div> And the user peek looks like this: <div class="peek"> <div class="user"> <h3 class="username">{{support.peek.data.username}}</h3> <div class="about" ng-bind-html-unsafe="support.peek.data['formatted-about']"></div> </div> <div class="recent-posts"> <h4>Recent Posts</h4> <div class="post" ng-repeat="post in support.peek.data.posts"> <date data="post['created-at']"></date> <div> <a href="#/topics/{{post.topic.id}}">{{post.topic.title || 'view topic'}}</a> </div> <div class="content" ng-bind-html-unsafe="post.content"> </div> </div> </div> </div> So, ultimately, what's happenins is that when you call Support.peek.show("user", data), it sets some variables so that the view associated with the "user" peek is shown. That view then accesses the data you passed to Support.peek.show with, e.g., support.peek.data.username. I know this isn't a super-detailed explanation of what's going on, but I hope some investigation of the code will answer any questions you might have. Directives to the Rescue Angular directives are as powerful as everyone says they are, and I think I'm finally utilizing them well. You can see all my directives on github. This article is already 500 times to long so I won't go into any details, but if you're looking to understand Angular better, read this excellent SO response to How do I “think in AngularJS/EmberJS(or other client MVC framework)” if I have a jQuery background?. Infrastructure Because GP2 uses Datomic Free, I couldn't deploy to Heroku. This meant having to actually handle provisioning a server myself and deploying without following a tutorial. In the end things are working well. The site's residing on a [Digital Ocean][https://www.digitalocean.com/] server, which has been very easy to work with. Creating a Local Sandbox with Vagrant Creating a local sandbox lets you make all your provisioning mistakes more quickly. If you're creating a new provisioning script of tweaking your existing one, you should do it in a virtual machine. Vagrant makes this process as easy as an old shoe. Once you've installed virtualbox and vagrant all you have to do is run vagrant up from the infrastructure directory and you'll have a virtual machine ready to go. The Vagrant web site has excellent tutorials so check it out if you want to learn more. Provisioning with Ansible Ansible's supposed to be super simple compared to Puppet and Chef. I found it easy to learn. It's also simple enough to easily modify scripts and powerful enough to do exactly what I want it to, which is provision a server with Java and Datomic and deploy my app to it. You can check out my setup in infrastructure/ansible. If you're using Datomic free please do use it as a starting point. provision.yml has just about everything you need to get a server up and running, with the exception of uploading SSH keys. deploy.yml is used by the janky bash script below to upload an uberjar, run migrations, and restart the server. Building and Deploying with a Janky Bash Script and Ansible Here's my janky Bash scripts which first build the app and then deploys it with Ansible: # build.sh #!/bin/bash cd html-app grunt build rm -Rf ../server/resources/html-app cp -R targets/public ../server/resources/html-app cd ../server lein uberjar cd .. cp server/target/gratefulplace-0.1.0-SNAPSHOT-standalone.jar infrastructure/ansible/files/gp2.jar # deploy.sh #!/bin/bash die () { echo >&2 "$@" exit 1 } if [ "$#" -eq 0 ] then INVENTORY="dev" else INVENTORY=$1 fi [ -e infrastructure/ansible/$INVENTORY ] || die "Inventory file $INVENTORY not found" ./build.sh cd infrastructure/ansible/ ansible-playbook -i $INVENTORY deploy.yml Workflow OMG this article is almost over! Listen, I know you don't need to know this and it makes no difference to you but I am out here in the North Carolina heat sweating my ass off trying to finish this article so I can get on with my day. So it's pretty exciting that we're almost done. Anyway - here are workflow improvements I developed over the course of this project. You might also want to check out this My Clojure Workflow, Reloaded. Emacs Bookmarks, Snippets, and Keybindings I created a bookmark to open my server/src/gratefulplace/server.clj file with just a few keystrokes instead of having to navigate to it. I recommend doing this for any project which you'll be toiling over for months on end! Keybindings Behold, my very first keybinding! This starts the Jetty server: (defun nrepl-start-http-server () (interactive) (nrepl-load-current-buffer) (nrepl-set-ns (nrepl-current-ns)) ;; (with-current-buffer (nrepl-current-repl-buffer) ;; (nrepl-send-string "(def server (-main)) (println server)")) (nrepl-interactive-eval (format "(println '(def server (%s/-main))) (println 'server)" (nrepl-current-ns))) (nrepl-interactive-eval (format "(def server (%s/-main)) (println server)" (nrepl-current-ns)))) (eval-after-load 'nrepl '(define-key clojure-mode-map (kbd "C-c C-v") 'nrepl-start-http-server)) So, once you have server.clj open and you've run nrepl-jack-in you can hit C-c C-v to start the server. Also check out the nrepl keybindings for some great workflow helpers. tmuxinator config In order to do development you need to have Datomic and Grunt running. Instead of having to open up a bunch of terminal tabs and handle all that manually every time I want to start working, I use tmuxinator so that I can get my environment set up in one comand. Here's my config: # ~/.tmuxinator/nicu.yml # you can make as many tabs as you wish... project_name: gp2 project_root: ~/projects/web_sites/gp2 rvm: 1.9.3 tabs: - angular_server: git pull && cd html-app && grunt server - datomic: datomic - shell: I also have these nice little bash aliases: alias "tmk"="tmux kill-session -t" alias "datomic"="~/src/datomic/bin/transactor ~/src/datomic/config/samples/free-transactor-template.properties" Actually Doing Development So, in order to get to the point where you can actually start writing code and seeing the results, do the following: Install datomic and set up your own datomic alias Run mux gp2 to start tmux with your tmuxinator conf Open emacs Hit C-x r l to open your list of bookmarks and choose the bookmark for server.clj Run M-x nrepl-jack-in in emacs Hit C-c C-v to start the jetty server The End That's it! I hope you've found this article useful. I'm going to go have a life for a little while now. Haha, just kidding! I'm going to spend the next two hours hitting refresh on my reddit submission!

A Taste of the λ Calculus

about 8 years ago | Daniel Higginbotham: Flying Machine Studios

I've been having a brain-bending good time reading An Introduction to Functional Programming Through Lambda Calculus. Using examples from that book, this article will walk you through the basics of λ calculus. We'll then look at the surprising, counterintuitive way that the λ calculus lets us represent conditional expressions and boolean operations — all with functions as the only values. It was a very different way of thinking for me, and exciting to learn. I hope it's exciting for you, too! A Bit of History As every aspiring greybeard knows, the λ calculus was invented by Alonzo Church in response to David Hilbert's 1928 Entscheidungsproblem. The Entscheidungsproblem inspired another computational model which you may have heard of, the Turing Machine. The λ calculus is one of the foundations of computer science. It's perhaps most famous for serving as the basis of Lisp, invented (or discovered, if you prefer to think of Lisp as being on par with the theory of gravity or the theory of evolution) by [John McCarthy](http://en.wikipedia.org/wiki/JohnMcCarthy(computer_scientist)) in 1958. Indeed, by examining the λ calculus, you can see where Lisp derives its beauty. The λ calculus had a lean syntax and dead-simple semantics, the very definition of mathematical elegance, yet it's capable of representing all computable functions. Enough history! Tell Me About λ Expressions! The λ calculus is all about manipulating λ expressions. Below is its specification. If you don't know what something means, don't worry about it at this point - this is just an overview and we'll dig into it more. <expression> ::= <name> | <function> | <application> <name> ::= any sequence of non-blank characters <function> ::= λ<name>.<body> <body> ::= <expression> <application> ::= (<function expression> <argument expression>) <function expression> ::= <expression> <argument expression> ::= <expression> ;; Examples ;; Names x joey queen-amidala ;; Functions ;; Note that functions always have one and only one parameter λx.x λy.y ;; equivalent to above; we'll get into that more λfirst.λsecond.first ;; the body of a function can itself be a function λfn.λarg.(fn arg) ;; Application (λx.x λx.x) ((λfirst.λsecond.first x) y) There are two super-cool things about this specification. First, it really boils down to four elements: names, functions, application, and "expressions" which can be any of the above. That's awesome! Second, function bodies and function application arguments can be any expression at all, meaning that a) functions can take functions as arguments and b) functions can return functions. You can see how this is directly related to functional programming, where you have first class functions and higher order functions. This is interesting in itself as it gives you a glimpse of the theoretical underpinnings of functional programming. But it gets way, way cooler. By the end of this article you'll see how conditions and boolean operations can be represented in terms of functions and functions that operate on functions. In order to get there, let's first look at how function application works. Then we'll go over some basic but crucial functions. Function Application When you apply a function to an argument expression, you replace all instances of name within the function's body with the argument expression. Keep in mind that we're talking about a mathematical system here, not a programming language. This is pure symbol manipulation, without any regard for how actual hardware will carry out the replace operation mentioned above. Let's start to flesh out this purely abstract notion of function application with some examples, starting with the identity function: ;; Identity function λx.x As you would expect, applying this function to an argument expression returns the argument expression. In the example below, don't worry about where "foo" comes from: ;; Apply the identity function to foo (λx.x foo) ;; After replacing all instances of x within the body, you get: foo Makes sense, right? I'm sure that you can intuitively understand what's going on in function application. Nevertheless, I think we can make it clearer by looking at a few examples: (λs.(s s) foo) => (foo foo) (λx.λy.x foo) λy.foo (λa.λb.λc.((a b) c) foo) λb.λc.((foo b) c) For a more thorough explanation of what's going on here, please see Jon Sterling's comment below! Now that we understand how to apply functions, let's explore a few more basic functions. The Self-Application Function The self-application function evaluates to the application of its argument to itself: λs.(s s) Let's see an example: ;; Apply the self-application function to the identity function (λs.(s s) λx.x) ;; Perform replacement - results in an application (λx.x λx.x) ;; Perform another replacement λx.x Now let's make things interesting: ;; Apply the self-application function to itself (λs.(s s) λs.(s s)) ;; Perform replacement (λs.(s s) λs.(s s)) ;; Hmmm this is exactly like the first expression. Let's perform ;; replacement again just for kicks (λs.(s s) λs.(s s)) How about that, it turns out that it's possible for evaluation to never terminate. Fun! The Function Application Function Check this out: λfunc.λarg.(func arg) This function takes a function as its argument, returning a function: (λfunc.λarg.(func arg) λx.x) => λarg.(λx.x arg) When you apply this resulting function to an argument, the end result is that the function you supplied as the first argument gets applied to the current argument: ;; Notice the identity function nestled next to the ;; second left parenthesis (λarg.(λx.x arg) λs.(s s)) Here's the whole application: ((λfunc.λarg.(func arg) λx.x) λs.(s s)) => (λarg.(λx.x arg) λs.(s s)) => (λx.x λs.(s s)) λs.(s s) Is your head hurting yet? I sure hope so! That's your brain's way of letting you know that it's learning! We're starting to get a hint of the cool things you can do with λ calculus. It only gets cooler from here! Interlude: Give the Functions Names, Already! Before this post gets overwhelmed with "λx.x" and "λs.(s s)" and such, let's introduce some syntax: ;; Name functions def <name> = <function> ;; Examples def identity = λx.x def self_apply = λs.(s s) def apply = λfunc.λarg.(func arg) Now wherever we see <name>, we can substitute <function>. Examples: (identity identity) => (λx.x identity) => identity (self_apply identity) => (λs.(s s) identity) => (identity identity) => identity ((apply idenity) self_apply) => ((λfunc.λarg.(func arg) identity) self_apply) => (λarg.(identity arg) self_apply) => (identity self_apply) => self_apply Make sense? Excellent! This will let us break your brain with greater efficiency. Now pay attention, because things are about to get super flippin' fantastic. Argument Selection and Argument Pairing Functions In the λ calculus, functions by definition have one and only one parameter, the name. This might seem limiting, but it turns out that you can build functions which allow you to work on multiple arguments. The following functions together allow you to select either the first or the second of two arguments. We'll look at them all together first and then dig in to see how they work together. def make_pair = λfirst.λsecond.λfunc.((func first) second) def select_first = λfirst.λsecond.first def select_second = λfirst.λsecond.second select_first and select_second do what their names suggest, selecting either the first or second of two arguments. They have the same underlying structure; they're both functions which take a first argument and evaluate to a function. This function is applied to a second argument. select_first returns first, and select_second returns second. Let's see how this works with select_first: ;; Start here ((select_first identity) apply) ;; Substitute the function itself for "select_first" ((λfirst.λsecond.first identity) apply) ;; Perform the first function application, replacing "first" with "identity". ;; This returns another function, which we'll apply to a second argument. ;; Notice that the body of the resulting function is "identity", and ;; the name "second" doesn't appear in the body at all (λsecond.identity apply) ;; Apply function. Since "second" doesn't appear in the function body, ;; it disappears into the ether. identity select_second uses the same principle: ((select_second identity) apply) ((λfirst.λsecond.second identity) apply) (λsecond.second apply) apply So, select_first and select_second are able to operate on a pair of arguments. But how do we create pairs for the to work on? make_pair creates a "pair" by returning a function which expects either select_first or select_second as its argument. This is awesome - we don't need any data structures to represent a pair, all we need are functions! Let's actually create a pair: ;; Start here ((make_pair identity) apply) ;; Substitute the actual function ((λfirst.λsecond.λfunc.((func first) second) identity) apply) ;; Perform first function application, replacing "first" with "identity" (λsecond.λfunc.((func identity) second) apply) ;; Perform remaining function application, replacing "second" with "apply" λfunc.((func identity) apply) This resulting function looks very familiar! Let's compare it with our select_first and select_second applications above: ;; Result of ((make_pair identity) apply) λfunc.((func identity) apply) ;; Application of select_first and select_second ((select_first identity) apply) ((select_second identity) apply) ;; Apply the result of make_pair to select_first (λfunc.((func identity) apply) select_first) => ((select_first identity) apply) So, to reiterate, make_pair works by taking a first argument. This returns a function with takes a second argument. The result is a function which you can apply to either select_first or select_second to get the argument you want. This is super freaking cool! A pair is a function which has "captured" two arguments and which you then apply to a selection function. Starting with just four basic constructs – names, functions, applications, expressions – and five simple rules for performing function application, we've been able to construct pairs of arguments and select between them. And things are about to get even more fun! We're now ready to see how we can create conditional expressions and boolean operations purely using λ expressions. Conditional Expressions and Boolean Operations The upcoming treatment of conditional expressions and boolean operations is going to look kinda weird at first. You'll want to keep in mind that in abstract math, elements don't have any inherent meaning but are defined by the way with they interact with each other — by their behavior. For our purposes, the behavior of a conditional expression is to select between one of two expressions, as shown by the following pseudocode: if true <expression> else <expression> end Hmm... selecting between two expressions... we just went over that! make_pair gave us a pair of expressions to choose between using either select_first or select_second. Because these functions result in the exact same behavior as if/else, let's gon ahead repurpose these: ;; This is identical to make_pair def cond = λe1.λe2.λc((c e1) e2) def true = select_first def false = select_second ;; Apply a conditional expression to true, aka select_first (((cond <e1>) <e2>) true) => <e1> ;; Apply a conditional expression to false aka select_second (((cond <e1>) <e2>) false) => <e2> You're probably not used to thinking of a conditional expression as a function which you apply to either true or false, but it works! NOT, AND, OR NOT can be seen as if x false else true end So, if x is true then false is selected, and if x is false then true is selected. Let's look at this using the cond expressions above: ;; In general (((cond <e1>) <e2>) true) => <e1> (((cond <e1>) <e2>) false) => <e2> ;; For NOT (((cond false) true) true) => false (((cond false) true) false) => true ;; So in general, we can say: def not = λbool.(((cond false) true) bool) ;; We can simplify this, though I'm lazy and won't show how: def not = λbool.((bool false) true) AND can be seen as if x y else false end In other words, if x is true then the value of the expression is the value of y, otherwise it's false. Here's how we can represent that: def and = λx.λy.((x y) false) Keep in mind that true is select_first and false is select_second: ;; select_first is true ;; when x is true, the value of the entire expression is the value of y (λx.λy.((x y) false) select_first) => λy.((select_first y) false) ;; select_second is false ;; when x is false, the second argument, false, is selected (λx.λy.((x y) false) select_second) => λy.((select_second y) false) We can treat OR similarly: if x true else y end We can capture this with def or = λx.λy.((x true) y) I won't work this one out - I'll leave it "as an exercise for the reader." :) The End I hope you've enjoyed this brief taste of the λ calculus! We've only scratched the surface of the kinds of neat things it's capable of. If you thought this article was fun, then I definitely recommend An Introduction to Functional Programming Through Lambda Calculus. This fun tome provided most or all of the examples I've used, though I've tried to present them in a way that's easier to understand. I also recommend The Art of Lisp & Writing, which conveys the beauty and joy of coding in Lisp.

How Clojure Babies are Made: What Leiningen Is

over 8 years ago | Daniel Higginbotham: Flying Machine Studios

"What the hell is Leiningen?" is a question you've probably overheard many times in your day-to-day life. You've probably even asked it yourself. Up until now we've described specific capabilities that Leiningen has and how those capabilities are implemented. It can build and run your app, it has a trampoline, and so forth. But it's time to take a step back and get a high-level understanding of what Leiningen is. It's time to stare deeply into Leiningen's eyes and say "I see you," like Jake Sully in that Avatar documentary. We'll do this by giving an overview of the non-coding related tasks you need to accomplish when building software. Next, we'll describe how Leiningen helps you accomplish these tasks and compare it to similar tools in Ruby. This post isn't as nitty-gritty as the previous posts in the Clojure Babies series, but it will help lay the groundwork for an upcoming post on packaging. Additionally, I hope it will clarify what a programming language artifact ecosystem is. This concept is often overlooked when teaching a programming language, and when it is covered it's not covered in a systematic way. Together, noble-chinned reader, we will remedy that situation. For our generation and all generations to come. Programming Language Artifact Ecosystems In order to become proficient at a language, you need to know much more than just its syntax and semantics. You need to familiarize yourself with the entire programming language ecosystem, which is comprised of everything you need in order to build working software in that language. It can be broken down into at least the following sub-ecosystems: The documentation ecosystem The developer community The development environment ecosystem (editor support) The artifact ecosystem Right now we only care about the artifact ecosystem. For our purposes, a programming artifact is a library or executable. Ruby gems, shell scripts, Java jars, shared libraries, and "HAL9000.exe" are all programming artifacts. An artifact ecosystem is the set of tools and services that allow you to do the following with regard to artifacts: Retrieve them from repositories Incorporate them in your own project, (possibly) resolving conflicts Build them Publish them to repositories Run them Tools are often layered on top of each other, one tool smoothing out the warts of the tools it wraps. For example, the following tools (and more) are part of the Ruby artifact ecosystem: Ruby Gems provides a package specification, the means to incorporate gems in your project, and the means to build and publish gems rubygems.org is a central repo for gems Bundler provides a layer on top of Ruby Gems, providing dependency resolution and gem retrieval Jeweler is one of many tools for easing the process of creating gemspecs and building gems. Other languages have their own tools. Java has Maven, PHP has Pear or whatever. Artifact management is a common need across languages. In previous Clojure Baby posts, we've seen that we can use Leiningen to build and run Clojure programs. It turns out that Leiningen also handles the remaining tasks - retrieving packages, incorporating them in your project, and publishing them. It's truly the Swiss Army Bazooka (I'm going to keep repeating that phrase until it catches on) of the Clojure artifact ecosystem. But why is it that in Ruby you need an entire constellation of tools, while in Clojure you only need one? Leiningen Is a Task Runner with Clojure Tasks Built In Leiningen is able to handle so many responsibilities because it is, at heart, a task runner. It just happens to come with an excellent set of built-in tasks for handling Clojure artifacts. (Incidentally, this is probably where Leiningen's name came from. "Leiningen Versus the Ants" is a short story where the protagonist fights ants. Ant is a Java build tool that evidently is unpleasant to use for Clojure builds.) By comparison, Ruby's Rake is also a task runner used by many of Ruby's artifact tools, but Rake provides no built-in tasks for working with Ruby artifacts. "Task runner" is a little bit ambiguous, so let's break it down. Ultimately, all Leiningen tasks are just Clojure functions. However, in previous posts we've seen how fun it is to try and run Clojure functions from the command line. In case you need a short refresher: it's not fun at all! Leiningen allows the Clojure party to remain fun by serving as an adapter between the CLI and Clojure. It takes care of the plumbing required for you to run a Clojure function. Whether the function is provided by your project, by Leiningen's built-in tasks, or by a Leiningen plugin, Leiningen does everything necessary to get the function to run. In a way, Leiningen's like an attentive butler who quietly and competently takes care of all your chores so that you can focus on your true passions, like knitting sweaters for gerbils or whatever. This manner of executing code was foreign to me when I first came to Clojure. At that time I had mostly coded in Ruby and JavaScript, and I had a decent amount of experience in Objective C. Those languages employ two different paradigms of code execution. Ruby and Javascript, being scripting languages, don't require compilation and execute statements as they're encountered. Objective C requires compilation and always starts by executing a main method. With Leiningen, Clojure has achieved an amalgamation of the two paradigms by allowing you to easily run arbitrary functions with a compiled language. The End Hopefully, this article has given you a firmer grasp of what Leiningen is. The idea that Leiningen is a task runner with a powerful set of built-in tasks designed to aid Clojure artifact management should help you organize your disparate chunks of Leiningen knowledge. In the next article, we'll add another chunk of Leiningen knowledge by examining the way Leiningen retrieves artifacts from repositories and incorporates them in your project. Goodbye for now! Shout Outs Thanks to Pat Shaughnessy and technomancy for reviewing this article. technomancy provided the line "Leiningen is an adapter between the CLI and Clojure", which really helped!

How Clojure Babies are Made: Leiningen's Trampoline

over 8 years ago | Daniel Higginbotham: Flying Machine Studios

In the last "How Clojure Babies are Made" article I hinted that Leiningen avoids waste by using a trampoline. In this installment of this awkwardly-named Clojure series, you'll learn what the trampoline does and how it does it. The Rationale Behind the Trampoline I know what you're thinking, wise-hearted reader. "Why would anyone need to provide a rationale for a trampoline? That's like trying to give a reason for rainbows, or a newborn's laughter, or Michael Jackson's Thriller." Allow me to explain myself. lein trampoline does indeed give you a feeling of weightless freedom, just not in the way that you're used to. See, whenever you use Leiningen to run code from your project, you end up with two Java processes running. Each process loads a separate instance of the JVM. We saw this in the previous article in the output of ps | grep lein. The first process is for Leiningen itself, and it's responsible for setting up everything necessary for your project code to run. The second process is where your code actually executes. If you were to run lein -h, you would only start one Java process, as none of your project code would need to be executed. Leiningen starts a separate process for your project in order to enforce isolation. This is because Leiningen is a true gentleman who does not allow his namespaces and dependencies (like, say, a completely different version of Clojure) to interfere with your meticulously hand-crafted, artisinal program. However, like a doting father, the Leiningen process continues to stay open for the entire duration of your program's execution. If you have a long-running process, like a web server for your Justin Bieber fan site, then Leiningen stays open the whole time consuming memory that could be put to use in compositing images of the Biebs with hearts all over them. This is where the trampoline comes into play. It allows the Leiningen process to exit completely before your project's process starts. Now, instead of two JVM's running, you only have one. I think the name "trampoline" was chosen to evoke an image of Leiningen providing a launching point for your program. However, I choose to envision the trampoline process as follows: Leiningen takes your program, which is embodied as one of those cymbal-clanging monkeys with unsettling eyes, and winds it up. Leiningen gingerly places the wind-up monkey which is your program on the floor of a cozy Hobbit hole. Leiningen steps outside and mounts a gigantic trampoline in Bilbo's front yard. Leiningen takes two warm-up bounces and then, with a mighty "Hyup!", rockets himself into the sky, his mustaches flapping in the wind. He grows smaller and smaller until, with a bright sparkle, he disappears entirely. Your cymbal-clanging web server starts up, allowing users to profess their undying love for teenage pop stars. How lein trampoline Works Though you don't really need to understand how lein trampoline works in order to use it, I think it's pretty cool. Below I walk you through it step by step, with relevant code. We'll be using the project under leiningen/lein-build of the make-a-clojure-baby github repo. Run lein trampoline run from the command line. If you're on a linux machine, this executes a bash script. This script is probably at ~/bin/lein on your system. The bash script sets the TRAMPOLINE_FILE environment variable to a path. Later in this process, Leiningen will write a command to this file. Here's the part of the script that sets the environment variable: if ([ "$LEIN_FAST_TRAMPOLINE" != "" ] || [ -r .lein-fast-trampoline ]) && [ -r project.clj ]; then INPUTS="$@ $(cat project.clj) $LEIN_VERSION $(cat "$LEIN_HOME/profiles.clj")" INPUT_CHECKSUM=$(echo $INPUTS | shasum - | cut -f 1 -d " ") # Just don't change :target-path in project.clj, mkay? TRAMPOLINE_FILE="target/trampolines/$INPUT_CHECKSUM" else TRAMPOLINE_FILE="/tmp/lein-trampoline-$$" trap "rm -f $TRAMPOLINE_FILE" EXIT fi The first Java process starts: /usr/bin/java \ -client -XX:+TieredCompilation \ -Xbootclasspath/a:/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \ -Dfile.encoding=UTF-8 \ -Dmaven.wagon.http.ssl.easy=false \ -Dleiningen.original.pwd=/Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build \ -Dleiningen.script=/Users/daniel/bin/lein \ -classpath :/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \ clojure.main \ `# clojure.main is the entry point` \ -m leiningen.core.main \ trampoline run 1 This causes Java to execute the -main method in clojure.main, which in turn loads leiningen.core.main and executes its -main function. leiningen.core.main/-main applies the trampoline task: (defn ^:higher-order trampoline "Run a task without nesting the project's JVM inside Leiningen's. Calculates the Clojure code to run in the project's process for the given task and allows Leiningen's own JVM process to exit before running it rather than launching a subprocess of Leiningen's JVM. Use this to save memory or to work around stdin issues." [project task-name & args] (when (= :leiningen (:eval-in project)) (main/info "Warning: trampoline has no effect with :eval-in-leiningen.")) (binding [*trampoline?* true] (main/apply-task (main/lookup-alias task-name project) (-> (assoc project :eval-in :trampoline) (vary-meta update-in [:without-profiles] assoc :eval-in :trampoline)) args)) (if (seq @eval/trampoline-forms) (write-trampoline project @eval/trampoline-forms @eval/trampoline-profiles) (main/abort task-name "did not run any project code for trampolining."))) trampoline calls leiningen.core.main/apply-task but with a twist: it passes that function an updated project configuration, setting :eval-in to :trampoline. You can see this is the snippet above. Eventually, leiningen.core.eval/eval-in-project gets applied. The cool thing about this function is that it then calls leiningen.core.eval/eval-in, which is a multi-method. It has different definitions for :subprocess, :trampoline, :nrepl, and a few more. This is one of the first times I've seen defmethod "in the wild" and it really tickled my pfeffernuesse. Definitely check it out on github. Since we updated our project configuration in the last step so that :eval-in is :trampoline, the :trampoline method gets matched: (defmethod eval-in :trampoline [project form] (swap! trampoline-forms conj form) (swap! trampoline-profiles conj (select-keys project [:dependencies :source-paths :resource-paths :test-paths]))) This updates the trampoline-forms and trampoline-profiles atoms within the leiningen.core.eval namespace. The trampoline function shown in step 5 above continues executing: (if (seq @eval/trampoline-forms) (write-trampoline project @eval/trampoline-forms @eval/trampoline-profiles) (main/abort task-name "did not run any project code for trampolining."))) write-trampoline writes out the entire Java command necessary to finally run our project's main function. It writes this command to the path in the TRAMPOLINE_FILE environment variable set by the bash script in step 2 above. The Leiningen process exits and the bash process from step 2 continues. It checks for the existence of TRAMPOLINE_FILE, and since it exists, it essentially evaluates the command in it, kicking off the Java process which will run your code: if [ -r "$TRAMPOLINE_FILE" ] && [ "$LEIN_TRAMPOLINE_WARMUP" = "" ]; then TRAMPOLINE="$(cat $TRAMPOLINE_FILE)" if [ "$INPUT_CHECKSUM" = "" ]; then rm $TRAMPOLINE_FILE fi exec sh -c "exec $TRAMPOLINE" else exit $EXIT_CODE fi It's a bit of circuitous route, but Leiningen is not one to shy away from hard work! The End I hope you found this article interesting! The following topics are next on my list of things to write about: How Leiningen manages dependencies How to distribute a full application How to distribute a library So long for now, and may the lambda be with you!

How Clojure Babies Are Made: Understanding lein run

over 8 years ago | Daniel Higginbotham: Flying Machine Studios

Leiningen reminds me a lot of Major Alex Louis Armstrong from Fullmetal Alchemist: What artistry! They are both manly They are both artistic Sometimes you don't know what the f* they're saying (but it's amazing anyway) Mustaches Sparkles Though the Strong Arm Alchemist will forever remain inscrutable, we do have some hope in understanding Leiningen better. In this post, we'll peek under Leiningen's lederhosen so that we can begin to learn precisely what it does and how to make effective use of it in developing, testing, running, and deploying Clojure applications. Here's what we'll learn: How to build and run a Clojure program without Leiningen What happens when you run lein run Basic Leiningen architecture What happens when you run a Leiningen task In the next post, we'll learn: How Leiningen avoids waste with a trampoline How Leiningen manages dependencies How to distribute a full application How to distribute a library Other stuff Leiningen does This post builds on How Clojure Babies Are Made: Compiling and Running a Java Program, so make sure you understand everything in that post first. It doesn't have any pictures of sparkly, sensitive, muscle-bound men though so if that's what you're hoping for you're out of luck. We're here to learn Clojure, dammit, not ogle cartoons! Brief Overview of Leiningen Leiningen is the Swiss Army Bazooka of Clojure development. It handles: Project compilation. Your Clojure has to get converted into Java bytecode somehow, and Leiningen automates the process. Dependency management. Similar to Ruby's bundler and gemfiles, Leiningen automates the process of resolving and downloading the Java jars your project depends on Running tasks. Superficially similar to Ruby's Rake, except that Rake is not very declarative. This could have its own blog post. Deployment. Helps with creating Java jars which can be executed and/or incorporated in other projects. Similar to Ruby gems. How to Build and Run a Clojure Program Without Leiningen By understanding how to build and run a Clojure program manually, you'll better understand what Leiningen does to automate the process. We'll be working with the files under leiningen/manual-build which you can get from the make a clojure baby repo. All path references will be relative to make-a-clojure-baby/leiningen/manual-build. The program we're compiling will make us fluent in the ways of German Love: (ns learn-a-language.important-phrases (:gen-class)) ;; It's time for some German love! (def german [["I love you." "Ich liebe dich."] ["You make me so happy!" "Du machst mich so glucklich!"] ["I miss you." "Ich vermisse dich./Du fehlst mir."] ["Pass me the mustard." "Gib mir den Senf."] ["Kiss me!" "Kuss mich!"]]) (defn -main [which] (let [phrases (get german (Integer. which))] (println "English: " (first phrases)) (println "German: " (second phrases)))) One important thing to note here is that we included the :gen-class directive in the ns declaration. This tells Clojure to generate a named Java class when it compiles the namespace. Remember that Java requires a public main method within a public class to serve as the entry point for the JVM. Using :gen-class allows us to use learn-a-language.important-phrases/-main as the entry point. Let's go ahead and compile. First, start up a Clojure repl (note that the git repo includes the 1.5.1 release of Clojure as clojure.jar): java -cp .:clojure.jar clojure.main Notice that we specifed the classpath with -cp .:clojure.jar. This does two things: It allows us to execute the main method in clojure.main similarly to what we saw at the end of the last post It adds the current directory to the classpath so that when you actually start loading Clojure files and compiling namespaces, the Clojure repl can find them. To see what I mean, try starting the repl with java -jar clojure.jar and then running the code below. You should now have a Clojure repl running in your terminal. Copy and paste the following lines into it: (load "learn_a_language/important_phrases") (compile 'learn-a-language.important-phrases) The first line reads the specified file. The second actually compiles the learn-a-language.important-phrases namespace, generating a boatload of Java class files in the classes/learn_a_language directory: $ ls classes/learn_a_language/ important_phrases$_main.class important_phrases$fn__19.class important_phrases$fn__48.class important_phrases$loading__4910__auto__.class important_phrases.class important_phrases__init.class (I won't go into into detail about the purposes of the various class files, but you can start to learn more about that in clojure's compilation documentation.) After going through the above steps, you might have a question on your mind grapes: "Where did the classes directory come from?" Oh, gentle-hearted reader. Your dedication to learning has touched my heart. I shall answer your query: Clojure places compiled files under *compile-path*, which is classes by default. Therefore, classes must exist and be accessible from the classpath. You'll notice that there's a classes directory in the git repo with a .gitkeep file in it. Never change, dear, inquisitive reader. Never! Now that we've compiled our Clojure program, let's get it running: # Notice that you have to provide a numerical argument $ java -cp classes:clojure.jar learn_a_language/important_phrases 0 English: I love you. German: Ich liebe dich. $ java -cp classes:clojure.jar learn_a_language/important_phrases 3 English: Pass me the mustard. German: Gib mir den Senf. Success! You are now ready to express your love for Leiningen in its native tongue. I highly recommend you use this program to come up with tweets to send to @technomancy to express your appreciation. But before you do that, notice the classpath. We need to include both the classes directory, because that's where the learn_a_language/important_phrases class files live, and clojure.jar, because the class files generated by compile need to be able to access Clojure class files. To sum up: # load clojure repl $ java -cp .:clojure.jar clojure.main # in Clojure repl user=> (load "learn_a_language/important_phrases") user=> (compile 'learn-a-language.important-phrases) # back to command line java -cp classes:clojure.jar learn_a_language/important_phrases 0 I hope this brief foray into the world of manually building and running a Clojure program has been educational. You can see how it would be burdensome to go through his process over and over again while developing a program. Let's finally bring in our pal Leiningen to automate this process. How Leiningen Compiles and Runs a Basic Clojure Program Let's build the "learn a language" program with Leiningen. We have a very basic project at make-a-clojure-baby/leiningen/lein-build. Under that the directory, the file src/learn_a_language/important_phrases.clj is the same as the one listed above. lein run Lein automates the build + run process with lein run. Go ahead and try that now: $ lein run 2 Compiling learn-a-language.important-phrases English: I miss you. German: Ich vermisse dich./Du fehlst mir. You can probably guess what's happening at this point, at least to a degree. Leiningen is compiling important_phrases.clj resulting in a number of Java .class files. We can, in fact, see these class files: $ ls target/classes/learn_a_language important_phrases$_main.class important_phrases$loading__4784__auto__.class important_phrases__init.class Leiningen then somehow constructs a classpath such that both Clojure and the "important phrases" classes are accessible by Java. Finally, the -main function is executed. I know what you're thinking at this point, noble reader. You're thinking that the "somehow" in "somehow constructs a classpath" is a particularly un-manly, un-artistic, un-mustached, un-sparkly word, unbefitting an article on Leiningen. And you are absolutely right. To avoid your wrath, let's dig into Leiningen's source code so that we can understand what's going on with complete clarity. Walking Through "lein run" To get an idea of where to start, let's run lein run 1 again and then run ps | grep lein. The output has been broken up to make more sense (apologies for the goofy highlighting): # you can actually copy and paste the part after the PID to try this # in your terminal 8420 /usr/bin/java \ -client -XX:+TieredCompilation \ -Xbootclasspath/a:/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \ -Dfile.encoding=UTF-8 \ -Dmaven.wagon.http.ssl.easy=false \ -Dleiningen.original.pwd=/Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build \ -Dleiningen.script=/Users/daniel/bin/lein \ -classpath :/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \ clojure.main \ `# clojure.main is the entry point` \ -m leiningen.core.main \ run 1 8432 /usr/bin/java \ `# your classpath will be different` \ -classpath \ /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/test:\ /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/src:\ /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/dev-resources:\ /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/resources:\ /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/target/classes:\ /Users/daniel/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar -XX:+TieredCompilation \ -Dclojure.compile.path=/Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/target/classes \ -Dlearn-a-language.version=0.1.0-SNAPSHOT \ -Dfile.encoding=UTF-8 \ -Dclojure.debug=false \ clojure.main \ `# clojure.main is the entry point` \ `# this next part specifies code to be evaluated by Clojure` \ -e (do \ (try \ (clojure.core/require 'learn-a-language.important-phrases) \ (catch java.io.FileNotFoundException ___6081__auto__)) \ (set! *warn-on-reflection* nil) \ (clojure.core/let \ [v__6079__auto__ \ (clojure.core/resolve 'learn-a-language.important-phrases/-main)] \ (if \ (clojure.core/ifn? v__6079__auto__) \ (v__6079__auto__ "1") \ (clojure.lang.Reflector/invokeStaticMethod \ "learn-a-language.important-phrases" \ "main" \ (clojure.core/into-array \ [(clojure.core/into-array java.lang.String '("1"))]))))) There are two things happening here. When you first run lein run, then the process with PID 8420 starts. There are a lot of configuration variables that we don't necessarily need to care about. What's essentially happening is we're saying: Start up the JVM with the leiningen standalone jar on the classpath Use clojure.main as the Java entry point Pass -m leiningen.core.main run 1 as arguments to clojure.main That last step specifies the Clojure entry point, as opposed to the Java entry point. Clojure uses it to load the leiningen.core.main namespace and then execute the -main function within it. leiningen.core.main/-main receives the arguments run 1. We can view Leiningen's leiningen.core.main/-main function on github: (defn -main "Command-line entry point." [& raw-args] (try (user/init) (let [project (project/init-project (if (.exists (io/file "project.clj")) (project/read) (assoc (project/make (:user (user/profiles))) :eval-in :leiningen :prep-tasks []))) [task-name args] (task-args raw-args project)] (when (:min-lein-version project) (verify-min-version project)) (configure-http) (warn-chaining task-name args) (apply-task task-name project args)) (catch Exception e (if (or *debug* (not (:exit-code (ex-data e)))) (.printStackTrace e) (when-not (:suppress-msg (ex-data e)) (println (.getMessage e)))) (exit (:exit-code (ex-data e) 1)))) (exit 0)) Just as we would suspect from the ps output, this is the command line entry point for lein. One of the first things this function does is figure out your project configuration from your project.clj file. Here's ours: (defproject learn-a-language "0.1.0-SNAPSHOT" :description "FIXME: write description" :url "http://example.com/FIXME" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.5.1"]] :main learn-a-language.important-phrases) The most important line, for our purposes, is :main learn-a-language.important-phrases This tells Leiningen what function to execute when you run lein run. If you specify a namespace without a function, as we do above, then Leiningen defaults to using the -main function. Next, if you look about 2/3 of the way down in the leiningen.core.main/-main function, you'll see the apply-task function being called. This calls resolve-task which eventually resolves to leiningen.run, which you can also see on github. This is pretty cool — run is just another task from Leiningen's point of view. Wait... did I just say "cool"? I meant MANLY and ARTISTIC and SPARKLY. But yeah, it looks like basic leiningen architecture includes the leiningen.core collection of namespaces, which handle task resolution and application, and plain ol' leiningen, which appears to be mostly a collection of default tasks. Leiningen uses this same mechanism to execute any function in your Clojure project as a task. Bodacious! I recommend checking out the code for the other Leiningen tasks. You'll see how they do more than just require and run a single function, but the task-running infrastructure remains the same. Anyway, once the run task has been resolved, it is executed and the result is the second process we saw in the ps | grep lein output above, the process with PID 8432. A sub-process is created to enforce isolation between Leiningen and your project. I won't go into how the command for the sub-process gets constructed, as you can figure that all out from leiningen/run. What's more important is what it actually does. The command loads Clojure and tells it to evaluate the following: (do (try (clojure.core/require 'learn-a-language.important-phrases) (catch java.io.FileNotFoundException ___6081__auto__)) (set! *warn-on-reflection* nil) (clojure.core/let [v__6079__auto__ (clojure.core/resolve 'learn-a-language.important-phrases/-main)] (if (clojure.core/ifn? v__6079__auto__) (v__6079__auto__ "1") (clojure.lang.Reflector/invokeStaticMethod "learn-a-language.important-phrases" "main" (clojure.core/into-array [(clojure.core/into-array java.lang.String '("1"))]))))) You can see how the end result is that learn-a-language.important-phrases/-main gets executed with the argument "1". One interesting thing to note about this approach is that the :gen-class directive isn't actually needed. In the manual build at the beginning of this article, we needed :gen-class because we were specifying learn-a-language.important-phrases as the JVM entry point. When we use lein run, the entry point is clojure.main. Once Clojure is loaded, we use it to evaluate some code which loads the learn-a-language.important-phrases namespace and then calls the -main function. Wrapping Things Up So now we know how Leiningen compiles and runs a basic Clojure program! Can you feel your mustache growing? Can you feel your artistry blooming? Are you feeling just a smidge more sparklier? I sure hope so! Here's everything we covered: Building and running a Clojure program manually: Load the Clojure repl Load your Clojure code (make sure it includes :gen-class) Compile your Clojure code. By default code gets put in classes directory Run your code, making sure the classpath includes the classes directory and clojure.jar # load clojure repl $ java -cp .:clojure.jar clojure.main # in Clojure repl user=> (load "learn_a_language/important_phrases") user=> (compile 'learn-a-language.important-phrases) # back to command line java -cp classes:clojure.jar learn_a_language/important_phrases 0 Building and running a Clojure program with Leiningen: Run lein run Magic! Start Java with clojure.main as the entry point Execute the leiningen.core/-main function Setup project config with project.clj Resolve the task to be run Run the task Automagically set the classpath Use clojure.main as the Java entry point Construct some Clojure code to evaluate so that the project's main function gets executed. This happens in a sub-process to enforce isolcation from Leiningen. Check out Leiningen's eval-in-project function for more info. In the next article we'll cover: How Leiningen avoids waste with a trampoline How Leiningen manages dependencies How to distribute a full application How to distribute a library Manliness I hope you enjoyed this article. Please do leave me any feedback - I'm always looking for ways to improve my writing!

How Clojure Babies Are Made: Compiling and Running a Java Program

over 8 years ago | Daniel Higginbotham: Flying Machine Studios

If you’re at all like me, the moment you got your first Clojure program running you belted out, “SOOO MUUUUUUUCH POOOOOOOWEEEEEEEER!” and thrust one or both fists into the air. Then, fifteen minutes later, you found yourself muttering things like “What’s a maven?” and “Classpath what now?” and “What is Leiningen actually doing, anyway? Sure, those are the most handsome mustaches I ever laid eyes on, but can I really trust them?” This post is here to help you out. In it, I will describe how Java compiles and runs programs, and how this relates to Clojure. In an upcoming post, I will also describe how Leiningen makes this process way easier. Here’s an outline of what’s to come: Java Compilation javac Class lookup rules & classpath imports packages JAR files Running Class file classpath Executable JAR Leiningen Running an app without Leiningen classpath management Incorporating Clojure automatically Dependency managemenet (Update: Part 2 of this series is now available!) Java Java is going to rule the universe. The sooner you accept that, the better. If you don’t have the JDK installed installed, you might want to do that now. To check, just try running java and javac in your terminal. By the way - this post assumes you’re somewhat familiar with the terminal. It also assumes you’re familiar with object-oriented programming. Compiling and Running a Basic Program This section will lay a foundation for your understanding of Java. It doesn’t address Clojure directly, but the knowledge you gain will be useful in your Clojuring. Let’s put our game faces on and start with a basic program. Visit the github repo for the examples in this post and clone that baby for all you’re worth. In the folder ShiverMeTimbers, you should find an article named ShiverMeTimbers.java. Here are its contents: public class ShiverMeTimbers { public static void main(String[] args) { System.out.println("Shiver me timbers!!!"); } } Once you’ve recovered from your swoon, cd to ShiverMeTimbers and run javac ShiverMeTimbers.java. If you typed everything correctly and you’re pure of heart, you should now see a file named ShiverMeTimbers.class. $ ls ShiverMeTimbers.class ShiverMeTimbers.java You’ve just compiled your first Java program, son! Now run it with java ShiverMeTimbers. You should see: Shiver me timbers!!! Which I’m pretty sure is like the Lord’s prayer, but for pirates. Anyway, what’s happening here is you used the Java compiler, javac, to create a Java class file, ShiverMeTimbers.class. This file is packed with oodles of Java bytecode which the Java Virtual Machine executes when running a program. When you ran java ShiverMeTimbers, the JVM first looked on your classpath for a class named ShiverMeTimbers. In Java, you’re only allowed to have one public class per file and the filename and class name must be the same. This is how java knows to try looking in ShiverMeTimbers.class for the ShiverMeTimbers class’s bytecode. Also, by default, the classpath includes the directory ., or the current directory. Try running java -classpath /tmp ShiverMeTimbers and you will get an error, even though ShiverMeTimbers.class is right there in the same directory. After java found the bytecode for the ShiverMeTimbers class, it executed that class’s main method. Java’s kind of like C that way, in that whenever you say “run something, and use this class as your entry point”, it always will run that class’s main method. Which means that that method has to be public, as you can see above. (Here’s more info on Java’s access modifiers if you’re curious.) Key Ideas javac compiles java source code to bytecode which the JVM can execute java searches the classpath for the class you specified and executes its main method. That method must be public. In the next section you’ll learn about handling program code that’s spread over more than one file. If you don’t remove your socks now, they’re liable to get knocked off! Packages and Imports In this section, you’ll learn about how Java handles programs which are spread over more than one file. You’ll also learn how to use Java libraries. Once again, we’ll look at both compiling and running a program. This section has direct implications for Clojure programming, so pay attention! Let’s start with a couple definitions: package: Similar to Clojure’s namespaces, packages serve two purposes. They provide code organization, just like clojure namespaces. They also enforce access rules, which we don’t really care about. The directory that a Java file lives in must mirror the package it belongs to. If a file has the line package com.shapemaster in it, then it must be located at com/shapemaster somewhere on your classpath. More about classpath later. import: Java allows you to import classes, which basically means that you can refer to them without using their namespace prefix. So, if you have a class in com.shapemaster named Square, you could write import com.shapemaster.Square; or import com.shapemaster.*; at the top of a .java file so that you can use Square in your code instead of com.shapemaster.Square. Code example below. Now let’s see package and import in action. Here they are as used by the files in make-a-clojure-baby/ContrivedPackageExample. Pay attention to the comments, as they explain a lot of what’s going on Contents of make-a-clojure-baby/ContrivedPackageExample/Conversation.java: public class Conversation { public static void main(String[] args) { // The "ns1" prefix is necessary because ShyGhost belongs to // the "ns1" package, and we haven't imported the classes in // that package ns1.ShyGhost shyGhost = new ns1.ShyGhost(); shyGhost.talk(); } } Contents of make-a-clojure-baby/ContrivedPackageExample/ns1/ShyGhost.java: // The classes defined in this file belong to the "ns1" package. // Notice that this file is in the "ns1" directory. package ns1; // This basically means, "I hate typing the namespace prefix all the // time so please allow me to not do that" import ns2.*; public class ShyGhost { public void talk() { // the shy ghost can't speak for himself and has to get // someone else to do it for him // Notice that even though SuperManlyIguana belongs to the ns2 // namespace, we don't have to use the ns2 prefix SuperManlyIguana smi = new SuperManlyIguana(); smi.talk(); } } Contents of make-a-clojure-baby/ContrivedPackageExample/ns2/SuperManlyIguana.java: // The classes defined in this file belong to the "ns2" package package ns2; public class SuperManlyIguana { public void talk() { System.out.println("Why hello there"); } } You can run all the above code with the following: cd make-a-clojure-baby/ContrivedPackageExample javac Conversation.java java Conversation Can you guess what this outputs? I bet you can! Anyway, so far we’ve established the relationship between importing, packages, and directory structure: Packages organize code and require a matching directory structure. Importing classes allows you to “de-namespace” them. One piece that’s missing, which I alluded to above, is the role of the classpath. Try the following: cd make-a-clojure-baby/ContrivedPackageExample/ns1 javac ../Conversation.java Boom! The Java compiler just told you to hang your head in shame, and maybe weep a little: ../Conversation.java:13: error: package ns1 does not exist ns1.ShyGhost shyGhost = new ns1.ShyGhost(); ^ ../Conversation.java:13: error: package ns1 does not exist ns1.ShyGhost shyGhost = new ns1.ShyGhost(); ^ It thinks that the ns1 package doesn’t exist. But that’s stupid, right? I mean, you’re in the ns1 directory and everything! What’s happening here is that the java compiler is looking for ./ns1/ShyGhost which it can’t find because you’re already in the ns1 directory. This is because the default classpath includes ’.’. Without changing directories, try running javac -classpath ../ ../Conversation.java Et voila! It works! So let’s amend our understanding of the relationship between importing, packages, and the directory structures: when you’re compiling a Java program, Java searches your classpath for packages. Guess what: the same things happens when you’re running a Java program, too. Run the following: cd make-a-clojure-baby/ContrivedPackageExample mkdir hide mv ns1 hide java Conversation Another explosion! Now try: java -classpath .:hide Converstaion Success! I hope this clarifies the relationship between your directory structure, the classpath, packages, and importing. Key Ideas Packages provide a way for you to organize classes. Your directory structure must reflect your package names Imports allow you to be lazy and not prefix a class name with its package javac and java search the classpath for packages JAR Files JAR, or Java ARchive, files allow you to bundle all your .class files into one single file. Run the following: cd make-a-clojure-baby/ContrivedPackageExample jar cvfe contrived.jar Conversation *.class ns*/*.class java -jar contrived.jar It works, just like before. You bundled all the class files into contrived.jar with the file patterns *.class and ns*/*.class. You also indicated that the Conversation class is the “entry point” with the e flag. The “entry point” is the class which contains the main method which should be executed when the JAR as a whole is run. Behind the scenes, java knows which class is the entry point because when you create a jar, the file META-INF/MANIFEST.MF automaticaally gets generated and added. When you add the e flag, the following line gets added: Main-Class: Conversation By the way, you might be wondering why Java isn’t throwing any exceptions like “can’t find package”. The reason is that the JAR file maintains the directory structure. You can see its contents with: jar tf contrived.jar You’ll see that the directory structure is maintained. One final point: a JAR file is really just a Zip file with a “.jar” extension. You can treat it just the same as any other zip file. Pulling it All Together: clojure.jar Let’s pull all of this together with some Clojure! Download the 1.5.1 stable release and unzip it. Then cd to the directory that gets created and run java -jar clojure-1.5.1.jar You should get that most soothing of sights, the Clojure REPL. So, how did it actually start up? Let’s have a look at META-INF/MANIFEST.MF in the jar file: Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: hudson Build-Jdk: 1.6.0_20 Main-Class: clojure.main It looks like clojure.main is specified as the entry point. Where does this class come from? Well, have a look at clojure/main.java on github: /** * Copyright (c) Rich Hickey. All rights reserved. * The use and distribution terms for this software are covered by the * Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php) * which can be found in the file epl-v10.html at the root of this distribution. * By using this software in any fashion, you are agreeing to be bound by * the terms of this license. * You must not remove this notice, or any other, from this software. **/ package clojure; import clojure.lang.Symbol; import clojure.lang.Var; import clojure.lang.RT; public class main{ final static private Symbol CLOJURE_MAIN = Symbol.intern("clojure.main"); final static private Var REQUIRE = RT.var("clojure.core", "require"); final static private Var LEGACY_REPL = RT.var("clojure.main", "legacy-repl"); final static private Var LEGACY_SCRIPT = RT.var("clojure.main", "legacy-script"); final static private Var MAIN = RT.var("clojure.main", "main"); public static void legacy_repl(String[] args) { REQUIRE.invoke(CLOJURE_MAIN); LEGACY_REPL.invoke(RT.seq(args)); } public static void legacy_script(String[] args) { REQUIRE.invoke(CLOJURE_MAIN); LEGACY_SCRIPT.invoke(RT.seq(args)); } public static void main(String[] args) { REQUIRE.invoke(CLOJURE_MAIN); MAIN.applyTo(RT.seq(args)); } } As you can see, the class main is defined. It belongs to the package clojure and defines a public static main method. This is all Java needs to run a program. I hope this has helped clarify Java and how it relates to Clojure! In my next post, I’ll dig in to Leiningen. Fun for everybody!!!

Notes From Tapestry 2013

over 8 years ago | Daniel Higginbotham: Flying Machine Studios

Yesterday I had the immense pleasure of joining about 100 other designers, programmers, educators, data scientists, journalists and storytellers in a day-long conference "designed to advance interactive online storytelling with data." Below is a list of links to many of the presentation examples followed by notes on the presentations. You might like the way it's presented on github better - my blog isn't ideal for this kind of content. If you'd like to make any contributions or corrections, please leave a comment, tweet me @nonrecursive, or submit a pull request :) Resources 13 point, the studio of Jonathan corum, an information designer and science graphics editor at The New York Times. Long jump olympic shadow video nolan reimold strike zone pitches video (I can't find this one) Flea power - how flees jump 512 paths to the white house where 50k guns in chicago came from A Chicago Divided by Killings A New York Times analysis of homicides and census data in Chicago compared areas near murders to those that were not. Residents living near homicides in the last 12 years were much more likely to be black, earn less money and lack a college degree. The Guantanamo Docket: A History of the Detainee Population Methadone and the Politics of Pain Glamour Beasts: The dark side of elephant captivity Interactive: The tragic family tree of Thonglaw one of the interactive storytelling pieces associated with Glamour Beasts above Driving Shifts Into Reverse unconventional mapping of time as a function of the x and y axes Driving Safety, in Fits and Starts same ideas as above Snow Fall: The Avalanche at Tunnel Creek WOW The Invention of Hugo Cabret the book that inspired "Snow Fall" "twenty eleven" no longer sure what this refers to "with olga" not sure what this refers to either Chris Ware comic artist Stephen Few leading thinker in data viz Monstrous Costs, Nigel Holmes controversial (?) chart US job loss bar chart, bush administration vs. obama administration clever to make the bars point downward - association downward with worse the jobless rate for people like you the grimace project - combine facial expressions The Influencing Machine recommended (?) by Scott McCloud www.thersa.org The Infinite Canvas His Face All Red Bongcheon-Dong Ghost Creepy-ass Korean horror comic Bloom Like an Artist Weathercraft: A Frank Comic Keynote by Jonathan Corum When we tell stories we adjust them based on audience - can't do that with a graphic Have an audience He partitions his audience into reader viewer listener user Focus on the people rather than the mechanism/content delivery vehicle A colleague think s of designing for bart simpsons vs lisa simpsons quick overview (Bart) vs ability to deep dive (Lisa) three types of people (science graphics) high school science student busy commuter - how to keep them interested instead of going to their phone, playing a game his grandmother - does it pull together as a cohesive visual whole? whole goal is to design for someone else tensions oversimplification vs. overwhelming detail explanation vs. decoration storytelling vs interactivity - narrative vs exploration don't be your own audience show ideas + evidence understand, translate, display, explain find the central idea. find one idea to use as the basis for your graphic respect the reader - help them through the story allow for multiple entry points - compartmentalized graphical sections interactive tools to allow reader to pace themselves through use disparate scales to give context. Example, mixing longjump distances with free-throw line add meaningful annotations close proximity between graphics and labels don't make people go back and forth between graphics and labels another way to provide context example: annotate each step in a sequence. flea jumping show change motion show large scale, small scale what's happening each step change in form is another kind of change reduce complexity and opportunities for confusion adding interface can be adding complexity reduce tedium interact with data, not the interface strip out tedious activities - usability visualization is not explanation dont let technology drive add enough information beyond your visualization to explain a pattern in data or structure your visualization to reveal and explain patterns reveal patterns layer multiple data sets respect the data show what's unique about it if your visualization can apply to something completely different, you might not be telling the unique story. detainees vs cups of tea edit - throw things away. throw as much away as possible but actually tell a story If you're spending most of your time editing you know you're on the right track apply common sense vigorously Showing is Not Explaining, Pat Hanrahan trying to explain Euclid's algorithm for Greatest Common Divisor algorithm animation / explanation problems with animation motion is fleeting and transient cannot simultaneously attend to multiple animations ... more Pat showed the animation of the algorithm, but it didn't really explain how the algorim worked I had trouble thinking of what to note for this one, would greatly appreciate contributions here. Choosing the Right Visual Story, Cheryl Phillips Side note: This seemed mostly aimed at journalists. What's the story? data without a theme is just a bunch of data - not a story who what when where why how - oldies but goodies interview your data. think of it as the man on the street. keep asking it questions avoid "notebook dump" don't put every last detail in the story use the nutgraf (theme) to help define a strong visualization data is more than numbers -- what little stories make up the larger whole which can be visualized? example: methadone the politics of pain example: family tree of songlaw 29, Nigel Holmes My sparse notes here don't really do the talk justice, probably because I was enthralled with the presentation. Especially Nigel attempting the long jump - I doubt any of us will forget seeing that! 29 is not interesting in itself, but interesting in context you understand something when you see it next to something you already something understand context is the key to understanding The Art of Honest Theft: Evolution of a connected scatterplot, Hannah Fairfield How graphics influence each other if you move away from plotting time against the horizontal you can reveal interesting trends See Driving Shifts Into Reverse and Driving Safety, in Fits and Starts what's next? one technique: associate ancillary content (animations) with scroll so that extra information shows up in a way that it's tied to what the reader is reading at that moment (this idea shows up a lot during the conf) * focusing on immersive content * it's important to carve out time, even just 10%, to play The Why Axis, Bryan Connor nick felt (?) was inspiration is a critic on the why axis, but doesn't mean that in a negative way "the finished piece frequently acts as a seductive screen that distracts us from the higher level of investigation" move past being psychics into being an investigator: as a critic, move from guessing to asking once you know the objective of the visualization you're able to judge whether it succeeded or failed designers: provide retros. be accessible to critics Visual storytelling in the Age of Data, Robert Kosara academics don't get the idea of presenting data, communicating data. it's just an afterthought argues that stylizing charts is quite useful. helps to tell the story. emotional impact? example: monstrous data by Nigel Holmes there's a danger to telling stories can lead you down the wrong path example: driving an electric car in the parking lot until the battery runs down story telling potential of charts Story Depth Tells a Story Facts Narrative Information Scent Focus Audience Author storytelling affordances (I love the idea of storytelling affordances) the form which lends itself to storytelling what are they? reading direction, left to right in the famous napoleon chart, the area gets thinner follow along a line, like following a journey on a map uses the driving safety in fits and starts article as example animations direction - the bush admin vs. obama admin us job loss bar chart effective way of walking you through developments narrative ties facts together provides causality walks you through a story facts - story depth focus - tells a story kind of the natural enemy of more data you must be selective in presenting data for it to be a story information scent - story depth hints used to guide people, indicate that there's more data. example: the jobless rate for people like you present a lot of information, but focus only on one bit. provide other data in a less visually prominent manner author - tells a story audience - story depth "we're at the cusp of something amazing and powerful that goes way beyond what's out there right now" comics and visual communication, scott mccloud in a 10" cube box of air is information wifi, cell, radio until there's something to decode that air, it really is just empty we both receive meaning and create it on the fly the artist gives a hint of life, and the audience will meet them half way cartoonists simplification creating a kind of human calligraphy write with pictures human calligraphy takes many forms body language facial expressions there are six primary emotional expressions anger disgust happiness surprise sadness fear the six primary combine anger + happiness = cruelty you can paint on the face the variety of emotions the grimace project apparently useful for kids on the autism spectrum secret ingredient of the look of love is sadness all pictures are words all pictures speak all pictures have something to say something happens in lower grades - teach kids to write and to draw at a certain point we teach kids words can be used for lists poems express themselves but pictures - we try not to be too specific this overlooks the fact that pictures have a multiplicity of uses can transmit messages, emotions as a result we have a society that divorces pictures from... specificity? we have downstream pictures - advertising don't have as much going upstream comics are a way to send messages upstream - can go out to a mass audience but retain their subjectivity will eisner believed comics can teach the influencing machine tone is important readiness.gov spawned great parodies now people are empowered to combine words and pictures anyway they want to example: historical event facebook pages we need to be vigilant of cognitive load time the speed at which individual parts load is the speed at which we can convey complex info the rsanimate series, www.thersa.org synchronization of data/visuals with content "i'm not going to even allow you to think about anything other than what I'm talking about right now" order of presentation matters "if I don't need to think it, I don't need to see it" "the quicker the parts the richer the whole" cognitive load "form and content must never apologize for each other" the grammar of comics is putting one picture after another creating a flow of time between images as we move through space we move through time any two images, we'll find a story, we'll find a narrative cartoonists are finding the poetry in the gaps between frames as storytellers, you want your audience to lose themselves in the story all narrative art forms are based on extremely simple principles in comics: space = time "each successive technology would appropriate the previous technology as its content" grumpy old man rant about form factor we see the world as rectangles in landscape mode what possibilities open up when you consider the monitor to be a window? lots of cool examples of comics taking advantage of the possibilities inherent in browsers responsive web design separating content from presentation a noble impulse but at the same time there are art forms which are fixed. comics is one of them the form matters! in comics, the spatial relationship is part of the artistic intent when you have violent shifts in the landscape, the only thing you can do is hold on to basic principles in comics, people losing themselves in the story Many thanks to the hosts for putting together a great conference. I'm looking forward to next year's!

Datomic for Five Year Olds

over 8 years ago | Daniel Higginbotham: Flying Machine Studios

If you're interested in Datomic, you may have been deterred from checking it out because you lack the enthusiasm necessary to slog through 3 keynotes, 8 interviews, and innumerable walls of text in order to get a basic idea of what the hell it is and whether you should use it. Well, good news! I've done all that for you! How could I resist your winning smile, you charmer you? After you're done with this article, you will have a solid conceptual grasp of the three key ways that Datomic is unique. You will also understand key Datomic terms. This will make it much easier to actually get your hands dirty with Datomic if you decide to investigate it further. Overview of Datomic's Three Righteous Pillars of Databasing Datomic differs from existing solutions in its information model, its architecture and its programmability. Below is a quick overview. We'll cover each of these righteous pillars of databasing in much more detail. Relational DB Schemaless DB Datomic Information Model The unit of information is an entity in a relation and you alter it in place, forgetting previous values The unit of information is a schemaless document and you update it in place, forgetting previous values The unit of information is a fact comprised of an entity, attributes, values, and time. You accrete the assertion and retraction of facts instead of updating in place. Architecture Database is a monolithic system responsible for querying, ACID compliance, and storage Same as RDBMS Separate querying, transacting, and storage in such a way that you get ACID compliance, read scalability, and more power within the application using Datomic Programmability Applications abstract SQL by performing string manipulation Interact with proprietary data structures which are more programming-friendly than SQL but less powerful Datalog, completely uses data structures and has as much power as SQL First Righteous Pillar of Databasing: Information Model A database's information model is defined by its rules regarding the way entities and attributes relate to each other — for lack of a better term, its schema system. Yea, the choice between a relational and schemaless database is probably your primary concern in choosing a database because of its influence on how you write and grow your programs. You'll see that Datomic's schema system isn't as rigid as the relational model but affords more power than a schemaless database. Additionally, a database's information model is defined by its approach to time. Datomic's approach to time is different from most, if not all, existing databases. Below are the schema systems for relational dbs, schemaless dbs, and datomic, followed by a comparison of the way the information models handle time. Relational Schemas You're probably already familiar with the relational model. Here are key definitions: Entity An entity is tuple within a relation. It is comprised of a fixed set of attributes. In practice, an entity is a row in a table. Attribute A name + a data type. A column in a table. Attributes do not exist outside of relations. Attributes in different relations are always logically distinct, even if they hold the same type of data. Relational Probably the most important fact about the relational model is that it's rigid. Every entity must belong to a rigid relation. There's no way to add or remove attributes except by altering the structure of the relation, which in turn alters every other entity in that relation. Schemaless Schemaless (NoSQL) databases were created to address the rigidity of relational databases. They offer a few facilities for organization - for example, collections in MongoDB - but forego any structure for entities. This lack of structure comes at a cost: it limits your query power and forces data integrity concerns into your application. Entity For our purposes, a document. A document has no restrictions on what attributes it can have. Attribute A name. Attributes can hold values of any type, and an attribute belonging to one entity is logically distinct from all other attributes, even those belonging to other entities within the same collection. Relationships between attributes are enforced within the client application. Schemaless Datomic Schemas In Datomic, a schema defines a core set of attributes which effectively act as data types. An entity can possess any attribute without restriction. In this way, entities have more structure than a schemaless database but are more flexible than in relational databases. Additionally, you retain the query power of the relational model without having to handle data integrity concerns in your client applications. Entity A map of attribute/value pairs. Entities have no fixed shape; they can be comprised of any attributes defined in the schema. Attribute Name + data type + cardinality. Attributes themselves can be thought of as data types. They differ from attributes in the relational model in that they exist independently of rigid tables Datomic Time Datomic's notion of time might be unfamiliar for those unacquainted with Rich Hickey's thoughts on time and how it relates to identity, value, and state. To get up to speed, check out my article, "The Unofficial Guide to Rich Hickey's Brain" In both relational and schemaless databases, there's no built-in notion of time. It's always "now" - the database doesn't hold on to the past. Whenever you update or delete a row, you have no way of retrieving previous states for that row. This is the source of many headaches for us. If you've ever used or any code that requires multiple trips to the database in order to perform one unit of work, you know what I mean. In between each trip, you have no way of knowing whether the database has been been altered. You can end up writing all kinds of hacks to avoid race conditions and still not end up with a bulletproof system. In addition, it's virtually impossible to query past states. You might not think this is really matters because you're used to the limitation, but being able to easily query past states is very powerful. In Datomic, time is a first-class concept. All changes to the database are associated with a transaction, and each transaction has a timestamp. Rather than updating an entity in place and "forgetting" its previous state, datomic "accretes" states. In this way, you're able to retrieve all previous states for an entity. Datomic Another way of thinking about it is that the database is an identity which we superimpose on a time-ordered collection of values. In this line of thinking, a "value" is the database as whole - its collection of entities and attributes. When you run a transaction which creates, updates, or deletes entities, you create a new database value. You're always able to say, "I want to work with the value of the database at this point in time." Datomic's Information Model Summarized In Datomic, the unit of information is a fact. A fact is comprised of an entity, an attribute, a value, and time. If you were to say "The princess", that would not be a fact. If you were to say, "The princess's favorite condiment", that would not be a fact. A fact would be "The princess's favorite condiment is mustard," that would be very close to a fact. "The princess's favorite condiment is mustard as of 10pm on February 10th." is a fact. Facts don't go away. If the princess's tastes change so that she prefers sriracha, it's still useful to know that in the past she preferred mustard. More importantly, new facts don't obliterate old facts. In Datomic, all prior facts are available. Also, in Datomic, facts are called "datoms" and not "facts" because it's more exciting that way. For more, check out The Datomic Information Model, an article by Datomic's creator, Rich Hickey. Second Righteous Pillar of Databasing: Architecture Existing databases couple the following into one monolithic product: Reading Writing Storage Datomic decouples these capabilities. All writing is handled by a single "transactor", ensuring ACID compliance. This is the only point of similarity to existing solutions. Side note: the diagrams at the Datomic Architectural Overview page are quite awful. Besides being ugly, it is difficult to tell what the visual components signify. What do the arrows mean? What does the dashed border mean? WTF is "Comm"? Please, please fix these. OK, rant over. Your application communicates with datomic through the Peer Library, a component which exists within your applicaiton. Using the Peer Library makes your application a Peer. I think the Datomic guys were trying to avoid the term "client" here, but a Peer is essentially a client. What makes a Peer diffeerent from the usual database client, however, is that querying happens primarily in the Peer. By the way, when I say "querying" I mean "read querying." Instead of sending a query "over there" to the database server, which runs the query and gives the client results, the Peer Library is responsible for pulling the "value of the database" into the Peer so that querying happens within your actual application. If you were to create a script out of this action-packed drama, it would look like: Peer: Peer Library! Find me all redheads who know how to play the ukelele! Peer Library: Yes yes! Right away sir! Database - give me your data! (Peer Library retrieves the database and performs query on it) Peer Library: Here are all ukelele-playing redheads! Peer: Hooraaaaaaaay! The most novel thing here is "retrieves the database". In Datomic, you have the concept of the "value of the database" - the database as a whole, as a data structure. The Peer Library is responsible for pulling as much of that data structure into memory as necessary to satisfy your queries. In this way, your queries run largely in your application rather than in a central database server, allowing for easy read scalability. It gives more power to your application rather than keeping all the power in a central server, where you have to worry about bottlenecks. Finally, Datomic does not implement its own storage solution but instead relies on storage as a service, allowing you to choose which storage backend you use. Right now, the following backends are supported: The filesystem DynamoDB Riak Couchbase Infinispan SQL database Third Righteous Pillar of Databasing: Programmability In Datomic, everything is data! Your schema is data! Queries are data! Transactions are data! This is great because it's easier to manipulate data structures than it is to perform crazy string concatenation like you do with SQL. Check out these example queries. Resources The Design of Datomic Rich Hickey on Datomic The Value of Values The Datomic Information Model Datomic docs The End That's it for now! If you have any suggestions or corrections please let me know.

The Unofficial Guide to Rich Hickey's Brain

almost 9 years ago | Daniel Higginbotham: Flying Machine Studios

The Rich Hickey Function Part of my excitement in learning Clojure has been being exposed to Rich Hickey's thoughts on programming. Rich Hickey has a clear, consistent way of viewing fundamental programming concepts that I think any programmer would benefit from. Every time I watch one of his talks, I feel like someone has gone in and organized my brain. In this article (and more to come (possibly)), I begin my attempt to catalog Mr. Hickey's unique viewpoint. Eventually, I would like to produce a concise summary of his ideas. My hope is that this will provide an easily-scannable reference to Clojurists and an accessible introduction to non-Clojurists. What follows is derived from Rich Hickey's talk, "Are we there yet?" Introduction Today's OOP languages - Ruby, Java, Python, etc. - are fundamentally flawed. They introduce accidental complexity by building on an inaccurate model of reality. Where they have explicit definitions for the following concepts, the definitions are wrong: Value State Identity Time Behavior Below, we'll contrast the OOP viewpoint on each of these topics with the Functional Programming viewpoint. But first, we'll compare the models of reality which underlie OOP and FP. It's this underlying difference which gives rise to their different approach to the topics above. Metaphysics, Programming, and You: Comparing OOP and FP When talking about metaphysics things tend to get a little fuzzy, but hopefully this will all make sense. As the ever-trusty Wikipedia explains, metaphysics attempts to answer two basic questions in the broadest possible terms: What is there? What is it like? Rich Hickey explains the difference between OOP and FP metaphysics by contrasting their explanations of what a river is. Object Oriented Programming In OOP metaphysics, a river is something which actually exists in the world. I know, I know, I can hear what you're saying: "Uh... yeah? So?" But believe me, the accuracy of that statement has caused many a sleepless night for philosophers. The wrinkle is that the river is always changing. Its water never ceases to flow. In OOP terms, we would say that it has mutable state, and that its state is ever fluctuating. The fact that the state of the River Object and that Objects in general are never stable doesn't stop us from nevertheless treating them as the fundamental building blocks of programs. In fact, this is seen as an advantage of OOP - it doesn't matter how the state changes, you can still interact with a stable interface and all will work as it should. An object can change, but to all observers it is still considered the same object. This conforms to our intuitive sense of the world. The position of the electrons of the coffee in my mug matters naught; the coffee still interacts with my taste buds in the way I expect. Finally, in OOP, objects do things. They act on each other. Again, this conforms to our intuitive sense of the world: change is the result of objects acting upon each other. A Person object pushes on a Door object and enters a House object. Functional Programming FP Metaphysics In FP metaphysics, we would say that we never step in the same river twice. What we see as a discrete thing which actually exists in the world independent of its mutations is in reality a succession of discrete, unchanging things. The "river" is not a thing in and of itself; it's a concept that we superimpose on a succession of related phenomena. This concept is very useful - I won't belabor that point - but it is just a concept. What really exists are atoms (in the sense of atomic, unchanging, stable entities) and processes. The river is not a stable object; rather, it is a succession of related atoms which are generated by some kind of process. These atoms don't act upon each other and they can't be changed. They can't do anything. Change is not the result of one object acting on another; change is the result of a process being applied to an unchangeable atom. To say that something has changed is to say, "Oh, there's a new atom in this stream of atoms." It's like saying that HEAD points to a new commit in your Git repo. OK! Enough with metaphysics. Now let's describe the more immediately useful topics, starting with Value. Value It's obvious that numbers like 3 and 6 and 42 are values. Numbers are stable, unchanging. It should also be obvious that OO languages have "no proper notion" of values in this sense. As Rich Hickey points out, you can create a class whose instances are composed of immutable components, but there is no high-level concept of immutable value implemented as a first class construct within the class. This is one of the main causes of headaches when doing OOP. How many times have you pulled your hair out trying to figure out how an object's attribute got changed? The fact is, in OO languages there is no built-in mechanism to ensure that the object you're dealing with is stable. This is the big reason why concurrent programming is so difficult. Even in single-threaded programs this is a problem, and it's one of the reasons why we develop sprawling test suites. You can't be sure if a method call on your Toupee object is somehow going to cause a change in your HipsterGlasses object. By contrast, in FP languages emphasis is placed on working with immutable values. Since these values can't change, a whole class of problems simply disappears. Identity In the video, Mr. Hickey says: The biggest problem we have is we've conflated two things. We've said the idea that I attach to this thing that lasts over time is the thing that lasts over time. In FP, identity is essentially a name we give to a sequence of related atoms. "River" refers to the sequence R1, R2, R3, etc, produced by the river process. This is directly analogous to HEAD in Git - it's just a name which is used to refer to actual values. In OO, there really isn't such a distinction. Or as the man himself says, Identity is a putative entity we associate with a series of causally related values (states) over time. It's a label, a construct we use to collect a time series. State In OO, there is no real clear definition of state. Maybe it's, "the values of all the attributes within an object right now." And it has to be "right now", because there's no language-supported way of holding on to the past. This becomes clearer if you contrast it with the notion of identity in FP. In the Hickeysian universe, a State is a specific value for an identity at a point in time. (For me, this definition really clarified my own thoughts.) Time There's no real notion of time in OO. Everything's just "right now". This is part of what causes problems in concurrent programs. By contrast, in the FP worldview we've been exploring, time is well-defined. Time is a by-product of ordering states within an identity. (By the way, I'd really like to write more here, and would appreciate any suggestions.) Behavior Finally, behavior. I don't have too much to write here - I might need to watch some more talks - but I'll include Mr. Hickey's thought-provoking observation: There is no behavior. When you get hit by lightning, who's behaving? This may be what inspired Steve Yegge's post, Execution in the Kingdom of Nouns. The End Well, that's it for now. I hope you've found this post useful. If you have any suggestions, I'd love to hear them!

My Failed(?) Social Web Experiment Still Haunts Me

almost 9 years ago | Daniel Higginbotham: Flying Machine Studios

What will people post if they can't include links and can't use a nickname? This was the question I was looking to answer back in 2007 I launched nobodynotes.com. It was a very simple web site, consisting mainly of a text field and all the posts which people had entered in the text field. You can see it, unstyled, thanks to the wayback machine. Initially, the posts were interesting to me, but probably unremarkable to anyone else. Bits of poetry, parts of song, random curses. But then something unexpected happened. More and more of the posts were written in Persian. Persian!? How did that happen? When I looked at Google Analytics, it showed that the majority of my visits were from Iran, and that most of my visitors were referred from Yahoo Mail. This was not something that I had anticipated. It seemed like the site was going viral in Iran, with people emailing it to each other specifically in Yahoo Mail. Going through the site, I noticed that some of the posts referenced others; each post had a number, and visitors would specify the number of the post they were replying to. But why? Why all this activity from Iranians? I wanted to know, but sadly had no direct way of finding out. After all, everyone was anonymous. Not knowing what else, to do, I tried using Google Translate to see what people were saying. One post in particular stood out. It's probably irretrievable now, but a young man essentially wrote, "Help, I am a gay man living in Iran. I cannot tell anyone, or I will be killed. Please, someone help me." To this day, that post haunts me. Who was this young man? Was he able to find help? Where is he today, and is he OK? I doubt I'll ever know the answer. I eventually shut down the site as traffic slowed to a trickle. But pieces of it remain. It was connected to a twitter account, where you can still read fragments of those posts by anonymous Iranians. And there's the wayback machine, which offers larger slices of conversation. And there's Google Translate, offering me its demented assistance in trying to understand what was happening a world away: Staring at the horizon, leaving the logo side of life, rectified love sitting in contemplation, madness, stood an innocent face, a tired face , relying on the fact, sad face smiling at what an idea! that madness pounding deep into each side. .

A Detailed Look at a Small DCI Refactoring in Ruby

almost 9 years ago | Daniel Higginbotham: Flying Machine Studios

UPDATE: this isn’t actually DCI! Whoops! In this post I go over a small refactoring to clean up some code in Whoops by implementing the DCI pattern. I’ll cover the actual code changes and include my usual hand-wringing about what could be done better. This refactoring was inspired by Jim Gay’s book Clean Ruby. Jim Does an excellent job of concisely explaining what the DCI pattern (data, context, interaction) is, why it’s useful, and how you can implement it in Ruby. This post isn’t a review of Clean Ruby, but I will say that it’s worth the money – it’s actually rekindled my enthusiasm for Ruby! Background Whoops is a free, open-source Rails engine for logging. It’s similar to Airbrake, Errbit, and Exceptional except that it’s not limited to exceptions – you can log any event. The Original Code and Its Defects The code that got refactored is concerned with processing new events. You can see the entry point to this use case at lines 18-28 here: whoops-dci-refactoring/event.rb 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92class Whoops::Event include Mongoid::Document include FieldNames belongs_to :event_group, :class_name => "Whoops::EventGroup", :index=>true field :details field :keywords, :type => String field :message, :type => String field :event_time, :type => DateTime index([[:event_group_id,Mongo::ASCENDING],[:event_time, Mongo::DESCENDING]]) validates_presence_of :message before_save :set_keywords, :sanitize_details def self.record(params) params = params.with_indifferent_access event_group_params = params.slice(*Whoops::EventGroup.field_names) event_group_params[:last_recorded_at] = params[:event_time] event_group_params event_group = Whoops::EventGroup.handle_new_event(event_group_params) event_params = params.slice(*Whoops::Event.field_names) event_group.events.create(event_params) end def self.search(query) conditions = Whoops::MongoidSearchParser.new(query).conditions where(conditions) end def set_keywords keywords_array = [] keywords_array << self.message add_details_to_keywords(keywords_array) self.keywords = keywords_array.join(" ") end def sanitize_details if details.is_a? Hash sanitized_details = {} details.each do |key, value| if key =~ /\./ key = key.gsub(/\./, "_") end if value.is_a? Hash child_keys = all_keys([value]) if child_keys.any?{ |child_key| child_key =~ /\./ } value = value.to_s end end sanitized_details[key] = value end self.details = sanitized_details end end def all_keys(values) keys = [] values.each do |value| if value.is_a? Hash keys |= value.keys keys |= all_keys(value.values) end end keys end private def add_details_to_keywords(keywords_array) flattened = details.to_a.flatten flattened -= details.keys if details.respond_to?(:keys) until flattened.empty? non_hash = flattened.select{ |i| !i.is_a?(Hash) } keywords_array.replace(keywords_array | non_hash) flattened -= non_hash flattened.collect! do |i| i.to_a.flatten - i.keys end.flatten! end end end The above record method calls handle_new_event in Whoops::EventGroup (lines 21-41): whoops-dci-refactoring/event_group.rb 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65class Whoops::EventGroup # notifier responsible for creating identifier from notice details include Mongoid::Document include FieldNames [ :service, :environment, :event_type, :message, :event_group_identifier, :logging_strategy_name ].each do |string_field| field string_field, :type => String end field :last_recorded_at, :type => DateTime field :archived, :type => Boolean, :default => false field :event_count, :type => Integer, :default => 0 class << self def handle_new_event(params) identifying_params = params.slice(*Whoops::EventGroup.identifying_fields) event_group = Whoops::EventGroup.first(:conditions => identifying_params) if event_group event_group.attributes = params else event_group = Whoops::EventGroup.new(params) end if event_group.valid? event_group.send_notifications event_group.archived = false event_group.event_count += 1 event_group.save end event_group end end has_many :events, :class_name => "Whoops::Event" validates_presence_of :event_group_identifier, :event_type, :service, :message def self.identifying_fields field_names - ["message", "last_recorded_at"] end # @return sorted set of all applicable namespaces def self.services all.distinct(:service).sort end def should_send_notifications? (archived || new_record) && Rails.application.config.whoops_sender end def send_notifications return unless should_send_notifications? matcher = Whoops::NotificationSubscription::Matcher.new(self) Whoops::NotificationMailer.event_notification(self, matcher.matching_emails).deliver unless matcher.matching_emails.empty? end end I’ve included the full source of the files to show why this code doesn’t belong. See, one of the main ideas in Jim’s book is that code becomes unmaintainable when we start overburdening classes with methods that are only needed in specific contexts, or use cases. Additionally, we scatter all the code needed in one context across multiple classes and files rather than keeping it all in one place, creating a cognitive burden. Violating The Single Responsibility Principle In this case, the context is “record an event.” The code we’ve added to the Whoops::Event class begins to overburden the class by giving it a new responsibility and requiring that it “know” more about the outside world. Here’s what the class is already responsible for, along with what we’re adding: CRUD events Massage event data before persistence NEW Prepare event group data (lines 21-23) NEW Initiate event group handling of new event (line 24) NEW Figure out how to separate event-relevant data from event params (line 27) In this case, the impact isn’t that high because we still don’t even have 100 LOC for the entire class. But by continuing to shove code into a class just because a class’s data or existing behavior is somehow involved, we transform our classes from usable, focused abstractions into code warehouses. And I really do mean warehouse. You have to start using signage in the form of comments or mixins to find your way around. “Aaah yes, here we are, the email notification section!” or “Oh shit! it looks like someone put the methods for handling user registration at different ends of the file. Let’s put those two guys together. There, what a tidy warehouse!” The Whoops::EventGroup class was becoming a code warehouse. Here are the new responsibilities brought on by handle_new_event CRUD event groups validate event groups NEW send a notification on new events NEW figure out whether a notification should be sent (lines 56-58) This notification code is most definitely not the responsibility of a Whoops::EventGroup. But here it is, because hey, fat model skinny controller. Spreading Out Related Code The above two files also show how related code gets spread out. In this case, to understand the full behavior of “handle new event” you have to start in the Whoops::Event file, then go to the Whoops::EventGroup file, then return back to the Whoops::Event file. Burdening Your Mind Grapes (For more on mind grapes, see #6 in this cracked article) As anyone who’s dealt with this kind of code can attest, organizing your code this way adds cognitive load. By violating the single responsibility principle, we make it more difficult for ourselves to find the methods we’re interested in. The class’s name starts to lose its meaning as its conceptual category becomes broader and broader until it becomes useless as an organizational unit. It’s like trying to find the latest teen vampire romance novel at a bookstore only to find it in the computer section because the sexy, misunderstood male vampire love interest has an iphone. And everyone knows that having to navigate multiple files in order to understand one use case is a huge pain in the ass. I mean, that’s pretty much why Chris Granger is making Light Table, right? The Refactoring This github diff shows all the changes to DCI-ify the code. As you can see, our Whoops::Event and Whoops::EventGroup classes are no longer violating the single responsibility principle. Now when you dig into those classes you no longer have to wade through code that’s only tangentially related to the real purpose of those closses. We’re no longer burdening our classes with knowledge that they shouldn’t have, reducing coupling. Additionally, all of the code related to handling a new event is now in one place, the Whoops::NewEvent class. This makes it much easier to understand completely everything that happens when a new event is handled. Rejoice! I’m not sure how much more to say about the benefits of this refactoring. It feels a lot nicer to me, and it un-does the problems that existed earlier. Hopefully that’s evident, but if it’s not then please leave a comment or tweet me! The Promised Hand-Wringing There are still some things that I’m not completely happy with in Whoops::NewEvent. First, it basically looks one gigantic method that’s been split up into tiny methods just so that I could name groups of related code. I don’t know that there’s much value in that. Second, lines 13-15 look like they need some sort of explanation. It’s especially not obvious why I’m setting archived to false. Last, I’m not completely happy with the name NewEvent. If you were new to the project you’d probably ask yourself, “Uh… how is that not Whoops::Event.new?” I also considered “Whoops::NewEventHandler” but I was afraid that would summon Steve Yegge and he’d make me read one of his 10,000 word essays. I’d love to get some thoughts on this! Whether or not this is a reasonable DCI refactoring, I think that DCI is a great tool and something OOP folks should investigate.

Look, Ma! No batteries! A Clojure Web App Without Noir

almost 9 years ago | Daniel Higginbotham: Flying Machine Studios

UPDATE http://gratefulplace.com now uses a completely different codebase. You can find it on github. You can read about the new codebase at Building a Forum with Clojure, Datomic, Angular, and Ansible . Awhile back I released OMG! SMACKDOWN!!!, which still holds the title for dumbest Clojure app in existence. To make up for that ludicrostrosity, I’ve created Grateful Place. This time I around I didn’t use Noir, instead going closer to the metal with Compojure and Enlive. The experience was a lot easier than I anticipated, perhaps easier than using Noir. This post will cover some of the more interesting bits of code: Templating heaven with Enlive and Middleman Using macros to enforce full stack consistency Roll-your-own validations Pagination is more fun with macros Using fierce mustaches to run tasks Fun utilities You can find all source code on github. Also, I’m going to be at the Clojure Conj and would love do meet folks. If you’d like to chat, please do tweet me (@nonrecursive) or find me at the conj! App Overview Grateful Place is a social app with only a few models: Users, Comments, Posts, and Likes. Nothing too fancy. If you’re interested in why I made it, here’s a post I wrote about gratitude. Templating heaven with Enlive and Middleman Enlive is the bee’s knees I can say without reservation that Enlive is the best templating library I’ve ever used. I absolutely love it! If you try out one thing from this post, try out the workflow I describe here. Enlive is completely different from Hiccup, ERB, HAML and the like and it’s worth giving it a shot. If you’re not familiar with Enlive, it’s a selector-based templating library. In the same way that CSS allows you to separate structure and presentation, Enlive lets you separate structure and view logic. This is huge! Unless you’re the kind of person who still likes to use font and bold tags in your HTML, you should give Enlive a try. Middleman/Enlive Workflow Using Enlive allowed me to do all my design work using the Ruby library Middleman. This allowed me to build stand-alone web pages so that I could see how they looked without worrying about any view logic. For example, here’s the example home page I put together: clojure/gratefulplace-dissected/v1/assets/generator/source/index.html.haml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42--- title: Grateful Place --- #posts .post .content %p Here is some content. %footer .date Jan 03, 2012 .author by %a{:href => ""} Joe Schmoe .favorite %a{:href => "#"} &#9733; .status Like this %a.comments{:href => ""} 35 Comments .post .content %p Here is some content, too. %footer .date Jan 03, 2012 .author by %a{:href => ""} Joe Schmoe .favorite %a{:href => "#"} &#9733; .status Like this %a.comments{:href => ""} 35 Comments .pagination %a.previous{:href => "#"} Previous %span.page-link.current 1 %a.page-link{:href => "#"} 2 %a.page-link{:href => "#"} 3 %a.next{:href => "#"} Next You might have noticed a few weird things. First, this is written using HAML. Yep, HAML. I prefer HAML over HTML because it’s more concise and the consistent structure allows me to visually scan it more easily. And since my HTML templates and view logic are completely separate, I can write my designs in HAML, preview them with Middleman, and then generate an HTML file for Enlive to work with. Here’s my process in full: Build on every change by cd’ing to assets/generator/ and running fswatch . ./g. This runs a bash script which generates the HTML, CSS, and Javascript files and moves them to the proper directories. Start up Middleman by cd’ing to assets/generator/ and running middleman View site at http://localhost:4567 and make changes to my heart’s content When I’m happy with the changes, create or recompile a Clojure file which makes use of the HTML file with Enlive. It’s necessary to recompile the files so that Enlive uses the updated HTML. That doesn’t happen automatically. Another thing you might have noticed is that in the HAML file I’m showing two example posts. With Enlive, you actually have to add some logic to remove one of the post divs: clojure/gratefulplace-dissected/v1/src/gratefulplace/views/posts.clj39 40 41 42(defpage all "index.html" [posts current-auth record-count page max-pages] ;; don't show the second post as it's just an example [[:.post (h/nth-of-type 2)]] nil Some code to reduce boilerplate Finally, here’s some code I put together to reduce boilerplate: clojure/gratefulplace-dissected/v1/src/gratefulplace/views/common.clj 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47(defonce template-dir "gratefulplace/templates/") (h/defsnippet nav (str template-dir "index.html") [:nav] [logged-in] [:.auth] (if logged-in (h/do-> (h/content "Log Out") (h/set-attr :href "/logout")) (h/do-> (h/content "Log In") (h/set-attr :href "/login")))) (h/deftemplate layout (str template-dir "index.html") [html] [:html] (h/substitute html) [:nav] (h/substitute (nav (current-authentication))) [:nav :ul.secondary :#logged-in :a] (if-let [username (:username (current-authentication))] (h/do-> (h/content username) (h/set-attr :href (str "/users/" username)))) [:nav :ul.secondary :#logged-in :span] (when (current-authentication) (h/content "Logged in as"))) ;; Need to come up with better name ;; Bundles together some defsnippet commonalities for user with the ;; layout template ;; ;; TODO destructuring doesn't work in argnames (defmacro defpage [name file [& argnames] & body] `(do (h/defsnippet ~(symbol (str name "*")) (str template-dir ~file) [:html] [~@argnames] ~@body) (defn ~name [{:keys [~@argnames]}] (layout (~(symbol (str name "*")) ~@argnames))))) The ultimate purpose of this code is the defpage macro. This helps me in a few ways. It lets me easily write view code for each page without having to worry about the surrounding layout. For example, before writing the above code I had to include the navigation transformations in every page, which quickly got annoying. The defpage macro also bundles up the template-dir variable, saving me some keystrokes and some duplication. Finally, it establishes a consistent way of interacting with the controller, a topic I’ll explore more in the next section. Before that, here’s an example of defpage in action: clojure/gratefulplace-dissected/v1/src/gratefulplace/views/posts.clj77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94(defpage show "posts/show.html" [post comments current-auth] [:.post :.author :a] (linked-username post) [:.post :.date] (h/content (created-on post)) [:.post :.content] (md-content post) [:.post :.edit] (keep-when (can-modify-record? post)) [:.post :.edit :a] (set-path post post-edit-path) [:.post :.moderate] (keep-when (moderator? (:username current-auth))) [:.post :.moderate :a] (h/do-> (set-path post post-path) (h/content (if (:hidden post) "unhide" "hide"))) [:#post_id] (h/set-attr :value (:id post)) [:.favorite] (favorite current-auth post) Thanks to defpage, my view code contains only logic specific to that view. I love it! Using macros to enforce full-stack consistency Creating a route→controller interface When I was putting the app together, each of my compojure routes was a unique and beautiful snowflake: (GET "/users/new" [] (users/show-new)) (POST "/users" {params :params} (users/create! params)) (GET "/users/:username" [username] (users/show username)) As you can see, each route captures variables in a completely different way. At first I thought this made sense; I was passing exactly the data that was needed to each controller. However, this quickly became a pain in the ass. For example, if I wanted to make a request param available to a view, I had to modify functions all up and down the stack: in the routes, in the controller function, and in the view function. I ended up making all routes send the full request object to their functions: clojure/gratefulplace-dissected/v1/src/gratefulplace/middleware/routes.clj14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30(defmacro route [method path & handlers] `(~method ~path req# (->> req# ~@handlers))) (defroutes routes (compojure.route/files "/" {:root "public"}) ;; posts (route GET "/" posts/all) (route GET "/posts" posts/all) (route GET "/posts/new" posts/show-new) (route POST "/posts" posts/create! (friend/authorize #{:user})) (route GET "/posts/:id" posts/show) (route GET "/posts/:id/edit" posts/edit) (route POST "/posts/:id" posts/update) The route macro is very handy. It saves me from typing and (for the most part) from burning precious brain cells on thinking, and it enforces the “send the full request” constraint by not allowing me to deviate. Of course, I can still deviate, and this is useful if you’re using a library: clojure/gratefulplace-dissected/v1/src/gratefulplace/middleware/routes.clj56 57 58(friend/logout (ANY "/logout" [] (ring.util.response/redirect "/"))) If I want to get to the params, I just use a let; no big deal: clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/posts.clj38 39 40(defn show [req] (let [id (get-in req [:params :id]) Creating a controller→view interface In the same way that it was useful to create a route→controller interface, it became useful to create a controller→view interface. That was accomplished with this macro: clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/common.clj55 56 57 58 59 60 61 62 63 64 65(defmacro view "provides defaults for the map provided to view functions and allows you to provide additional key value pairs. Assumes that a variable named req exists" [view-fn & keys] `(let [x# {:current-auth (friend/current-authentication) :errors {} :params (:params ~'req) :req ~'req}] (~view-fn (into x# (map vec (partition 2 ~(vec keys))))))) This also helped with one weird problem I ran into with cemerick’s friend library where (friend/current-authentication) returns nil from within an anonymous function (and there are many anonymous functions in my view code). I could circumvent this problem by assigning doing something like (let [current-auth (friend/current-authentication)]). But it seemed to make more sense to just always pass that data along when calling view functions, as you can see in the macro above. The defpage macro also helped in establishing this interface in that it allowed me to destructure the arguments sent by the controller. For example, if a controller had (view posts/show {:post post}) I could create the corresponding view with (defpage show [post current-authentication]) And I wouldn’t even have to worry about the order of the arguments. It really reduced some “incidental complexity” (and I’m almost certainly using that term incorrectly). Roll-your-own validations It wasn’t too difficult to write my own way of doing validations, and I’m satisfied with it for now. Here’s a description of the data structure, with an example following: Validation: a combination of a key and “validation check groups” Validation check group: first element is an error message and the rest are “validation checks”. If any validation check returns false then the error message is added to a list of error messages for the given key. Validation check: a boolean function to apply to one of a record’s values. The value corresponds to the validation key Here’s an example: (validate {:username "joebob", :password "hey"} {:password ["Your password must be at least 4 characters long" #(>= (count %) 4)]}) This validation would fail because the value "hey" for the :password is not at least four characters long. Here’s the main validation code: clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/common.clj16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48(defmacro if-valid [to-validate validations errors-name & then-else] `(let [to-validate# ~to-validate validations# ~validations ~errors-name (validate to-validate# validations#)] (if (empty? ~errors-name) ~(first then-else) ~(second then-else)))) (defn error-messages-for "return a vector of error messages or nil if no errors validation-check-groups is a seq of alternating messages and validation checks" [value validation-check-groups] (filter identity (map #(when (not ((last %) value)) (first %)) (partition 2 validation-check-groups)))) (defn validate "returns a map of errors" [to-validate validations] (let [validations (vec validations)] (loop [errors {} v validations] (if-let [validation (first v)] (let [[fieldname validation-check-groups] validation value (get to-validate fieldname) error-messages (error-messages-for value validation-check-groups)] (if (empty? error-messages) (recur errors (rest v)) (recur (assoc errors fieldname error-messages) (rest v)))) errors)))) if-valid wasn’t strictly necessary but I made it and ended up liking it. It’s a small improvement, but a handy one. Pagination is more fun with macros It was a bit difficult for me to pull the pagination code together. I tried many approaches, and I’m not 100% satisfied with what I have, but it’s better than nothing! The basic approach was to add a couple Korma clauses to a base set of clauses. Here are the base clauses, along with a macro that allows me to add more clauses willy-nilly: clojure/gratefulplace-dissected/v1/src/gratefulplace/models/post.clj23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39(def base (-> (select* e/post) (with e/user (fields :username)) (with e/comment (aggregate (count :*) :count) (where {:hidden false})) (with e/favorite (aggregate (count :*) :count)) (order :created_on :DESC))) (defmacro all [& clauses] `(-> base ~@clauses select)) Here’s the pagination macro: clojure/gratefulplace-dissected/v1/src/gratefulplace/models/helpers.clj4 5 6 7 8 9;; TODO make better! (defmacro paginate ([page num-per-page query] `(~@query (limit ~num-per-page) (offset (dec ~page))))) And here’s the whole thing in action: clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/posts.clj19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36(defn all [req] (let [current-auth (friend/current-authentication) per-page 20 page (str->int (or (get-in req [:params :page] 1))) conditions (with-visibility current-auth {:moderator true :logged-in (or {:hidden false} {:user_id [= (:id current-auth)]}) :not-logged-in {:hidden false}}) record-count (post/record-count (where conditions))] (view view/all :posts (paginate page per-page (post/all (where conditions))) :record-count record-count :page page :max-pages (ceil (/ record-count per-page))))) I definitely think there’s room for improvement here and would love suggestions. Using fierce mustaches to run tasks So it turns out that you can use leiningen to run tasks, kind of like Rake. As far as I know, it’s not as powerful in that you can’t specify dependencies, but it’s still better than nothing! Here’s a file I wrote to allow me to rebuild and re-seed by db: clojure/gratefulplace-dissected/v1/src/tasks/db.clj 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22(ns tasks.db (:refer-clojure :exclude [alter drop complement bigint boolean char double float time]) (:require [gratefulplace.models.user :as user]) (:use (lobos core connectivity))) (defn rebuild [] (rollback :all) (migrate)) (defn seed [] (println (user/create! {:username "higginbotham" :email "daniel@flyingmachinestudios.com" :display_name "higginbotham" :password "password"}))) (defn -main [task-name] (condp = task-name "rebuild" (rebuild) "seed" (seed))) I ran lein run -m tasks.db rebuild and lein run -m tasks.db seed. Fun utilities Here’s some stuff that was fun to write: clojure/gratefulplace-dissected/v1/src/gratefulplace/utils.clj 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24(ns gratefulplace.utils) (defn str->int ([str] (Integer. str)) ([m & keys] (reduce (fn [m k] (if-let [val (k m)] (assoc m k (str->int val)) m)) m keys))) (defn deserialize [m & ks] (reduce #(assoc % %2 (read-string (%2 %))) m ks)) (defmacro self-unless-fn [self fn otherwise] `(let [self# ~self] (if (~fn self#) ~otherwise self#))) The End And that’s it! I hope you’ve found this useful. I’d love any feedback, so please do leave a comment or tweet me or email me. And like I mentioned above – it’d be great to meet some folks at the Conj next week!