Ruby Under a Microscope is an illustrated guide to Ruby internals. No C programming required!
One of the first things I tell potential clients that hire me for an agile transformation is that part of my goals is to make myself superfluous. Although more or less common sense for people with an agile background this provokes more often than never puzzled faces. “Isn’t he supposed to lead the product development process by managing the product team with agile methodologies?” Transforming intangible perceptions into actionable items In fact I’m supposed to enable the product development team to start managing itself in an agile manner. Reads itself like a minor difference in the formulation, but has a major impact on the proceeding. <!-- more -->Of course I start leading & managing the team. Introduce the necessary events, evolve daily stand-ups to short & precise meetings, get everybody on the same page and facilitate the retrospectives. Apart from the latter, a fully developed agile team is able to answer the three magic daily questions and organize the events on its own. When it comes to retrospectives, it’s a little bit different. You need a neutral facilitator here a little longer. For my last client this meant that after the time intensive initial phase of coaching I stopped attending on a daily basis after we both felt comfortable with it. I continued to facilitate the retrospectives for some more weeks, and after we tried and saw that there are still action items and the resulting improvements as tangible outcomes I realized I brought that team to the agile path and futhermore to a state in which it could walk this way without further assistance. Short & direct feedback. <3. Tangible outcomes are amongst the core metrics I am judged by. In the beginning the effects of the new process are still very intangible. “We like that we now communicate more with each other, but we are affraid losing our efficiency!”. When moving from batch processing your specialized task to think a little cross functional this is a natural reaction to the change. Measured by the quantity of product features you deliver it in fact will look like losing efficiency. When judging by the quality of what you deliver the efficiency equation looks quite attractive. In fact after a short period of time the propelled quality becomes quite obvious - be it the lowered technical debts, the increased harmony of the team or the ability to react to customer wishes faster. As you may have realized I used the formulation “measure quantity” and “judge quality”. I did this on purpose since it stresses the very tangible nature of quantity and the more intangible way we perceive quality. In my retrospectives I’m quite strict about actionable items as a result. They make our progress tangible and I see them as a basic metric to measure the progress we make. In fact I put a little square on every action item sticky and we check this box if we accomplished the task in the next retrospective. This is just a small example of validated learning. Which is one of the aspects in the focus of lean entrepreneurship. It is about learning based on the data you collect from your product and finding the right metrics to measure the value and the growth of it. Which is why I see Agile & lean entrepreneurship as a synergy of greatest potential. Participating in the 2012 Agile Design Camp I bootstrapped some provocative slides to bring the audience into the mood for discussing the topic. The core hypothesis here is that agile focusses on the process success while lean entrepreneurship focusses on the product success: ‘Agile vs Lean’ on Slideshare Of course this is only true to some extent. When working in agile environments the chance of meeting practicioners that do not only think in terms of their special subject but also have the product in their vision is quite high. When iterating over your processes lean entrepreneurial thinking as a result can be an outcome quite naturally. This is a result of the fact that the right process framework takes away unnecessary pressure from your team. This enables the practicioeners to actually think about what they build as opposed to being busy with figuring out how to finish tasks in time. I would say this is similar to moving up on the Maslov pyramid - just in the domain of your product quality and not concerning personal life’s quality where Maslov’s famous geometric metaphor is originally pointed at. To quantify if our newly gained freedom actually results in higher value for the customer moving from judging prodcut quality to measuring product quality is essential. One way to achieve this goal is the path of lean entrepreneurship. Viewing from that angle I think it becomes quote obvious why I chose the provocative hypothesis that agile focusses on the process success while lean entrepreneurship focusses on the product success.
I was at Strata New York 2012 last month. Great conference! Thanks O'Reilly media for assembling the industry leaders and running it well.I understand it was too crowded for some of my out-of-town friends. Stepping out to the streets of mid-town Manhattan for a breath of fresh air and calmness wasn't an option either. Maybe O'Reilly can get a bigger space next year?My primary interest in Big Data analysis was structured data analysis i.e. crunching, munging (ETL) and analysis of large dataset in columns and rows.My team deals with 1-2 Terabytes (~ 1 billion rows) of structured data (e.g sales transaction data) regularly for marketing/retail/healthcare analytics. Like others, we're spending a lot of time in Big Data ETL processes and less on Big Data Analysis. Someone at Strata New York captured this well,80% of a Big Data development effort goes into data integration efforts and only 20% of our effort/time is spent on analysis, i.e. interesting things we want to work onI want to flip this equation. I want to be able to spend 20% of our time/effort on Big Data ETL/integration efforts and 80% on Big Data analysis.At Strata, I wanted to check if vendors and open source communities had simplified the Hadoop stack for my use. There have been many improvements in the past year and the products on my list is quite long. We've more players in Big Data space and the solution space is muddy. It is great to have more vendors, experts, communities focusing on Big Data but the product space is crowded, fragmented and CONFUSING (just listing all Apache products discussed at Strata needs 1 full page.)I created a list of products to try out. I wish these products were easy to evaluate (legal paperwork, infrastructure footprint, and ease of setup and execute my use cases).As far as ease of use and powerful data science workbench is concerned, I want to use something like R or even Excel for these steps (Big Data ETL and Big Data analysis), but they are both memory constrained. So, I need other options.Did someone say, Hadoop? Yup, on it. I tried it a few years ago and we're exploring it now. Hadoop/MapReduce is THE infrastructure to power Big Data ETL and Big Data analysis.I also believe that MapReduce and Databases are complementary technologies and the experts agree! See MapReduce and Parallel DBMSs: friends or foes?Here's how I've framed my problem and thinking about the solution space.Big Data ETL processTake big structured dataset (multiple CSV files with total 100M-1B rows) and create DDL/clean/transform/split/sample/separate errors in minutes.Solution options:Unix scripts (shell, awk, perl). Do all of the above in one pass (i.e. read each row only once) quickly. Start with Unix parallel processes, scale to multiple machines (mapreduce-style) only if neededBig Data ETL tools like Kafka?Open source ETL tools (e.g. Talend)Can commercial ETL tools do this in a few minutes/hours?Others?Given any structured data from client (csv), our Big Data ETL workbench takes the data and processes it super fast (detect data types, clean, transform eg. change date formats to our internal standards, separate error rows, create sample, split into multiple clean files)Raw data files have different schema that we auto-detect in processing (only string vs numeric types to begin with).Big Data AnalysisThen we load this data for analysis:RDBMS for well-defined arithmetic/set-based analysisnoSQL database (Lucene/Solr with Blacklight front-end for discovery). Blacklight project: Open source discovery app built on Lucene/Solr. Thinking of it as a discovery app for Big Data analysis. Facets on top of structured data. Slide-dice large structured dataset. We can add visualizations later (e.g. summaries etc.) Checkout this Stata session on Lucene-powered Big Data analysis which confirmed this design hypothesisThe clean split files can be used in any stats tool as well for statistical analysis e.g. SAS for larger data sets, R for smaller ones (often the clean split files are small enough for R)New Big Data tools like ImpalaWe're building proof-of-concepts for Big Data ETL (Unix scripts) and Blacklight discovery app on top of Lucene/Solr. I will share it when its ready. Stay tuned.
I just bought a new laptop with the cool new Windows 8 installed. I must admit that I was a bit skeptical of how the new OS would be, but its totally taken me by surprise, and in a good … Continue reading →
I just bought a new laptop with the cool new Windows 8 installed. I must admit that I was a bit skeptical of how the new OS would be, but its totally taken me by surprise, and in a good way! First, I absolutely adore the new Metro layout. Its like a cool dashboard where I have all my tools and information handy for me to get started. From facebook to gmail to google search, everything is a shortcut on there. And its not just a shortcut! The mails tile shows new ones and facebook one shows the highlights - everything that makes you decide if you want to click that icon or not. Similarly, there are news feeds, weather updates and other handy info. Its really much more useful than the mostly empty windows desktop that exists on previous versions. And its much brighter and colorful too! Secondly, IE 10 is a total pleasure to use. Its super fast, and feels much lighter. I also hear that it finally adheres to global standards too (yay, developers rejoice) ! Thirdly, each new window/ app is a full screen by default. No task bar, no title bar, nothing. It makes the full use of the screen - can it get better ?! Oh well, some things require 2 clicks (including a right click) that should really just need one. Like closing windows. Yeah. You either drag a window to the bottom to ask it to go to hell (or where ever), or you right click, find yourself a cross button and close the damn thing. I wish there was a close button that would appear when I hover around one of the corners, closing a window should be a one click thing. And I could not install Google Chrome on Win 8 - its just kept on hanging up on me. I think the performance of the OS and system in general is pretty good, and the fact that the same system will run on mobile and desktop/ laptop computers is pretty encouraging. May be we should consider how we’ll develop apps for windows along with Android and iOS ? ha ha! I hear Windows 8 is all HTML and JS anyways!
I was sitting at the Cambridge Public Library, and for the first time in months - years? - I felt at peace. In my journal I had just written what's become a kind of mantra for me: "Love without fear." At the time I was going through a very difficult breakup. My relationship of 6 years had been slowly and painfully dissolving, in part because of my own inability to end it decisively. Part of my difficulty what was that I had a lot of guilt over wanting to end the relationship because the person I was with had a long-term illness and I was supporting her both financially and as a kind of house helper or ersatz nurse. I had watched her suffer terribly. I'm not sure how to convey what that was like. How do you share the experience of watching someone you love get a spinal tap? Of wheeling them out of the hospital and to the hotel next door, trying to do it quickly but smoothly because every bump causes excruciating pain? Or how about cleaning the vomit out of a mixing bowl because antibiotics are causing her nausea? Or giving the antibiotics shots yourself, night after night? Or knowing that there was no end in sight? How could I leave someone to face that on her own? After six years, I was in a state of chronic exhaustion and stress. Often I would start my day by sitting at my desk for a couple hours and just staring, not really able to do anything. I knew that something had to change, but I felt too guilty to leave. And, to be completely honest, I was scared about what it might say about me if I left her. Would it mean that I was a callous, heartless person? More, I was scared to live without her. We'd been together for 6 years, and the life I knew was familiar and provided comfort in a way. In the state I was in, I did not feel confident that I could make a large life change. Michael Singer said it well: You will get to a point in your growth where you understand that if you protect yourself, you will never be free. It’s that simple. Because you’re scared, you have locked yourself within your house and pulled down all the shades. Now it’s dark and you want to feel the sunlight, but you can’t. It’s impossible. If you close and protect yourself, you are locking this scared, insecure person within your heart. You will never be free that way. -- The Untethered Soul, by Michael Singer But those words, "love without fear", helped give me strength. Beneath the fear, beneath the raw ache of being a perpetual witness to suffering, I could still feel the beating heart of love. I felt it there, ready, waiting for me without judgment. I felt the desire to open up again, to build honest and meaningful relationships with others. I felt like I had nothing to fear from embracing every moment of life whole-heartedly. The months that followed that day at the library were messy and painful in many ways. But despite many mistakes, I knew that basically I was on the right path, that my heart was opening up more and more and I was learning to love myself and be myself without giving in to my fears of what other people might think or what vague disaster might befall me. Now, more than two and a half years later, I can say without reservation that I took the right path. My life has not been problem-free and I don't ever expect it to be. In embracing love, though, I've learned that freedom and joy can be found in every moment, even when problems do arise. My life has improved as well. Actually, that's a major understatement. My life is freaking awesome right now. About a month ago, I got engaged to the girl I love. A few days ago, I moved out of a drafty, gross old apartment into a beautiful new house that's close enough to my brother and mom to allow me to visit them much more often. And my latest project, something I put together with the idea of lifting up other people, is almost ready to ship. And through it all, I keep telling myself: Love without fear. Love without fear. Love without fear.
Want to do a line chart with d3? There are no ready APIs right? At least none that I could find. What I did find was http://benjchristensen.com/2012/05/02/line-graphs-using-d3-js/ (very useful!) and I hacked up a line chart taking cue from there. … Continue reading →