Goshido service shutdown
Posted on 07. Jan, 2017 by admin in Product
I’m sorry to announce the goshido.net and goshido Jive app have been shutdown. Thanks to all our customers and everyone who worked on the product.
Kind Regards,
/ger.
Web Performance Tuning Pattern: Database Sharding
Posted on 13. Aug, 2013 by ger in Performance Tuning Patterns
I was recently talking to an engineer at another company and he was asking about my opinions on database sharding.
I’ve worked on a number of web-applications including Goshido and a customer’s application that I’ve recently helped to scale. We successfully used a number of scaling techniques but so far we haven’t had to go as far as database sharding.
In the past, I wrote a pattern language for performance tuning embedded applications [1]. Patterns are a simple literary mechanism to share experience and impart solutions to commonly occurring problems.
So out of curiosity, I decided to do some research on sharding and write it up as a pattern for web application tuning.
Context
- Scaling a web application with a backend database
- Database write performance or capacity is a confirmed bottleneck
- Database configuration tuning has already been implemented
- It is no longer feasible to move the database to a larger server
Problem
Eventually you can arrive at a situation where database write performance or database capacity is the limiting factor for your system scalability.
Solution
Distribute some of your application data across a number of database shards. The application or database implements a translation from an object id to shard database. Essentially your application now has one logical database layered above a number of physical databases.
For example, imagine a micro-blogging application where users are frequently submitting posts. You could use a modulus of the “post id” to distribute each post to one of many database shards. Later when someone wants to read a post you would translate the “post id” to a shard id before retrieving the post from the appropriate database.
This example demonstrates a key-based partitioning strategy. Other strategies include:
- Vertical partitioning: where different types of data are stored on different databases (profiles, posts, users)
- Directory-based partitioning: a lookup table that translates object id to shard
For detailed description of other strategies (like date-time and master indexes) see Max Indelicato’s blog post below [2]. Eric Ries outlines a URL based sharding strategy that enables you to incrementally shard a system over time [3]. He recommends quickly building non-scalable solutions then when the product gets traction, start sharding by identifying “highly-trafficked tables that are rarely joined with any others”.
Forces
- Some NoSQL (NoJoin) databases (i.e. MongoDB) have integrated support for sharding and shard balancing.
- If you are using a traditional relational database (i.e. MySQL) you probably need to implement application level sharding (although some databases support forms of sharding like federations in Windows Azure SQL Database).
- Application level sharding adds complexity.
- Changing sharding strategy or number of shards can be complicated and may involve moving data from one shard to another increasing deployment complexity/time.
- Sharding can impact other system workloads. In the example above, if we wanted to show multiple posts for a specific user we would need to issue read requests to multiple database shards.
- Sharding limits your capability to join across tables. To get around this you may need to modify your application to decompose joins, denormalize the data or copy some data to the shards [6]. This will increase complexity.
- Directory based partitioning has a single point of failure.
Further reading
- “Tricks and techniques for performance tuning your embedded system using patterns”: a series of articles I co-authored with Peter Barry for Embedded Magazine a few years ago part 1, part 2, part 3.
- Max Indelicato’s Primer on Database Sharding has more details on implementing partitioning strategies
- Sharding for Startups by Eric Ries
- Book – Cloud Architecture Patterns by Bill Wilder. While the sub-title mentions Azure, this book is general enough to be applied to other cloud platforms.
- Book – Scalability Rules: 50 Principles for Scaling Web Sites by Martin L. Abbott; Michael T. Fisher.
- Book – Professional Website Performance: Optimizing the Front-End and Back-End by Peter Smith
Photo Credit: Winter Ice Mosaic by Ctd 2005.
Scrum and Agile for devops
Posted on 16. Jul, 2013 by ger in Agile, Guides, Scrum
Devops engineers were part of two recent Agile and Scrum training workshops I ran in Dublin. Here’s some resources that you might find useful if you’re coming to agile from a devops perspective.
Blogs and blog posts
Two of the best blogs I found are the Agile admin and Agile Web Operations. The Agile admin has a good post describing Scrum from an ops perspective. Some devops teams combine Scrum for longer term projects and Kanban for shorter term work. This blog post I wrote recently summarizes a great article by Jim Coplien about the linkages between Kanban and Scrum.
Books
These books include sections on agile devops
- Scott W. Ambler; Mark Lines, Disciplined Agile Delivery.
- Paul Swartout, Continuous Delivery and DevOps
- Michael Hüttermann, DevOps for Developers
- Jez Humble; David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
Learn more
Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for Scrum, Lean and projects with traditional work-breakdown plans. It’s great for both long and short term devops projects.
Photo Credit
Internet Splat Map by Steve Jurvetson
User Stories for Agile Requirements Management
Posted on 26. Mar, 2013 by ger in Agile, Guides, Scrum
We use “user stories” to document and manage requirements. Recently I provided training for one of our customers. Here’s some interesting resources we recommend.
What are user stories?
User stories are a short placeholder for a detailed conversation about requirements – deferring detail until needed. Stories represent customer value and are written in the customers’ terminology.
The general formula for a user story is: As a <type of user>, I want <some activity> so that <some reason>.
For example: “as a registered user, I want the system to warn if my password is easy to guess, so my account is harder to hack.”
Why use them?
I’ve seen studies which state incorrect requirements are the cause of 80% of software project failures. User stories are a simple, low-overhead way of documenting and managing requirements.
Resources
One of the trickiest parts of using user stories is breaking them down to the right size. Richard Lawrence has written an excellent pattern language to help you in this process.
There’s been an interesting debate on the value of acceptance testing.
We recommend the following books:
- User Stories Applied: Mike Cohn
- Agile Software Requirements: Dean Leffingwell
- Specification by Example: Gojko Adzic
- The Art of Agile Development: James Shore
Learn more
Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for Scrum, Lean and projects with traditional work-breakdown plans. It’s great for managing product roadmaps and requirements documented as user-stories.
Project Collaboration Overview & 7 Great Books : Siemens Talk
Posted on 12. Mar, 2013 by ger in Agile, Events, Guides, Leadership, New Ways to Work, blog
Last night in Dublin I spoke to a group leaders from Siemens at an event coordinated by WDHB.
The key points of my talk were:
- Existing tools still don’t solve the problems of communication within large organizations
- Businesses need a flexible system that can scale from gigantic to many small-informal projects
- Culture is even more important than tools. Agile ways of working & leading with short iterations and empowered teams are crucial.
I recommended a number of books that explore the last point:
- Radical Management: Stephen Denning
- Succeeding with Agile: Mike Cohn
- The Future of Management: Gary Hamel
- Management 3.0: Jurgen Appelo
- Getting Things Done: David Allen
- The Way We’re Working isn’t Working: Tony Schwartz
- What Were They Thinking?: Jeffrey Pfeffer
Other speakers at the event were Marie Wallace of IBM and Polly Sumner of Salesforce. Polly recommended a really interesting followup action at end of her talk. She suggested everyone in the room try to buy or get customer support for their own product.
By coincidence I read the following quote in the Siemens “Pictures of the Future” magazine this morning.
-
“Modern information and communication media promote a short attention span and a superficial understanding of the world.” — David Gelernter.
By another coincidence I read this in the Chess Media newsletter this morning.
-
“Communication barriers cost large companies an average of $26,000 per employee per year due to productivity losses.” — Jacob Morgan
Learn more
Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for project collaboration in any kind of business: product development, energy, healthcare & consulting.
You can download my slides.
Dispelling common misconceptions in Lean, Kanban and Scrum
Posted on 17. Nov, 2012 by ger in Agile, New Ways to Work, Scrum
Happiness Door Scrum Training Dublin
They say teaching is the best way to learn something. Recently, I delivered a Scrum training course for a multinational company in Dublin. Afterwards I did some reading to followup on some of the questions that came up. This reading lead me to some interesting new thinking on Scrum, Lean, Agile and even running training courses. It looks like many people have common misconceptions about how Kanban and Scrum fit together.
Some background
Scrum is one of the agile project management techniques that can make your team far more productive. Most of the time I’m CTO of Goshido (cloud software for project communication). Goshido is inspired in part by some of the concepts in Scrum. Some of our customers ask for training in agile project management and Scrum in particular. Our Scrum training is unique because I am still a practicing software developer and project manager, not a Scrum-trainer who delivers training every week.
Lean is a process improvement discipline that software development and startup companies.
Kanban is one of the Lean tools, originally a card or a box used in manufacturing to signal depletion of product, parts, or inventory. It is a way to “pull” work through a system.
Overlapping phases & organizing by project
The company I was training has structured the organization based on roles and skills rather than projects. The analysts are in one team, the UX people another, the developers and testers yet another, and operations another. During the project there’s a handover from one team to another. One of the principles of Scrum is to organize teams around the work that needs to be done for a customer, to mix the disciplines together with overlapping phases. Dustin Curtis tells a great story about the magic of Penn and Teller to illustrate the point that:
- Assembly lines work fantastically well for raw assembly, but they suck the life out of creative people.
Getting feedback during training
During each training day I asked people to provide anonymous feedback both half way through and at the end of the day. I applied Jurgen Appelo’s happiness door technique. I got great feedback from the attendees and was able to make adjustments both during the day and for subsequent days to make the training more relevant and valuable to the people attending.
Kaizen, retrospectives and continuous improvement
One of the technical leads was concerned about the business focus of the product backlog. How would he get technical tasks like a transition to github into the backlog? He knew this would boost the productivity of the software team but what if the product owner didn’t feel the same way?
Conversely, I’ve seen situations in other companies where technical leads have refactored parts of a system without concern for the business priorities. This in turn lead to a surprise then an overreaction by senior management who as a consequence put unnecessary controls in place to enforce visibility.
In my experience, Scrum retrospectives pay big dividends at the end of each sprint but it’s easy to let them slide. In the post “Getting Lean with Scrum”, Christine Hegarty describes how you can add a simple process to retrospectives.
She suggests capturing a single kaizen in each retrospective to be added to the next Sprint backlog as a task. I think it’s a great idea, and could balance the needs of continuous improvement and business priorities while making retrospectives more actionable.
Misconceptions about Lean and Kanban
While reading Christine’s post I also spotted a guest post by Jim Coplien describing Kanban, Lean and Scrum. I remember Jim from the early days of the software patterns movement. He wrote a pattern language for software organizations. Jim is a co-author of the recent book Lean Software Architecture. In the blog post, he clarifies some of the many misconceptions of Kanban and it’s goals.
- The challenge is to develop a learning organization that will find ways to reduce the number of kanban and thereby reduce and finally eliminate the inventory buffer. Remember: the kanban is an organized system of inventory buffers and, according to Ohno, inventory is waste…
He’s also concerned it’s been missappropriated in software development.
- Kanban applies to repetitive work — building the same item again and again… Not everything can be replenished based on a pull system; some things must be scheduled… That is what software is like: We rarely build the same thing again and again.
The ideal Scrum team “does a little analysis, a little design, a little building, and a little testing all at once in very short cycles”. While Kanban can be a good fit for teams that view work as firefighting…
- A good Scrum team automatically enjoys the provisions of kanban when practicing one-piece continuous flow. It is a built-in way to limit work in progress while encouraging teamwork. It gives team members time-boxed autonomy to carry out their plans while enabling them to better meet their planned forecast. Such maturity may be a foundation for later addition of techniques like kanban…
Learn more
Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for Scrum, Lean and projects with traditional work-breakdown plans. Some of our customers do all three together. They map out a long term plan using Gannt charts then each team runs sub-projects as Scrum-Sprints.
Guided Collaboration
Posted on 10. Oct, 2012 by tom in New Ways to Work, blog
At Goshido, we like to talk about “guided collaboration.” Why? Because work is dynamic. A revelation, I know! But I think it’s time to reiterate first principles. Namely, the nature of work has changed, and so must our tools if we’re going to keep up.
Whether it’s Steve Denning writing about self-organizing teams in “The Leader’s Guide to Radical Management” or Charles Pierce writing about “The Limits of Control” in professional sports, you don’t have to look far to see examples of command-and-control failures. Does your work ever fit together like this:
command and control
Of course not. So why even try to structure your work in this manner? Projects as formal entities can be invaluable, and structure is not just a necessary evil. But hierarchies don’t adapt or evolve. They’re static, they’re brittle, and sooner or later they break.
Many projects look more like this:
collaboration without coordination
You know the starting point, and you know the desired endpoint. What you can’t predict is how you’ll get from start to finish. Knowledge work projects evolve. You can’t eliminate complexity, but you can manage it if you get the right variables — who, what, and when — together in one place.
This concept is top of mind for me these days as a great TED talk by Clay Shirky on open-source government is receiving much well-deserved attention. What caught my eye, from a project management perspective, was his discussion of version control systems. Feudalism vs. open-source.
We’re not trying to eradicate feudalism here at Goshido, but we are trying to help make work better. This matters to us. Let us know how we’re helping you or your team be more effective or what we can do better.
Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for projects in any part of your organization.
Goshido and Jive Turn Project Collaboration into Action with !APP Experiences
Posted on 14. May, 2012 by tom in New Ways to Work, Product
SAN FRANCISCO, Calif. and LIMERICK, Ireland, May 2, 2012 — Goshido and Jive Software Inc., the industry’s largest pure-play social business provider, just made business collaboration even easier. Earlier today, Jive launched its Jive !App Experiences, a breakthrough capability that seamlessly integrates business applications into the Social Business workflow. Goshido’s new task and project management !App allows Jive users to turn team collaboration into productive action. It’s the ultimate project execution tool—scalable, flexible and easy to use. The app is available now via Jive Cloud.
“Goshido is an elegant social solution that boosts productivity,” said Jive Apps Market VP, Robin Bordoli. “It can effortlessly handle large or small projects within or across teams, functional areas or entire organizations. Few collaborative tools enable such rich flexibility. It’s a natural fit to illustrate the power of the new Jive !App Experiences.”
“For us, making business more agile is just the start,” said Goshido CEO Tom Brennan. “It’s action and results that really count. That imperative holds true for individuals and their daily to-do lists, and large groups driving complex multi-year projects. And that’s the promise of the new !App capability. Now, users can create and insert Goshido actions right into their Jive documents and discussions. Then, right from within Jive, activities and projects can be coordinated, managed and implemented in Goshido.”
Built for today’s knowledge worker, Goshido is changing how people work together. While conventional project management solutions are focused on ‘management,’ Goshido helps individual employees to ‘accomplish.’ This ‘action’ overlay is Goshido’s essential differentiator. It’s what’s valued most by Goshido users.
With the Jive !App integration, Goshido users can:
- Create actions from within content in Jive, and quickly and easily create actions that can stand alone or be part of a much bigger project.
- Work with actions right within their Jive What Matters stream.
- Launch Goshido from a Jive Action Menu, and get right to work.
Please try Goshido, our collaboration & project management platform. Everyone is a project manager these days. Goshido is the best way to run any project.
The Business Value in Becoming an Agile Enterprise
Posted on 29. Feb, 2012 by ger in Leadership, New Ways to Work
In the last few weeks I’ve been asked the same two questions a number of times:
- Does this agile stuff really make a difference in financial terms?
- We don’t do software development, does agile really apply to us?
In this post, I’ll answer these questions (the short answers are yes and yes). I’ll tell you three short stories. Two stories quantify the benefits of agile. The final story is about a team who uses agile to run marketing campaigns.
What’s in it for us?
Michael Mah in “How Agile Projects Measure Up, and What This Means to You” analyzes two software projects. He compares their performance to projects of similar sizes from a large project-performance database. The projects measured lower costs, shorter time-to-market, and increased quality.
Follett Software used a technique called eXtreme Programming and achieved:
- Dramatically lower costs ($2.2m versus an average of $3.5m for similar sized products)
- Shorter time-to-market (7.8 months versus an average of 12.6 months)
- Less defects (121 versus 242)
BMC Software used Distributed Scrum. They added more staff (92 versus 40 average) to optimize time-to-market. They achieved:
- Dramatically shorter time-to-market (6.3 months versus an average of 15 months)
- Slightly lower costs ($5.2m versus an average of $5.5m for similar sized products). The extra cost of staff was offset by much shorter project duration.
- Slightly less defects (635 versus 713)
These are just two specific examples. David Rico completed a much larger meta-study looking at a large number of projects (which referenced a paper published in the EJIS that I co-authored with Brian Fitzgerald and Kieran Conboy).
Agile beyond software development: a retail marketing campaign
One of our customers decided to apply agile on retail marketing campaigns across 13 European countries.
Before agile they organized everything quarterly. Every quarter their campaigns had a number of open issues that spilled over and took time to resolve the following quarter. The backlog of open issues was ever increasing and pressure was mounting every quarter.
When they applied agile they split the teams into smaller units of 5 people or less. They ran projects in shorter iterations (monthly sprints instead of three month iterations). They observed a number of key benefits:
- Very few issues spilled over from one quarter to the next
- They were able to eliminate the historical backlog
- Team morale improved dramatically
Again this is one specific example. Stephen Denning has written an excellent book “The Leaders Guide to Radical Management” about applying agile principles to management in all kinds of businesses.
Learn more
If you’re a time-pressed business person, read Israel Gat’s short book “The Concise Executive Guide to Agile”. He gives a good overview of agile that focuses on the numbers and how to introduce agile in your enterprise.
If you’re based in the UK, “The Agile Leader” is a team of consultants and coaches who deliver Agile transformation to organisations.
Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for projects in any part of your organization.
Photo Credit
New features: multi-project dashboard & calendar
Posted on 10. Jan, 2012 by ger in Product
We’ve built some new features into Goshido that help you get better visibility of your projects.
Multi-project dashboard
The project list now includes features to tell you the status of all your projects at a glance. For each project you can see:
- Automatically updated %-complete
- Automatic countdown of time-remaining
- Color coded status indicators for actions/projects
When someone completes actions deep within a project, the %-complete and time-remaining is updated automatically at higher levels of the project.
This is reflected on the multi-project dashboard.
Calendar View
The new calendar view shows you actions you’re involved in which have deadlines which are due soon.
Learn more
Thank you for reading. Please try Goshido, our collaboration & project management platform. Goshido can can give you great visibility of all of your projects. We’d love to know what you think of these new features.