July 2, 2011

Getting Started with Git and SVN

Filed under: Articles,Development — Tags: git git-svn code.google.com googlecode tutorial — Kris @ 9:32 am

Getting Started with Git and SVN

What is Git, and why would you want to use it?

Git is a distributed version control system that was written by Linus Torvalds (the developer of the Linux kernel). “Distributed” means that the entire revision history is stored on your local machine, which allows you to manage commits, rollback changes, merge branches, tag, etc., all without having to maintain a constant connection to the central svn server.

This also allows you to use very non-linear development methods, and incrementally commit changes and publish those changes on SVN in one batch once you know that everything that you’re working on works.

The other really nice part of git is that you can branch your code in-place on your local machine, meaning that you can branch the trunk and make multiple branches (and even branches of branches) to try things out and see what works best. Once you have your changes in place, you can merge the branches back together and commit them to svn.

I’ve has been using this for my development starting in May 2011, and I’ve been really happy with it, although it takes a little time to get used to. This page is working document of some of the tricks, lessons, and best practices that I’ve picked up as I’ve started to use git. (BTW, I’m so happy with git that all my personal development projects have been entirely migrated to git both on the workstation and my server).

If you have any questions, make sure to post a message below. I continue to add to this tutorial so that I have a central place to store my working knowledge of git svn.

Thanks!

Introductory Material (or reference material if you get stuck)

To start with, here is a cheatsheet of git commands, and how they relate to svn. Now, git can do a LOT more than this, but it will help you get started with understanding how git works.

For a more lengthy introduction to git, read the Git Tutorial and then the Everyday GIT with 20 commands or so page.

Typical Workflow with Git and SVN

Before you start this tutorial, if you want to also try committing to a svn server, send me an e-mail at kris@rkrishardy.com, and give me your e-mail address so that I can give you commit access. (I just ask that you be considerate with what you commit, and don’t commit anything that is not tasteful).

To get started, first, download and install Git. If you are on Windows, download Git from http://git-scm.com. If you are on a Unix/Linux variety, use your repository or package manager.

On Fedora/RedHat/CentOS:
$ sudo yum install git git-svn

On Debian/Ubuntu:
$ sudo apt-get install git-core git-svn

For FreeBSD, check out the instructions for installing packages and ports, depending on your preference.

Now, let’s use the git-svn-sandbox project at code.google.com as an example to show you how this works.

The directory that you want to clone from svn is the one with the trunk/tags/branches directories inside of it. For example, let’s clone git-svn-sandbox.

$ git svn clone https://git-svn-sandbox.googlecode.com/svn/ git-svn-sandbox -Ttrunk -bbranches -ttags --username your@emailaddress.com This may take a while on large repositories Checked through r... ... Checked out HEAD: http://code.google.com/p/git-svn-sandbox r...

This will create a git-svn-sandbox folder and clone the entire svn history into the git-svn-sandbox/.git folder. This may take some time because it needs to walk the entire svn revision tree, so grab some tea or coffee and give it a few minutes.

Once the cloning is done, you should have a local ‘master’ branch in place:

$ cd git-svn-sandbox $ git branch * master

The ‘*’ shows you which branch you currently have checked out. You should currently be in the “master” branch.

Now, see what remote tracking branches are available (these are on the SVN server):

$ git branch -r tags/1.0 test trunk

git branch is the command to use for managing local branches. Using git branch, you can create, delete, merge, etc. your local branches. (These are isolated from the SVN remote branches, and branches you create here are not automatically uploaded to the SVN server, so create and delete branches without fear).

SVN doesn’t handle conflicts smoothly, so here’s what I do. Before I start changing anything, I first create a new local branch and start working on that. If you forget to branch, that’s OK. If you get a conflict during the git svn rebase or git svn dcommit, see the “I did an git svn rebase, but I got a bunch of merge errors. What should I do?” section below.

$ git checkout -b my_branch Switched to a new branch 'my_branch'

You have created a new branch and switched to it. This command was
shorthand for:

$ git branch my_branch $ git checkout my_branch

Now, and and/or edit the files you need to. When you are done, take a
look at the list of files that changed.

$ git status # On branch my_branch # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: test.txt # # Untracked files: # (use "git add ..." to include in what will be committed) # # abcd.123 no changes added to commit (use "git add" and/or "git commit -a")

Here, text.txt is already in the git index and has been modified but not flagged to be committed. Also, abcd.123 has been added, but is not in the git index.

To add both to the git index, use git add -A (which adds all untracked and modified files to the index).

$ git add -A

Now you can commit the files to the git branch using git commit.

$ git commit [my_branch db1292d] Saving changes 2 files changed, 3 insertions(+), 0 deletions(-) create mode 100644 abcd.123

When you are done developing and you want to share your code on the svn server, make sure that you have all the files checked into your development branch.

$ git status

Add and commit any changes that show up, and then checkout the master branch.

$ git checkout master Switched to branch 'master'

If you didn’t remember to do your development in a different branch, and you made modifications in master, follow the instructions in “I did an git svn rebase, but I got a bunch of merge errors. What should I do?” if you run into any problems with the next set of steps.

Rebase your master branch with the svn server.

$ git svn rebase ... Current branch master is up to date.

If any files had been committed to svn by other developers, you should see the changes applied in the output of git svn rebase.

Merge in the changes that you want to commit to svn.

$ git merge temp Updating 3877696..f68e10c Fast-forward test.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 test.txt

If you see any conflicts in the output, you can find the conflicts by using git status.
$ git status

If you want to diff the conflicts, use git diff.
$ git diff

Once the conflicts have been resolved, make sure that all the changes have been committed to master. Then commit the changes to svn using git svn dcommit.

$ git commit $ git svn dcommit --username your@emailaddress.com Committing to https://git-svn-sandbox.googlecode.com/svn/trunk ... A test.txt Committed r4 A test.txt r4 = 4c55148ecefac8b83b626fa568b2bc1f99ea990e (refs/remotes/trunk) No changes between current HEAD and refs/remotes/trunk Resetting to the latest refs/remotes/trunk

If git svn dcommit failed with a message stating “Authorization failed: …”, I need to give you commit access to the repository. Please send me an e-mail at kris@rkrishardy.com, and give me your e-mail address so that I can give you commit access. (I just ask that you be considerate with what you commit, and don’t commit anything that is not tasteful).

If you got the “Committed r…” line, congratulations! Your changes have been published to the svn server!

If you want to continue working on your local development branch, check it back out and rebase it to bring it up to date with the master.

$ git checkout my_branch $ git rebase master First, rewinding head to replay your work on top of it...

When you are completely done with your branch, you can delete it by checking out a different branch and using git branch -d.

$ git checkout master $ git branch -d my_branch Deleted branch my_branch (was 15971cb).

Congratulations. You now have the basic workflow down. If you run into problems, checkout out the “Troubleshooting” section below, or take a look at the Git documentation at Git SVN Cheatsheet, the
Git Tutorial, the Everyday GIT with 20 commands or so page, or the extensive Git Users Manual.

Troubleshooting

I did a git svn rebase, but I got a bunch of merge errors. What should I do?

Before you do anything, revert the rebase by typing:

$ git rebase --abort

Now, make a backup of your current master by branching it.

$ git branch temp

Use git log to figure out which commit was the last one that was sync’d with svn. Here is an example of the output:

$ git log commit 4ebf8122c5b18ab16167268b1791fa49996e56cc Author: Kris Hardy Date: Sat Jul 2 10:01:55 2011 -0400

Modified test.txt

commit ec8a9e15a30c01c7ad7b83d6cb8cde39e1c6a650 Author: hardyrk@gmail.com Date: Sat Jul 2 13:58:02 2011 +0000

Adding initial files

git-svn-id: https://git-svn-sandbox.googlecode.com/svn/trunk@2 fb457a7e-32b5

commit 96acefc8a443a02568453c7a504064ffc6d8428e Author: (no author) <(no author)@fb457a7e-32b5-5db0-b168-d315aa6739ac> Date: Sat Jul 2 13:12:00 2011 +0000

Initial directory structure.

git-svn-id: https://git-svn-sandbox.googlecode.com/svn/trunk@1 fb457a7e-32b5

Look for the latest git-svn-id to find the latest commit. In this example, it
was commit ec8a9e15a30c01c7ad7b83d6cb8cde39e1c6a650.

You only have to use the first few characters of the commit id, so we’ll
use ec8a9 as a short-hand for the full commit string.

Now, reset the master branch to the last version that was sync’d with
SVN. NOTE: This command will erase commits that occurred after the
commit, so make sure you backed up your master branch (using git branch)!!!.

$ git reset --hard ec8a9 HEAD is now at ec8a9

Now, you are safe to rebase master with the svn trunk using git svn rebase.
$ git svn rebase ... Current branch master is up to date.

If any files had been committed to svn by other developers, you should see the changes applied in the output of git svn rebase.

Merge in the changes that you want to commit to svn.

$ git merge temp Updating ec8a9..4ebf8 Fast-forward test.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 test.txt

If you see any conflicts in the output, you can find the conflicts by using git status.

$ git status

If you want to diff the conflicts, use git diff.

$ git diff

Once the conflicts have been resolved, make sure that all the changes have been committed to master. Then commit the changes to svn using git svn dcommit.

I checked out an svn branch (or tag), and how svn dcommit in master pushes the changes to the svn branch!

When checking out a remote branch or tag, master sometimes stops tracking the svn trunk, and instead starts tracking the svn branch or tag. To fix this:

Check the git master branch back out
Back up any changes
Do a hard reset, setting the svn trunk as the remote tracking branch
Reapply the backed up changes

$ git checkout master $ git branch temp $ git reset --hard remotes/trunk HEAD is now at ... $ git svn dcommit -n Committing to https://git-svn-sandbox.googlecode.com/svn/trunk ... $ git merge temp Updating ... $ git branch -d temp Deleted branch temp (was ... $ git svn dcommit Committing to https://git-svn-sandbox.googlecode.com/svn/trunk ...

Adapted from StackOverflow: How do I make git-svn use a particular svn branch as the remote repository?

Stupid Git-SVN tricks

I want to ignore files from git (just like setting ‘svn:ignore’)

To ignore files from git, add the filename or filename pattern to a .gitignore file in the folder that the file is in, or in one of the folder’s ancestors if it is a global ignore (such as ignoring *.o or *~ files, for example).

$ touch .gitignore $ echo *.o >> .gitignore $ git add .gitignore $ git commit -m "Ignoring *.o files"

How do I create a tag in SVN?

Use git svn tag.

$ git svn tag 1.0 Copying https://git-svn-sandbox.googlecode.com/svn/trunk at r4 to https://git-svn-sandbox.googlecode.com/svn/tags/1.0... Found possible branch point: https://git-svn-sandbox.googlecode.com/svn/trunk => https://git-svn-sandbox.googlecode.com/svn/tags/1.0, 4 Found branch parent: (refs/remotes/tags/1.0) a029adfba661303ad73be48fcbb92372fba9a9f1 Following parent with do_switch Successfully followed parent r5 = 974647bc0548d3d21c67d7f6fe2904da2f02a846 (refs/remotes/tags/1.0) $ git branch -r tags/1.0 trunk

If you want to check to make sure that master is still tracking the svn trunk, use git svn dcommit -n.

$ git svn dcommit -n Committing to https://git-svn-sandbox.googlecode.com/svn/trunk ...

How do I create a branch in SVN?

Use git svn branch.

$ git svn branch temp Committing to https://git-svn-sandbox.googlecode.com/svn/trunk ... $ git svn branch temp Copying https://git-svn-sandbox.googlecode.com/svn/trunk at r4 to https://git-svn-sandbox.googlecode.com/svn/branches/temp... Found possible branch point: https://git-svn-sandbox.googlecode.com/svn/trunk => https://git-svn-sandbox.googlecode.com/svn/branches/temp, 4 Found branch parent: (refs/remotes/temp) a029adfba661303ad73be48fcbb92372fba9a9f1 Following parent with do_switch Successfully followed parent r6 = 4c0195271d0d586f2facd2707a83653844fec3a2 (refs/remotes/temp) $ git branch -r tags/1.0 temp trunk

If you want to check to make sure that master is still tracking the svn trunk, use git svn dcommit -n.

$ git svn dcommit -n Committing to https://git-svn-sandbox.googlecode.com/svn/trunk ...

How do I checkout/track a remote branch or tag from SVN, and manage the files using Git?

You can create a git branch, and have it track the remote svn branch using git branch. The git branch will now be synchronized with the svn tag or branch.

$ git branch local/tags/1.0 tags/1.0 $ git branch local/tags/1.0 * master $ git branch -r tags/1.0 temp trunk $ git checkout local/tags/1.0 Switched to branch 'local/tags/1.0' $ git svn dcommit -n Committing to https://git-svn-sandbox.googlecode.com/svn/tags/1.0 ...

My branches are forks of the svn trunk. How can I keep the forks up to date without the conflicts caused by git merge?

See this page for a great walkthrough (hint, use git rebase –onto):

Maintaining a Fork With Git

Technorati Tags: git git-svn code.google.com googlecode tutorial

Comments (0)

April 14, 2010

Testing an Enterprise Application Integration (EAI) Implementation

Filed under: Articles,Development,enterprise application integration — Kris @ 8:07 am

Adapted from the following presentation:

“Testing an EAI Implementation”

Matt VanVleet
VP Product Development and Practice Management
Pillar Technology

Enterprise Application Integration Alliance
Columbus, Ohio
http://www.meetup.com/Enterprise-Application-Integration
4/8/2010

Before I begin, I want to give my thanks to Matt VanVleet of Pillar Technology for giving us his time, knowledge and experience for his presentation. Most of the following article is directly from, or inspired by, what he shared.

I also want to thank the members of the Enterprise Application Integration Alliance for attending. Without them, valuable presentations like Matt’s would not be possible.

Now to the topic at hand…

If you look at the architecture of a standard Enterprise Application Integration (EAI) implementation, it typically consists of a series of independent applications or systems that are loosely-coupled to one another through a central integration system (such as an Enterprise Service Bus).

EAI projects usually consist of 4 phases, each of which create unique challenges to the enterprise:

Implementation
Upgrade the EAI
Upgrade the Back-end Systems, Data Warehouse or Applications
Upgrade a 3rd Party Integration

1. Implementation

When EAI projects are started, the EAI team typically works closely with the teams that are in charge of each application that needs to be integrated. Each team needs to cooperate with the development, documentation, and testing. They also need to coordinate releases so that the integration does not break if some critical component or feature of the application or the EAI system itself is being upgraded.

Since so many teams are involved and the implementations are complex, the cost of implementing an EAI system are high.

In addition, since releases need to by sync’d across the enterprise, it slows down the speed of development and adds at least one additional layer of management. Furthermore, while each endpoint is loosely-coupled to one another, it is tightly coupled to the EAI system itself. Each EAI-to-endpoint integration creates a potential point of failure, so each change needs to be thoroughly tested before being deployed.

2. Upgrade the EAI

Once an EAI implementation is in place, upgrading the EAI can be as time consuming and expensive as developing the EAI in the first place. Each team has to brought back together the plan, develop and synchronize the upgrade.

3. Upgrade the Back-end Systems, Data Warehouse or Applications

Some of the teams will be needed for this upgrade, including the EAI team and the team in charge of the application being upgraded.

4. Upgrade a 3rd Party Integration

The EAI team will be very involved in this, and it may require the coordination and help of the 3rd party service provider that you are integrating with.

Issues

Since each of these systems is now tightly-coupled to the EAI system, upgrades at any point along the enterprise become major events. One slight change, unless properly planned, managed and tested, can have a significant impact at some other end of the enterprise.

As the enterprise generates more and more data, the EAI implementation is put under more and more load. What could be simple tweaks to the persistance layer or server configuration now involve much more risk.

Also, once an EAI implementation is in place, it never gets the attention that the applications themselves get. Since EAI is behind the scenes, it tends to be an afterthought when it comes to the corporate budget. It’s necessary, but it’s not “sexy”.

What is the best way to manage this? How can you ensure that your EAI system is stable, while allowing the teams responsible for each application to be as autonomous as possible? How can we decrease the upgrade cost of an EAI and also make it flexible enough to now slow down the application developers?

Solution

If you look at an EAI implementation as a central data system and don’t worry about how it works on the inside (which is a reasonable assumption when you are just dealing with interfaces), you can simplify it as a black box connected to a series of interfaces: Application-to-EAI, Data Warehouse-to-EAI, Web Service-to-EAI, etc.

At each interface, you can then divide the interface into two halves: 1) Application-side and 2) EAI-side

If you are an application developer, you are developing your application to work with the interface that the EAI implementers expose. It could be a database socket, a web service, flat files, messaging, etc. This is essentially the same for the Database Warehouse and Back-end systems (such as SAP).

At each interface, there is typically inbound and outbound traffic. Some of the traffic will be the result of the application itself (such as adding a customer to a CRM which is then forwarded to the ESB), some of the traffic will be responses to data sent by the application (a reply from the ESB), and some of the traffic will sent without any aparent trigger (the ESB sends a message to the application, caused by new data from another application).

To ensure that your EAI implementation and the applications are communicating properly, you can put tests in place to make sure that the systems respond properly. You can also develop simulated systems, also known as mocks, that act like the system that is at the other end of the interface.

These tests and mocks can help you during any stage in the lifecycle of your EAI implementation:

During development – Test your applications and ESB, independently, to make sure that they respond to events in the right way.
During production – Regularly test your applications and ESB to identify any mismatches or problems immediately. With up-to-date tests, it is even possible to detect defects in the system that were somehow missed by a development team. You can then respond proactively and disconnect the communication to the bad system, for example, to keep the other systems across the enterprise running properly.

Now, let’s use the internal application-to-EAI implementation interface as an example.

Application-to-EAI Testing

When you are developing an EAI or application to be compliant with the Application-to-EAI interface, there are 4 test assets that need to be developed in order to assure that both the application and EAI system are working properly.

Mock of EAI Implementation – Used to test the application against an EAI system that responds correctly to requests and generates application-bound traffic.
Test of Application interface – Used to test the application to make sure that it responds properly traffic from the EAI implementation.
Mock of Application – Used to test the EAI system against a fake application that responds correctly to requests and generates EAI-bound traffic.
Test of EAI interface – Used to test the EAI system to make sure that it responds properly to traffic from the application.

3rd Party-to-EAI Testing

If you are integrating with a 3rd party, the layout is the same. However, there may be a larger barrier between the EAI team and the 3rd party team. In that case, you probably will have to assume that their testing is in place, or work with them to ensure that they have their tests and mocks in place.

If they don’t have a valid mock system (which, unfortunately happens), you may have to build a mock internally using what you can learn about the 3rd-party system.

If you don’t currently have tests or mocks in place, one way to start is to use a “wire-tap” or proxy to log messages, requests and responses in order to build the test cases.

End-to-End Integration Testing

The necessity of testing an entire enterprise application can be significantly reduced by doing as much testing at the atomic level as possible. By testing the application-to-EAI interfaces and the intra-EAI processing through EAI-specific tests, the need for end-to-end testing is, theoretically, eliminated. If the tests cover all the corner cases, all the EAI processes and interfaces pass the tests, then the EAI is working properly. At each interface, if all tests that invoke the mock services pass, then the applications work properly with the interfaces. End-to-end testing, at that point, isn’t necessary.

However, there are commonly reasons for doing automated end-to-end tests just to “be absolutely sure” that data is flowing from one endpoint application to another. In that case, it requires the collaboration of the application teams in order to build the tests and observers for the full end-to-end tests.

Technology

These are a few of the many open-source applications that can be used to help you develop the tests and mocks.

Tests

jUnit (or a member of the xUnit family)
Depending on the EAI implementation, some IDEs (such as Netbeans w/ OpenESB) have testing integrated into the system.

Mocks

Considerations

Someone has to “own” the specification for each interface. Personally, I believe that the specification should be owned by each application team, in collaboration with the EAI team. However, due to the complexity of the interface or the size of the implementation, it may be better to have the EAI team own the specification, or perhaps put it in the hands of a higher-level enterprise data architect.
Someone has to develop the mock of the EAI. A collaborative effort between the application team and the EAI team usually work best to build the interface specification, and then the mock is maintained by the EAI team so that it is always up-to-date with the production and development EAI implementations.
Someone has to develop the tests for the EAI. Again, a collaborative effort to build the specification is needed, and then the tests are maintained by the EAI team so that it is always up-to-date.
Someone has to develop the mock of the applications. A cross-team collaborative effort is needed, and then the mock is maintained by the application team so that it is always up-to-date.
Someone has to develop the tests for the applications. The same application-EAI collaborative efforts is needed to build the specification, and then the tests are maintained by the application team so that it is always up-to-date.

Summary

Testing and mocking for EAI implementations allows each team to stay independent by testing their applications against an interface and catching problems before they go to production. This decreases the cost of developing and maintaining enterprise architectures by reducing the interdependence between each development team, as well as by reducing the potential for regressions.

In one case that Matt talked about, a company he worked with was planning an upgrade to their EAI system, which ordinarily is a very expensive process. Matt’s company had created all the tests and mocks for their old EAI system, so during the upgrade, they ran all their tests and mocks against the new EAI and they were able to immediately find the regressions and fix them. This decreased the development and implementation time by over 50-fold.

Testing of EAI implementation is becoming more main-stream, but does involve some investment up front. That investment, however, will pay handsome dividends when you upgrade any of your systems in the enterprise and you need to retest to make sure that it works properly. Automated testing, built once, can save you weeks of hand-testing during each upgrade.

Thanks again to Matt VanVleet for his presentation and to the members of the EAI Alliance for attending. If you are in the Columbus, OH area and are interested in EAI at any level, from programmer through executive, be sure to sign up (it’s free), and take part in our meetings. http://www.meetup.com/Enterprise-Application-Integration

Comments (2)

December 12, 2009

Subversion Fix: svn copy causes “Repository moved permanently to ‘…’; please relocate

Filed under: Articles,Debugging — Tags: apache, repository moved permanently, subversion, svn copy — Kris @ 12:54 pm

Background

Subversion is a version control system. It can run as either it’s own server (svnserve), or as an Apache module (mod_dav_svn.so).

When using the mod_dav_svn module for Apache, and doing an svn copy operation on the repository itself can fail if the VirtualHost configuration for subversion is not correct. Put simply, if Apache itself and mod_dav_svn are serving content from the same path, then conflicts can occur. Apache can get confused if it attempts to serve a physical file instead of routing the request through mod_dav_svn.

“svn copy …” operations will fail, while “svn update …”, “svn commit …”, and “svn checkout …” operations work fine.

Detail of the problem, diagnosing the problem, and the fix are below.

(more… >>)

Technorati Tags: apache, repository moved permanently, subversion, svn copy

Comments (12)

November 24, 2009

Confessions of a Researcher-turned-Engineer

Filed under: Articles,Development,Random Thoughts — Tags: education, einstein, feynman, newton, problem solving — Kris @ 1:59 pm

I wasn’t always a software engineer…

Before I began developing software for a living, I used to be in chemical research & development as a biochemist. For some reason, I always found myself gravitating back towards software and informatics, so I eventually gave in and started a software company. But I’ve learned a lot of lessons during my time as a scientist.

When solving a difficult problem and you know 25% of the solution, you can figure out 70% through hard work, patience and trial-and-error. The last 5% may never come, and if it does, is rarely when you’re looking for it.

“I am enough of an artist to draw freely upon my imagination. Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world.” – Albert Einstein

Just because you don’t know the answer right now, doesn’t mean that you can made headway and figure it out as you go. I’ve learned the most when I’ve tried things that haven’t worked, but figured out what went wrong and then tried again.

If you’re not continually learning and improving yourself, your working days are numbered.

“Genius without education is like silver in the mine.” – Benjamin Franklin

Don’t be afraid of challenging yourself and learning new languages, technologies or skills. Each one of them expands your experience and perspective, and can give you an opportunity to take a look at the status quo. Unfortunately, self-motivated learners are in short supply but in constant demand because they can adapt to any situation.

If you want to succeed, you have to train for it.

“You can know the name of a bird in all the languages of the world, but when you’re finished, you’ll know absolutely nothing whatever about the bird… So let’s look at the bird and see what it’s doing — that’s what counts. I learned very early the difference between knowing the name of something and knowing something.” – Richard Feynman

Let’s face it… The likelihood that you’re going to “knock one out of the park” your first time up to bat is pretty low. Whether it’s business or baseball, the odds are against you. It takes grit, determination, exercising yourself both physically and mentally, and lots of disappointment as you fail again and again. But each day, train yourself a little more, a little harder, and each day you get a little stronger. It’s cumulative, and it takes a lot of time.

Give credit to those who came before you.

“If I have seen further it is by standing on the shoulders of Giants.” – Sir Isaac Newton

In science, recognition is incredibly important and one of the first lessons that you learn. When you’re making a presentation, writing an article, or doing your homework, you have to cite any sources of information that when into your work. This is partly because scientists are focusing on sharing knowledge, and in order for the community to work together, there has to be trust between researchers that their information will not be stolen by one-another.

Even more so, no inventions are made in complete isolation. They are incremental improvements based on our understanding of our world and everything that has come before. The iPhone wouldn’t exist without the much earlier inventions of silicon wafers, transistors, plastics, aluminum, and light. And the next wave of inventions will be no different.

Take advantage of the information and knowledge that we have, but show respect to the community that this knowledge came from. You probably will need more help from it in the future.

Technorati Tags: education, einstein, feynman, newton, problem solving

Comments (0)

November 20, 2009

Micro-Scrum: Agile Development for the 1-Man Army

Filed under: Articles,Development — Tags: agile, junit, microscrum, phpunit, scrum, submerged solutions — Kris @ 10:32 am

I’ve found that Scrum, and Agile methods in general, are incredibly valuable in single-man “teams” if done properly. In my experience though, there are some sticking points that need to be worked through with how you work with your clients and manage communications. These will have a significant impact on your projects and how successful Scrum can be for you.

Before I get too far, let me give a quick overview of Scrum:

Scrum development is a business process that seeks to speed up development, decrease bugs, and increase the quality of products that are delivered. It doesn’t have to be focused on software development, but that’s where it has the greatest pull right now, and the use that I am going to focus on.

Scrum, at least how we use it (mixed with Extreme Programming), uses several key components:

Iterative and Incremental Development – A product is released after each “Sprint” (typically 1-4 weeks).
Test Driven Development – Automated unit tests are created to test the code, databases, and user interface. For PHP, we use PHPUnit and some in-house database testing tools. For Java and integration endpoint API tests (regardless of language), JUnit would be a good start.
One-Touch Build, Document & Deploy – Automated build and deployment sequences are developed to reduce variables, problems and time. Ant is a great tool for this.
Continual Integration – The software is compiled and thoroughly tested nightly (and as needed) throughout the sprint. CruiseControl is useful for this, although you can also do something as easy as schedule a nightly cron job to execute an ant build file.
Source Version Control – We use a central version control system (Subversion), to manage our code. This allows me to work from multiple computers on the same code (such as when I’m traveling), and it also allows me to work with other developers when I’m partnering or outsourcing work.
Product Backlog – The product backlog is a list of stories and feature sets that our customers want in their software that haven’t been completed yet. Each story or feature is then prioritized by business necessity and value to the business. I’ve heard good things about XPlanner and Pivotal Tracker, and IDEs such as Visual Studio, Eclipse and Netbeans support backlogs. We prefer to use Kanban boards, which are adapted from the Lean Manufacturing method.
Sprint Backlog – The sprint backlog is the list of stories and feature sets that we will complete during a single sprint. The items that are put in this backlog are negotiated between us and our client. Once the items are chosen for the Sprint, they are then fed into the “Work In Progress.” The stories and features are then fed into our web-based project management tool. We currently use dotProject for this, but a wiki, Google Wave, or a project management application like BaseCamp or MS Project would work just as well. The important thing is that your client must be able to read each item and make notes on them.
Work In Progress – A series of stages are used to track the progress of work that is in progress during a sprint:
1. Awaiting Tests
2. Test Writing in Progress – We develop the tests before we write the code.
3. Awaiting Code
4. Coding in Progress – In this stage, the application is actually being built
5. Awaiting Integration
6. Integration in Progress – The code changes are integrated into the main development code base for this sprint, and all tests must pass before continuing to the next phase.
7. Awaiting Documentation
8. Documentation in Progress – The documentation for the application is updated and rebuilt. PHPDoc is a great help for us in our PHP applications, and can extract documentation directly from our in-line documentation in the code. Similar applications exist for many other languages.
9. Awaiting Acceptance
10. Done – The story or feature has proper test code coverage, all tests pass, the code has been properly integrated, documented, and signed-off by person in charge of Quality Assurance. In our case, this is done through a demo with the client.
Sprint Review – During the review, we reflect on and catalog all our lessons learned during the sprint. By looking at went well and what didn’t go well, we can make changes in our process for our next sprint. Also, the client’s likes and dislikes of the last product release are included. We use a wiki for this, such as MediaWiki or dokuWiki.

A typical Scrum team consists of:

Product owner – The person who is responsible for the delivery of the project to the client. This is usually a Business Analyst or Project Manager.
Users - The end-users, commonly the client’s staff.
Stakeholders - Typically, these people are the client-side Project Managers and are the ones that are paying for the product.
Scrum Master – The member of the consulting or development team that is responsible for leading the Team Members, and reports to the Product Owner.
Team Members – The actual development team

When you are running a Micro-Scrum (such as if you are a lone consultant), the scrum master, team members and product owner are consolidated to a single person (you). When your clients are like ours, and are almost all small businesses, your client contact ends up filling multiple roles. The actual Scrum roles end up looking more like this:

Scrum Master/Team/Product Owner (Consultant)
Stakeholder/User (President, CEO, or General Manager, etc.)
User (Client’s Staff)

By consolidating the Scrum roles, I, as the Product Owner, now don’t just need to make sure that the product is developed and delivered. I also need to fill the roll of a Business Analyst, so I need to understand the business case, consolidate all the user’s needs and negotiate with the client to prioritize the importance of each user story. I also have to avoid focusing on my own “pet features”, which developers can sometimes do, and instead have to focus on what will bring my client the greatest value in the shortest amount of time.

Standing in the client’s shoes, the client needs to be both the stakeholder and the end user. As the stakeholder, the client needs to watch the business’s bottom line and help prioritize the user stories for the development team. The stakeholder’s goal is to get the best product that meets the most critical needs within both schedule and budget. As the end user, the client also needs to look critically at their business processes and workflows. He or she needs to be able to understand and explain the strengths and weaknesses of their current software to the rest of the team. The client also needs to provide his/her time and in-depth knowledge to the team, enabling them to architect and develop the software that best meets the client’s business needs.

Consolidating those roles makes things both easier and harder at the same time. It’s a bit easier because there are fewer people that you need to communicate with. However, it also makes things more difficult because there is more responsibility that is asked of each person, and, by its’ nature, each person has a greater impact on the success or failure of the project.

The ultimate success of a Micro-Scrum development project hinges on the ability of both the consultant and the client to be proactive, efficient and thorough communicators. In my experience, the quality of this communication, especially early in a project, will make all the difference.

There needs to be a very clear understanding from both the client and the consultant on what is going to happen throughout the life of the project and how the product is going to be developed and delivered. Both the consultant and client need to be transparent and honest with one-another. Micro-Scrum works especially well if both parties work hard at building a good rapport with one another.

When both the client and consultant are on the same page and can effectively communicate, the project works very well. When our client is not a willing communicator, delivering a product that meets their needs is a lot more difficult.

5 Hard-Learned Lessons in Effective Micro-Scrum Projects

1. In Micro-Scrum, the focus needs to be on quality, NOT on cost.

All too often, small Agile studios like mine are compared to freelancers or overseas developers on sites like Elance and Guru. The bottom line is that we cannot complete with their hourly cost, so we don’t even try. What we focus on is Quality, above all else.

We deliver unit-tested code, usability and performance tests, code reviews, security audits, and are actively involved in communication with our clients throughout the entire process.

It has worked well for us, and we’ve kept our current clients for nearly 2 years at full workload. The past year has been our best year ever, despite the recession.

2. Micro-Scrum development usually costs more, initially, than methodology-less development (aka cowboy coding).

This goes along with the previous lesson. The up-front costs are usually higher, however the bug rate is much lower and the documentation is better. Also, the risk and time involved when refactoring and modifying the code is MUCH lower. Software changes are anticipated, so we develop with that in mind.

3. When working with a new client, deliver early and often.

When working with non-tech-savvy businesses, keep the sprints short. We like to do 2-3 day sprints at the beginning to get them accustomed to the process. Even if that means just delivering them a framework, some raw code, a library or sketches, it pays off dividends over the course of the project. It opens up communication and makes sure that our direction matches their vision.

In addition, you get to run through a deployment dry run each time. If you notice problems in the process, you can fix them early on rather than waiting until the end and being surprised that their server doesn’t support a dependency that you built into your software. (It may sound silly, but it happens.)

The risk with rapid iteration is that if you don’t explain why you’re doing it, your client may feel like you just don’t understand what they want. They’re right!

The entire purpose of these short sprints at the beginning is to teach your client how to communicate with you, and to learn what your client wants by giving them something tangible that they can critique and use as a reference. That tangible prototype can help you understand what they really want, which often is different that what their Request for Proposal or Product Specification actually says.

4. Keep the up-front requirements gathering as short as possible.

Feel free to post flame-comments below.

I’ve run into many clients that don’t always know how to explain their product vision and be clear with what they ultimately want. For about 1 full month back in 2008, I beat my head against a wall because one of my clients seemed to contradict their needs on a weekly basis with an accounting system I was developing for them.

I decided, out of desperation, to cut the requirements gathering short and push forward with what I had at that point. Amazingly, I was pretty much right. After Sprint 1, there were some modifications and changes, of course, but I was able to hit the target with only about 30% of the requirements being known up-front. In addition, it forced us to build a very flexible architecture so that we could easily adapt to the unknowns and changes when we hit them.

What we do now is start with a very quick Sprint 0, maybe a week at most. During Sprint 0, we build the user stories, initial specifications and product backlog. There will be lots of unknowns and changes later on, so knowing every specific detail is neither important nor possible. Managing and working around those unknowns is just part of software development, especially when you’re working with companies that don’t understand software development.

5. Failure is GOOD!

This may seem counter-intuitive, but failure is a good thing. It opens the lines of communication for constructive feedback.

A goal we have for all our projects is to “fail fast.” What that means is that we deliver a product in an early iteration (ideally Sprint 1) that our client will see lots of problems with. Maybe the interface doesn’t meet their needs, or the work flow isn’t intuitive. Whatever it is, I actually want my client to come back to me and say “I don’t like this.” I then ask them “What specifically don’t you like?” Or I’ll ask my favorite question: “Can I set up a screencast for you and you can show me what’s happening?”

We take each failure as an opportunity to open the lines of communication even further. You can learn exactly what your client wants by watching him or her use your application. Developing a Micro-Scrum team takes time, and building the relationship and trust through open communication is essential.

I’m sure you’ve noticed that communication is a common theme in Micro-Scrum. Actually, it’s really the primary driver behind Scrum and any other Project Management methodology. In order to deliver what a client wants, you need to have clear communication to truly understand their needs. In my eyes, there is no other way to deliver a successful product.

Scrum formalizes the communications and works to get everyone, from the users and stakeholders to the developers, engaged in the success of the product. When you are Micro-Scrumming, this communication becomes even more vital since there are fewer people involved and more responsibility on each person. Harnessed wisely, it can lead to very successful projects that have good short-term value and outstanding long-term value.

If you’re in the Columbus, OH area and want to learn more about Agile Software Development, I highly suggest going to the Columbus Ohio Agile Alliance (COHAA) meetings and getting involved in TechLife Columbus meetup group.

Technorati Tags: agile, junit, microscrum, phpunit, scrum, submerged solutions

Comments (0)

November 17, 2009

The Art of Bootstrapping a Business

Filed under: Articles,Random Thoughts — Tags: bootstrapping, business development, columbus ohio — Kris @ 5:14 pm

I was sent this great article that Guy Kawasaki wrote way back in 2006 about bootstrapping that I just have to share.

At it’s essence, Bootstrappers are business owners that start and grow their business without any outside funding. That means no angel investors and no VC deals. You don’t hear about it much because it’s not glamorous, but it’s the way that most businesses are started.

I am a hard-core believer in bootstrapping, and I know for a fact that I have learned a lot of lessons the hard way that I wouldn’t have learned as easily if I had not been a bootstrapper. (Mo’ money, mo’ problems anyone?)

Especially when you start your first business (and if you’re like me, your second as well), you WILL make mistakes, and lot of them. I still make mistakes, but I like to think that my hard-fought experience has saved me several times. Sometimes I can get out of the way in time, and sometimes I still get run over. But I’ve become better and better at keeping the damages small and keeping the business agile.

If you’re in the Columbus, OH area, come out to a meeting of the International Bootstrapping Association and say hi! Our next meeting is Dec 9th, 2009 at TechColumbus hear Ohio State University. RSVP at the link below:

http://www.bootstrappingassociation.org

Technorati Tags: bootstrapping, business development, columbus ohio

Comments (0)

September 3, 2009

Macromedia Founder Creating “Digital City” in Ohio

Filed under: Articles,Development,News — Tags: broadband mechanics, columbus, macromedia, marc canter, people aggregator, social networking, techlife — Kris @ 7:30 am

The Digital City Project - Columbus, OH

Marc Canter, a founder of Macromedia and now CEO/Chairman and Founder of Broadband Mechanics, gave a very interesting presentation/pitch yesterday at the TechLife Columbus Meetup at the Chocolate Cafe near OSU.

If you don’t know Marc, he was with the MacroMind/Macromedia back when they had first developed and released Flash (or maybe it was the predecessor to Flash, I’m not 100% sure). Multimedia is a huge interest of Marc’s, and more recently, he’s been teaching a course at Case Western Reserve University on this concept of a “Digital City”.

The presentation was a little scattered, but here’s what his vision as how I understand it:

Using a special interest of “Workforce Development”, his company, Broadband Mechanics, is developing and deploying an open source social network software platform called “People Aggregator”. This software allows blogging, social networking, live video chat, and all the other usual social-media stuff. It is developed to OpenSocial specifications, so that it can be integrated with other open social media platforms (Facebook, Google, etc.). Now, at this point in the presentation, I was starting to think “so what?”

But here’s what is very interesting about this, and what makes this is unique business. Marc realized that there are a few needs that are unfulfilled in the market today:

(more… >>)

Technorati Tags: broadband mechanics, columbus, macromedia, marc canter, people aggregator, social networking, techlife

Comments (0)

June 1, 2009

Targeting Your Website Based on a Visitor’s Location

Filed under: Articles,Development,Marketing — Tags: geotargeting, ip address, ip geolocation, market segmentation — Kris @ 7:44 am

Here’s a question about geolocation-based online marketing I received today:

I use google analytics. The service I sell online is location specific. I am getting visits clustered around various locations.

Any creative ideas on how to target visitors from specific locations? Does this make any sense?

Something like: If you are from San Diego, click here for a special offer?

Thoughts? Ideas? Experience to share?

Geotargeting your advertising messages is INCREDIBLY beneficial in a lot of cases.

(more… >>)

Technorati Tags: geotargeting, ip address, ip geolocation, market segmentation

Comments (2)

May 30, 2009

Revolutionize Your Business Through Market Innovation

Filed under: Articles,Marketing — Tags: apple, ipod, itunes, market innovation, podcasting — Kris @ 11:09 am

I was reading Ed Dale’s blog the other day, and this term popped into my mind. I don’t think that I’ve heard it before, so I’m going to coin it…

“Market Innovation”

So what is Market Innovation and why am I writing about it?

Market Innovation is a 5-step process:

Start by looking at a large market.
Identify a segment of that market that has a common, unfulfilled need.
Pivot the market segment – Innovate a revolutionary product that spins that market segment 180 degrees with your product as the pivot point.
Develop a community of consumers around your product by selling complimentary products to your consumer base.
Control and Grow your New Market – Through shaping the market segment and becoming the pivot, you have created an entire new market around your products. You now control the market. Your mission at this point is to keep them happy and grow your market.

Let me show you an example…

Before Apple came out with the iPod, there were lots of MP3 players around. Most were cheeply made, were missing a lot of features, and required you to convert your own CDs to MP3 so that you could load them on your player.

The iPod changed all that. In late 2001, they completely changed the MP3 player market by solving those problems for the people who were unhappy with their current MP3 players or who wanted an MP3 player but were scared off by the other players at that time.

The iPod was a very nice-looking, easy to operate and well-featured MP3 player. Now the features and design of the iPod alone really grabbed the market’s attention, but I believe that the true market innovation happened with the iTunes software.

iTunes was Apple’s Pivot

iTunes made it REALLY easy to get songs onto the iPod, without having to go to the store, buy a CD, bring it home and convert it to MP3. You could download them directly from Apple to your iPod. Another problem solved.

The Podcast Built the Community

They then built the community of rabbid fans with the invention of the Podcast. By allowing users to syndicate their own MP3 recordings to anyone with iTunes, Apple extablished itself as the center of the internet audio consumer market. By becoming the conduit of music and information, iTunes completely took over the MP3 market as the player-of-choice for most people.

The success of iTunes, in my opinion, is what really is keeping the iPod ahead of the competition. iTunes has become the center of the MP3 market, and through market innovation, has grown the market from a small niche into the main stream.

Apple is still innovating with the iPod and iTunes and finding new things to sell to its consumers. iTunes is regularly being updated and new features are added.

They now have a 75% market share of the MP3 player market, and iTunes has become the 4th largest music retailer.

So what does this mean for you?

I could go through a similar process with every current market leader & long-lived successful product. Google, Microsoft, Wal-Mart, Lowes, the Apple Macintosh, etc.

Market Innovation is at the core of every market leader, but you don’t have to be the leader in order to start. Nor do you have to the be market leader in order to pivot the market. Each of these companies started small at one point, and they found the market that they could pivot. Once you pivot the market, even if it is a small niche, you become the center of it.

At its simplest, I really see it as a 5-step process:

Start by looking at a large market.
Identify a Segment of that market that has a common, unfulfilled need.
Pivot the Market Segment.
Develop a Community of Consumers.
Control and Grow your New Market.

I am putting together a video series on how you can do this in your own market, from market research to launching your product. If you would like to get on the insider’s list and learn how to innovate and control your own market, watch the video at Submerged Solutions:

What are your thoughts?

-Kris

Technorati Tags: apple, ipod, itunes, market innovation, podcasting

Comments (0)

April 6, 2009

AdWords Confusion Lawsuit: Google’s Dismissal Overturned

Filed under: Articles,News — Tags: google, lawsuit, paid search, rescuecom, use in commerce — Kris @ 9:51 am

2d Circuit: Rescuecom Defeats Google's Motion to Dismiss (4/3/09)

When I first started watching these paid search lawsuits, I personally thought that they were non-issues. In several cases of companies suing the search engines for running competitor’s adds on the search engine results pages for a trademarked term, the search engines had won. A precedent had already been set from these cases, right? At least I thought so.

Now, that leads to this article that was published on Friday: Google Loses Round in AdWords Lawsuit.

To boil down the article to the sticky residue that remains, the 2nd Circuit federal appellate court overturned Google’s motion to dismiss the case for allowing the trademark “rescuecom” to trigger paid ads.

While this doesn’t mean that Google will lose the case, it is an interesting reversal since the case was originally dismissed by the district court on the grounds that allowing the keyword “rescuecom” to trigger paid advertisements did not violate trademark law because it wasn’t a use in commerce.

(more… >>)

Technorati Tags: google, lawsuit, paid search, rescuecom, use in commerce

Comments (0)

Older Posts »